Are there plans for mixed/fixed effects analysis options in fitlins?

[adswa]

asking on behalf of @yarikoptic after we stumbled across https://github.com/nistats/nistats/issues/27 when we looked at how the higher level models are computed.

[effigies]

If pybids can produce the correct design matrices and we have an estimation engine that will calculate them, sure.

[tyarkoni]

Pybids doesn't support that at the moment, but I'll add it when needed, as it's part of the BIDS-StatsModel spec. The bottleneck right now is estimation—no imaging-oriented Python package can handled mixed-effects models right now, so our options are to either use FSL or AFNI, or a generic package like StatsModels. In all cases, the space of models those packages can fit is fairly restricted (e.g., none of the above options allow us to fit arbitrarily complex models with crossed random effects).

Basically, until there's a package out there that can efficiently fit arbitrarily complex mixed-effects models, I don't see a big incentive to work on this, as the kinds of models FSL and AFNI can fit at the moment are, to the best of my understanding, just variations on the summary statistics theme.

[adswa]

thank you for being so incredibly fast to respond!

[yarikoptic]

If statsmodels supports it -- what is the problem then, i.e. why not to use it? even if it wouldn't be as efficient as ideally desired, it would be better than nothing. Since it remains massively univariate, we could at least parallelize hell out of it. Backstory: I put poor @AdinaWagner to the task of replicating her prior FSL analysis with the fitlins as the flagman of "do it the right way using BIDS" just to discover now (when failures started to happen only on the 3rd level and not below) that apparently we can't get near in terms of replication. If we could get a typical FSL-like mixed effect modeling going via fitlins, even if only so slow -- then we could have declared a victory! and otherwise we, losers, would just be kicked back home with an eternal shame :-(

[tyarkoni]

If you want to fit FLAME-style models, I think it would make sense to add an FSL interface to fitlins via the existing Nipype interface rather than an entirely new Statsmodels interface. Aside from probably being less work, I don't know that Statsmodels can easily accommodate the FLAME approach (I'm not saying it can't, I just don't know). @effigies, do you have a sense of how much work it would be to add FSL support?

Theoretically, given reasonable simplifying assumptions, fitting the model with FLAME shouldn't require a different BIDS-StatsModels specification from the standard summary statistics approach. So I don't think pybids would need to do anything different; this would all be a matter of fitlins taking the specification from pybids and feeding it to Nipype's FSL interface(s). Statsmodels would be a different story, because it's flexible enough that we would probably want to support explicit representation of different variance components. For FSL, the scope of random-effects modeling is limited enough that I don't think an explicit specification is necessary.

Either way, I think this is likely to be a pretty big job, so I don't know how high of a priority it is for @effigies in the short term.

When you say you "can't get near in terms of replication", do you mean because it fails, or because the results look very different? For most analyses I don't think the nistats results should look that different from FLAME's, so if you're seeing really huge differences, that might suggest major bugs somewhere rather than just modeling differences...

[yarikoptic]

Doesn't flame rely on having varcopes (variances) as well in addition to copes (parameter estimates)?

[yarikoptic]

When you say you "can't get near in terms of replication", do you mean because it fails, or because the results look very different? For most analyses I don't think the nistats results should look that different from FLAME's, so if you're seeing really huge differences, that might suggest major bugs somewhere rather than just modeling differences...

It was just a theoretical/conceptual whining in that we can't do mixed modeling, we are trying to get third level results, and will be thrilled to report back on how congruent they are with the original. Meanwhile we are fighting the daemons of zeroed out designs at the first level, hehehe ;-) we will summon @effigies to the north to chase them away ;-)

[effigies]

@effigies, do you have a sense of how much work it would be to add FSL support?

I don't. It would probably be good to sit down and actually figure out priorities for FitLins, as I feel like there are a lot of big projects people want done, and I'm mostly trying to figure out how to get reports working again and setting up some regression tests.

[tyarkoni]

Doesn't flame rely on having varcopes (variances) as well in addition to copes (parameter estimates)?

Yes, but that doesn't seem like a big deal; it's just a matter of having a naming convention for the output images, and ensuring that fitlins knows which ones to pass to the FSL interface. I'm assuming that if you're going to use FSL for the higher levels, you're probably also using it for the first level, so it's not like we'd have to do anything special to produce the varcopes. From a BIDS-StatsModels standpoint, I don't know that one would (or should?) need to do anything other than say, in an FSL-specific section under "software_options", that you want to use FLAME to fit the model.

[tyarkoni]

My two cents: I think adding support for at least one other estimator should be a pretty high priority. It doesn't matter to me so much which one (either FSL or AFNI makes sense, maybe leaning towards AFNI because the AFNI folks have been very receptive and I think are open to helping out), but I'd feel better if we could establish that fitlins/nistats-generated results converge with other tools. Adding FSL or AFNI support would let us do that in two ways: (a) by assessing convergence between the two supported estimators, and (b) by assessing convergence with the existing GUI-based or command-line driven interface.

[yarikoptic]

maybe leaning towards AFNI because the AFNI folks have been very receptive and I think are open to helping out

+1 for AFNI Moreover AFNI has a clear license (or lack of such due to lack of copyright -- government work), so could be (re)used freely. FSL is non-commercial etc. Unfortunately myself I didn't use AFNI pragmatically yet, but IIRC nipype AFNI interfaces were catching up somewhat although I do not see any interface for higher level analysis of AFNI in nipype. @Shotgunosine -- I guess you guys are at the forefront of the effort there -- any plans to interface e.g. 3dLME so fitlins could potentially establish full GLM analysis using AFNI?

[Shotgunosine]

Sorry I didn't see this comment till now. No immediate plans, but might be a good project to hack on at the Code Convergence.

[yarikoptic]

For some reason I thought that there was some progress in the past year for this issue, but I do not see any references... I guess I was wrong?

[tyarkoni]

There's some progress on the PyBIDS side (https://github.com/bids-standard/pybids/pull/548). If you're okay with a statsmodel- or Stan-based solution (both slow, but the latter much more so), then it shouldn't be much more work to implement a mixed-effect estimator. At least for Stan though (which would allow you to fit arbitrarily complex MFX models), analysis is probably only viable at an ROI level. I don't think the spec itself supports ROI analysis yet; my recollection, which could very well be wrong, is we had planned to push that into the execution spec?