--drop-missing still throws errors with missing regressors

smeisler commented 2 years ago

Environment

FitLins v 0.10.1 run on Singularity 3.9.5 on CentOs 7.6

Expected Behavior

I am trying to implement a GLM that includes 5 aCompCor components as nuisance variables (among other variables). However, some of my functional runs do not have 5 aCompCor components. That is, in these runs, <5 components explained 50% of the variance. FitLins throws an error when trying to run my model on these runs since the 4th or 5th component (e.g. a_comp_cor04) are not present. I expected this error to not happen when including the --drop-missing argument, but it persists.

Observed Behavior

     [Node] Error on "fitlins_wf.loader" (/om2/scratch/Sat/smeisler/fitlins_MIND/sub-MIND3000_phono/fitlins_wf/loader)
220603-16:30:25,900 nipype.workflow ERROR:
     Node loader failed to run on host node032.
220603-16:30:25,900 nipype.workflow ERROR:
     Saving crash info to /om2/scratch/Sat/smeisler/fitlins_MIND/sub-MIND3000_phono/crash-20220603-163025-smeisler-loader-ace8a2cb-f86e-4a04-bb1c-8c156966532f.txt
Traceback (most recent call last):
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 521, in run
    result = self._run_interface(execute=True)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 639, in _run_interface
    return self._run_command(execute)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 750, in _run_command
    raise NodeExecutionError(
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node loader.

Traceback (most recent call last):
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/nipype/interfaces/base/core.py", line 398, in run
    runtime = self._run_interface(runtime)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/fitlins/interfaces/bids.py", line 250, in _run_interface
    self._results['all_specs'] = self._load_graph(runtime, graph)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/fitlins/interfaces/bids.py", line 258, in _load_graph
    specs = node.run(inputs, group_by=node.group_by, **filters)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/bids/modeling/statsmodels.py", line 464, in run
    node_output = BIDSStatsModelsNodeOutput(
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/bids/modeling/statsmodels.py", line 619, in __init__
    raise ValueError("X specification includes variable(s) {}, but "
ValueError: X specification includes variable(s) ['a_comp_cor_04'], but these were not found in data matrix.

Steps to Reproduce

Identify fMRIPrep run with fewer than 5 acompcor components (or otherwise expected regressors)
Run FitLins with --drop-missing argument: singularity run -e -B ${scratch},${bids_dir} $IMG ${scratch}/${subject}_db ${scratch}/${subject}_db/derivatives/fitlins_${model} run -m $scratch/${subject}_db/models/${task}-model-${model}.json -d ${scratch}/${subject}_db/derivatives/fmriprep/ -w $scratch --participant-label ${subject:4} --space $space --desc-label preproc -s 6 --drop-missing

adelavega commented 2 years ago

I have replicated this issue as well. This must be a new bug since this flag used to work.

Thanks for the report!

adelavega commented 2 years ago

Obviously as a work around I would suggest using fewer comp cor components, but this should be a "easy" fix, so hopefully we can push it through soon

poldracklab / fitlins