First post

Originally posted by @DVSneuro in https://github.com/nipreps/mriqc/issues/1087#issuecomment-2045223332

I'm getting the same IndexError with the latest pre-release for 24.0.0, but maybe it's a conflicting environment issue? I'm using --cleanenv so I thought that option would prevent that, but I am likely misunderstanding something.

Using the same data and code, I'm not getting any errors with 23.1.0. Maybe one notable feature of my dataset is that it is multiecho, and it seems like the handling for multiecho data has improved since 23.1.0. The IndexError appears starting with 23.1.1.

Happy to share data if it's helpful. It's not quite ready for OpenNeuro and public sharing, but maybe I send a few subjects privately via OneDrive or DropBox.

Thanks for any help!

Best wishes, David

Here is my code:

TEMPLATEFLOW_DIR=/ZPOOL/data/tools/templateflow
export APPTAINERENV_TEMPLATEFLOW_HOME=/opt/templateflow
/ZPOOL/data/tools/apptainer/bin/singularity run --cleanenv \
-B ${TEMPLATEFLOW_DIR}:/opt/templateflow \
-B $maindir:/base \
-B $scratchdir:/scratch \
/ZPOOL/data/tools/mriqc-24.0.0rc2.simg \
/base/bids /base/derivatives/mriqc \
participant --participant_label $sub \
--bids-filter-file /base/code/mriqc_config.json \
-w /scratch

Here is the contents of my mriqc_config.json file:

{
  "bold": {"datatype": "func", "suffix": "bold", "part": [null, "mag"]}
}

And here is the full output of the error:

------------------------------------------------------------------
  Running MRIQC version 24.1.0.dev0+g9b5c2f63.d20240408
  ----------------------------------------------------------------
  * BIDS dataset path: /base/bids.
  * Output folder: /base/derivatives/mriqc.
  * Analysis levels: ['participant'].
  * BIDS filters-file: /base/code/mriqc_config.json.------------------------------------------------------------------

Process Process-2:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/lib/python3.11/site-packages/mriqc/cli/workflow.py", line 45, in build_workflow
    config.load(config_file)
  File "/opt/conda/lib/python3.11/site-packages/mriqc/config.py", line 633, in load
    settings = loads(filename.read_text())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 511, in loads
    ret = decoder.load_line(line, currentlevel, multikey,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 778, in load_line
    value, vtype = self.load_value(pair[1], strictly_valid)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 880, in load_value
    return (self.load_array(v), "array")
            ^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 1002, in load_array
    a[b] = a[b] + ',' + a[b + 1]
                        ~^^^^^^^
IndexError: list index out of range
Traceback (most recent call last):
  File "/opt/conda/bin/mriqc", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/mriqc/cli/run.py", line 123, in main
    config.load(config_file)
  File "/opt/conda/lib/python3.11/site-packages/mriqc/config.py", line 633, in load
    settings = loads(filename.read_text())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 511, in loads
    ret = decoder.load_line(line, currentlevel, multikey,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 778, in load_line
    value, vtype = self.load_value(pair[1], strictly_valid)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 880, in load_value
    return (self.load_array(v), "array")
            ^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 1002, in load_array
    a[b] = a[b] + ',' + a[b + 1]
                        ~^^^^^^^
IndexError: list index out of range

Follow up post

Originally posted by @DVSneuro in https://github.com/nipreps/mriqc/issues/1087#issuecomment-2045432285

Thanks, @oesteban -- sorry, I wasn't sure if I should create a new issue for this error. I do get the same error with the 24.0.0rc2 version. I can also confirm that it happens without the config file being used.

I got a different error with a different subject (see below). So, just as a sanity check, I tripled checked the bids validator, and there are no errors there.

Same error, latest pre-release:

bash mriqc.sh 10003
------------------------------------------------------------------
  Running MRIQC version 24.1.0.dev0+g9b5c2f63.d20240408
  ----------------------------------------------------------------
  * BIDS dataset path: /base/bids.
  * Output folder: /base/derivatives/mriqc.
  * Analysis levels: ['participant'].
  * BIDS filters-file: /base/code/mriqc_config.json.------------------------------------------------------------------

Process Process-2:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/lib/python3.11/site-packages/mriqc/cli/workflow.py", line 45, in build_workflow
    config.load(config_file)
  File "/opt/conda/lib/python3.11/site-packages/mriqc/config.py", line 633, in load
    settings = loads(filename.read_text())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 511, in loads
    ret = decoder.load_line(line, currentlevel, multikey,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 778, in load_line
    value, vtype = self.load_value(pair[1], strictly_valid)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 880, in load_value
    return (self.load_array(v), "array")
            ^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 1002, in load_array
    a[b] = a[b] + ',' + a[b + 1]
                        ~^^^^^^^
IndexError: list index out of range
Traceback (most recent call last):
  File "/opt/conda/bin/mriqc", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/mriqc/cli/run.py", line 123, in main
    config.load(config_file)
  File "/opt/conda/lib/python3.11/site-packages/mriqc/config.py", line 633, in load
    settings = loads(filename.read_text())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 511, in loads
    ret = decoder.load_line(line, currentlevel, multikey,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 778, in load_line
    value, vtype = self.load_value(pair[1], strictly_valid)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 880, in load_value
    return (self.load_array(v), "array")
            ^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 1002, in load_array
    a[b] = a[b] + ',' + a[b + 1]
                        ~^^^^^^^
IndexError: list index out of range

Different subject, new error:

------------------------------------------------------------------
  Running MRIQC version 24.1.0.dev0+g9b5c2f63.d20240408
  ----------------------------------------------------------------
  * BIDS dataset path: /base/bids.
  * Output folder: /base/derivatives/mriqc.
  * Analysis levels: ['participant'].
------------------------------------------------------------------

Process Process-2:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 511, in loads
    ret = decoder.load_line(line, currentlevel, multikey,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 778, in load_line
    value, vtype = self.load_value(pair[1], strictly_valid)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 880, in load_value
    return (self.load_array(v), "array")
            ^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 1029, in load_array
    raise ValueError("Not a homogeneous array")
ValueError: Not a homogeneous array

Data

@DVSneuro posted a MWE on openneuro: https://openneuro.org/datasets/ds005085

oesteban commented 2 months ago

Okay let's dissect this:

Environment leaking: it does look like your environment is not well isolated because in both runs (first post and follow-up) you mention you are using RCs, but the initial banner says otherwise:
```
------------------------------------------------------------------
 Running MRIQC version 24.1.0.dev0+g9b5c2f63.d20240408
 ----------------------------------------------------------------
 * BIDS dataset path: /base/bids.
 * Output folder: /base/derivatives/mriqc.
 * Analysis levels: ['participant'].
------------------------------------------------------------------
```
Indeed, the stated version is 24.1.0.dev0+g9b5c2f63.d20240408 in both runs when it should be 24.0.0rc1 and 24.0.0rc2. Since you have the same version on both, I'm unsure whether you are executing different versions. However, this does not seem to explain the error.
BIDS-filter file: Then I thought the choke point was the BIDS-filter file. In particular, it seems that having "part": [null, "mag"] could be the culprit, in collaboration with this problem of toml: uiri/toml#270. Looks like your filter file is trying to (i) select functional data only and (ii) include only magnitude files (either defining part- or a single file without phase). You can totally remove the BIDS-filter, and add -m bold to your command line.
What the heck?: I was happily believing I had discovered the problem, but then your "Different subject, new error" traceback shows that you ran without bids-filter file.

Before replicating myself, can I ask you for a last piece of information? Can you please copy and paste the contents of your config file? The file should be named something like: $scratchdir/.mriqc.<some-random-string>.toml

DVSneuro commented 2 months ago

Thanks, @oesteban -- I'm also confused by the version number that is MRIQC outputs, and I've noticed that discrepancy for a few versions now. I'm not sure what it means for the environment to be leaking. For reference, here's how I built the container:

singularity build mriqc-24.0.0rc6.simg docker://nipreps/mriqc:24.0.0rc6

Also, here's the contents of the config file in scratch directory:

[environment]
cpu_count = 48
exec_env = "singularity"
free_mem = 36.5
freesurfer_home = "/opt/freesurfer"
overcommit_policy = "heuristic"
overcommit_limit = "50%"
nipype_version = "1.8.6"
synthstrip_path = "PosixPath('/opt/freesurfer/models/synthstrip.1.pt')"
templateflow_version = "24.2.0"
total_memory = 125.58418655395508
version = "24.1.0.dev0+g4d90e96b.d20240411"

[execution]
ants_float = false
bids_dir = "/base"
bids_database_dir = "/scratch/.bids_db-20240412-004639_e02641df-e373-46bb-b565-f8ba543e5367"
bids_database_wipe = false
cwd = "/home/tug87422@tu.temple.edu"
debug = false
dry_run = false
dsname = "<unset>"
float32 = true
layout = "BIDS Layout: /base"
log_dir = "/base/derivatives/mriqc/logs"
log_level = 25
modalities = [ "T1w", "T2w", "bold", "dwi",]
no_sub = false
notrack = false
output_dir = "/base/derivatives/mriqc"
participant_label = [ "10043",]
pdb = false
reports_only = false
resource_monitor = false
run_uuid = "20240412-004639_e02641df-e373-46bb-b565-f8ba543e5367"
templateflow_home = "/templateflow"
upload_strict = false
verbose_reports = false
webapi_url = "https://mriqc.nimh.nih.gov:443/api/v1"
work_dir = "/scratch"
write_graph = false

[workflow]
analysis_level = [ "participant",]
biggest_file_gb = 2.648588978346189e-10
deoblique = false
despike = false
fd_thres = 0.2
fd_radius = 50
fft_spikes_detector = false
min_len_dwi = 7
min_len_bold = 5
species = "human"
template_id = "MNI152NLin2009cAsym"

[nipype]
crashfile_format = "txt"
get_linked_libs = false
local_hash_check = true
nprocs = 48
omp_nthreads = 1
plugin = "MultiProc"
remove_node_directories = false
resource_monitor = false
stop_on_first_crash = true

[settings]
file_path = "/scratch/config-20240412-004639_e02641df-e373-46bb-b565-f8ba543e5367.toml"
start_time = 1712897199.605464

[execution.bids_filters]

[workflow.inputs]
t1w = [ "/base/sub-10043/anat/sub-10043_T1w.nii.gz",]
bold = [ "/base/sub-10043/func/sub-10043_task-sharedreward_acq-mb1me1_bold.nii.gz", [ "/base/sub-10043/func/sub-10043_task-sharedreward_acq-mb1me4_echo-1_part-mag_bold.nii.gz", "/base/sub-10043/func/sub-10043_task-sharedreward_acq-mb1me4_echo-2_part-mag_bold.nii.gz", "/base/sub-10043/func/sub-10043_task-sharedreward_acq-mb1me4_echo-3_part-mag_bold.nii.gz", "/base/sub-10043/func/sub-10043_task-sharedreward_acq-mb1me4_echo-4_part-mag_bold.nii.gz",], "/base/sub-10043/func/sub-10043_task-sharedreward_acq-mb3me1_bold.nii.gz", [ "/base/sub-10043/func/sub-10043_task-sharedreward_acq-mb3me4_echo-1_part-mag_bold.nii.gz", "/base/sub-10043/func/sub-10043_task-sharedreward_acq-mb3me4_echo-2_part-mag_bold.nii.gz", "/base/sub-10043/func/sub-10043_task-sharedreward_acq-mb3me4_echo-3_part-mag_bold.nii.gz", "/base/sub-10043/func/sub-10043_task-sharedreward_acq-mb3me4_echo-4_part-mag_bold.nii.gz",], "/base/sub-10043/func/sub-10043_task-sharedreward_acq-mb6me1_bold.nii.gz", [ "/base/sub-10043/func/sub-10043_task-sharedreward_acq-mb6me4_echo-1_part-mag_bold.nii.gz", "/base/sub-10043/func/sub-10043_task-sharedreward_acq-mb6me4_echo-2_part-mag_bold.nii.gz", "/base/sub-10043/func/sub-10043_task-sharedreward_acq-mb6me4_echo-3_part-mag_bold.nii.gz", "/base/sub-10043/func/sub-10043_task-sharedreward_acq-mb6me4_echo-4_part-mag_bold.nii.gz",],]

[nipype.plugin_args]
maxtasksperchild = 1
raise_insufficient = false

oesteban commented 2 months ago

I suspect this comes from mixing multi-echo and single-echo inputs - I'll check tomorrow and submit a fix.

DVSneuro commented 2 months ago

Thanks, Oscar! That sounds like a possibility given where it seems to break in these data (e.g., sub-10015 doesn't have multiecho and MRIQC works).

oesteban commented 2 months ago

Confirmed - the problem is not because some runs are multi-echo. The problem comes from having both (single-echo and multi-echo) in a single run.

nipreps / mriqc

IndexError in the initial steps of the participant level when saving the config file #1264

First post

Follow up post

Data