DWI bids compliant but MRIQC cannot process

[dkp]

What happened?

I'm trying MRIQC on 50 subjects on the HPC using apptainer. Most subjects run fine for all three modalities at once: T1w, bold, dwi. However, some crashed.
To investigate, I ran a subject with -m T1w bold This ran fine and produced reports for those 2 modalities. Then I ran with -m dwi and this one crashed. See log below.

What command did you use?

apptainer run --cleanenv --bind ${DATA}:/data:ro --bind ${APP_DERIV_DIR}:/outputs ${APP} \
/data /outputs participant --participant_label ${Subject} -m dwi -w ${WORK_DIR} --verbose-reports

What version of the software are you running?

Running MRIQC version 24.1.0.dev0+g3fe90466.d20240417

How are you running this software?

Singularity

Is your data BIDS valid?

Yes

Are you reusing any previously computed results?

No

Please copy and paste any relevant log output.

$ sbatch --export sub=083 runmriqc_dev.sh
Submitted batch job 1968242
$ cat slurm-1968242.out 
/xdisk/eplante/dkp/plante/data should contain dataset_description.json!
------------------------------------------------------------------
  Running MRIQC version 24.1.0.dev0+g3fe90466.d20240417
  ----------------------------------------------------------------
  * BIDS dataset path: /data.
  * Output folder: /outputs.
  * Analysis levels: ['participant'].
------------------------------------------------------------------

2024-06-01 13:19:07 | IMPORTANT | mriqc            | Building MRIQC's workflows...
2024-06-01 13:19:17 | IMPORTANT | mriqc            | Workflow building finished (exit code 0).
2024-06-01 13:19:25 | WARNING  | mriqc            | IMPORTANT: Anonymized quality metrics (IQMs) will be submitted to MRIQC's metrics repository. Submission of IQMs can be disabled using the ``--no-sub`` argument. Please visit https://mriqc.readthedocs.io/en/latest/dsa.html to revise MRIQC's Data Sharing Agreement.
2024-06-01 13:19:56 | WARNING  | nipype.workflow  | Storing result file without outputs
2024-06-01 13:19:56 | WARNING  | nipype.workflow  | [Node] Error on "mriqc_wf.dwiMRIQC.ComputeIQMs.datasink" (/xdisk/eplante/dkp/plante/mriqc_work_dev/mriqc_wf/dwiMRIQC/ComputeIQMs/_in_file_..data..sub-083..dwi..sub-083_acq-AP_dwi.nii.gz/datasink)
Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/site-packages/mriqc/engine/plugin.py", line 579, in _send_procs_to_workers
    self.procs[jobid].run(updatehash=updatehash)
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node datasink.

Traceback:
        Traceback (most recent call last):
          File "/opt/conda/lib/python3.11/site-packages/nipype/interfaces/base/core.py", line 397, in run
            runtime = self._run_interface(runtime)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/mriqc/interfaces/bids.py", line 188, in _run_interface
            json.dumps(
          File "/opt/conda/lib/python3.11/site-packages/simplejson/__init__.py", line 395, in dumps
            **kw).encode(obj)
                  ^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/simplejson/encoder.py", line 300, in encode
            chunks = list(chunks)
                     ^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/simplejson/encoder.py", line 714, in _iterencode
            for chunk in _iterencode_dict(o, _current_indent_level):
          File "/opt/conda/lib/python3.11/site-packages/simplejson/encoder.py", line 645, in _iterencode_dict
            yield _floatstr(value)
                  ^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/simplejson/encoder.py", line 351, in floatstr
            raise ValueError(
        ValueError: Out of range float values are not JSON compliant: nan

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/bin/mriqc", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/mriqc/cli/run.py", line 178, in main
    mriqc_wf.run(**_plugin)
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/workflows.py", line 638, in run
    runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/opt/conda/lib/python3.11/site-packages/mriqc/engine/plugin.py", line 214, in run
    self._send_procs_to_workers(updatehash=updatehash, graph=graph)
  File "/opt/conda/lib/python3.11/site-packages/mriqc/engine/plugin.py", line 582, in _send_procs_to_workers
    self._clean_queue(
  File "/opt/conda/lib/python3.11/site-packages/mriqc/engine/plugin.py", line 259, in _clean_queue
    raise RuntimeError(''.join(result['traceback']))
RuntimeError: Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/site-packages/mriqc/engine/plugin.py", line 579, in _send_procs_to_workers
    self.procs[jobid].run(updatehash=updatehash)
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node datasink.

Traceback:
        Traceback (most recent call last):
          File "/opt/conda/lib/python3.11/site-packages/nipype/interfaces/base/core.py", line 397, in run
            runtime = self._run_interface(runtime)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/mriqc/interfaces/bids.py", line 188, in _run_interface
            json.dumps(
          File "/opt/conda/lib/python3.11/site-packages/simplejson/__init__.py", line 395, in dumps
            **kw).encode(obj)
                  ^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/simplejson/encoder.py", line 300, in encode
            chunks = list(chunks)
                     ^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/simplejson/encoder.py", line 714, in _iterencode
            for chunk in _iterencode_dict(o, _current_indent_level):
          File "/opt/conda/lib/python3.11/site-packages/simplejson/encoder.py", line 645, in _iterencode_dict
            yield _floatstr(value)
                  ^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/simplejson/encoder.py", line 351, in floatstr
            raise ValueError(
        ValueError: Out of range float values are not JSON compliant: nan

Additional information / screenshots

No response

[araikes]

@dkp Can you post the JSON or put it somewhere on the HPC I can look at it?

[dkp]

@araikes I sent you a separate email sharing the dataset.

[araikes]

@oesteban and @dkp,

The offending component (at least for the dataset shared) is the computation of EFC on the b=0 shell. It encounters an invalid value in the computation:

2024-06-03 10:17:37 | DEBUG    | indexed_gzip.indexed_gzip | _IndexedGzipFile.close()
/opt/conda/lib/python3.11/site-packages/mriqc/qc/anatomical.py:391: RuntimeWarning: invalid value encountered in sqrt
  b_max = np.sqrt((img[framemask == 0] ** 2).sum()

which results in a nan for that shell (truncating the whole line for visibility):

2024-06-03 10:17:43 | DEBUG    | nipype.workflow  | [Node] datasink - setting input root = {... 'efc_shell01': nan, ...}

And then the JSON writer can't figure out how to deal with the NaN when expecting a double.

I haven't figured out what the offending value in the image is. There don't appear to be any unusual or invalid values as far as I can tell and I can manually compute the EFC for each of the two b=0 images independently (in their raw, unprocessed state... just loading the data, creating the framemask, and computing as per the code in mriqc/qc/anatomical.py). Notably, something seems to be off (for lack of a better term) about those b=0s because FBER is computed as -4527 which doesn't feel like a plausible value either.

[dkp]

This is older data, if that helps at all. Thanks for looking into it.

[s-madhavan]

Hi,

I'm running into the same issue as above with my dwi data.

Error:

Traceback: Traceback (most recent call last): File "/opt/conda/lib/python3.11/site-packages/nipype/interfaces/base/core.py", line 397, in run runtime = self._run_interface(runtime) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/mriqc/interfaces/bids.py", line 188, in _run_interface json.dumps( File "/opt/conda/lib/python3.11/site-packages/simplejson/init.py", line 395, in dumps **kw).encode(obj) ^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/simplejson/encoder.py", line 300, in encode chunks = list(chunks) ^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/simplejson/encoder.py", line 714, in _iterencode for chunk in _iterencode_dict(o, _current_indent_level): File "/opt/conda/lib/python3.11/site-packages/simplejson/encoder.py", line 645, in _iterencode_dict yield _floatstr(value) ^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/simplejson/encoder.py", line 351, in floatstr raise ValueError( ValueError: Out of range float values are not JSON compliant: nan

Finished subject SOARTravel01 20240524CCH15

My data is new, and BIDS compliant. Going to try using an older version of MRIQC for now without DWI processing. DWI doesn't seem to be working for now.

[dkp]

You can just set the -m (modality flag): -m T1w bold so you can still use the new version, it just won't try to run dwi.

[s-madhavan]

Hi,

Noted. Will do that for anat and func files. Thanks.

--Subbi M.

[costantinoai]

I am having a similar error (see here for the crash report and error), but I do not have DWI scans.

The problem for me seems to be the 'summary_bg_k': nan entry.

Any idea from where this nan comes from? I am running mriqc on a dataset preprocessed with fMRIprep – could this be the cause?

[pradhanhitesh]

Facing the same issue with DWI images. Any update on the issues?

Thanks.

[s-madhavan]

@pradhanhitesh I'm also having the same issue. On a different note does the new version of MRIQC take forever to run for some people. I set my SBATCH time limit to be 15 hours and it crashed because the time limit expired. What version is everyone else using for now?

[dkp]

I'm now running 24.0.1: It pretty fully implements the dwi report (just some little issues). I've never seen the time issue you mention.

[pradhanhitesh]

I have the latest docker image for MRI-QC. It's true that DWI QC Reports are generated but for few DWI images, there seems to be an error (the NaN value and subsequent JSON encoding).

I'm running QC for T1w, bold and DWI for 1600 subjects for my study. When I ran the QC, most of the time it crashed. Therefore, I'm forced to run the QC in a batch of 20. In these batches of 20, there are few DWI files which are producing the NaN errors and therefore I'm not able to generate reports.

[dkp]

Interesting, I ran 50 subjects and lost about 25% of them because of the dwi report.