search

nipreps / mriqc

Automated Quality Control and visual reports for Quality Assessment of structural (T1w, T2w) and functional MRI of the brain
http://mriqc.readthedocs.io
Apache License 2.0
297 stars 132 forks source link

Error with metric upload to API #1319

Closed mckenziephagen closed 1 month ago

mckenziephagen commented 2 months ago

What happened?

MRIQC fails when uploading IQMs to API.

Error log: ``` /opt/conda/lib/python3.11/site-packages/dipy/denoise/noise_estimate.py:252: RuntimeWarning: divide by zero encountered in divide s = sum_m2 / (2 * K * sigma**2) /opt/conda/lib/python3.11/site-packages/dipy/denoise/noise_estimate.py:252: RuntimeWarning: invalid value encountered in divide s = sum_m2 / (2 * K * sigma**2) /opt/conda/lib/python3.11/site-packages/nireports/interfaces/reporting/base.py:95: UserWarning: The given float value must not exceed 0.0. But, you have given threshold=0.001. mask_nii = threshold_img(fixed_image_nii, 1e-3) /opt/conda/lib/python3.11/site-packages/mriqc/interfaces/anatomical.py:490: RuntimeWarning: divide by zero encountered in divide bg_data[bg_data > 0] = bg_data[bg_data > 0] / bg_spread /opt/conda/lib/python3.11/site-packages/nilearn/plotting/displays/_slicers.py:420: UserWarning: empty mask xmin_, xmax_, ymin_, ymax_, zmin_, zmax_ = get_mask_bounds( Traceback (most recent call last): File "/opt/conda/bin/mriqc", line 8, in sys.exit(main()) ^^^^^^ File "/opt/conda/lib/python3.11/site-packages/mriqc/cli/run.py", line 178, in main mriqc_wf.run(**_plugin) File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/workflows.py", line 638, in run runner.run(execgraph, updatehash=updatehash, config=self.config) File "/opt/conda/lib/python3.11/site-packages/mriqc/engine/plugin.py", line 196, in run notrun.append(self._clean_queue(jobid, graph, result=result)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/mriqc/engine/plugin.py", line 259, in _clean_queue raise RuntimeError(''.join(result['traceback'])) RuntimeError: Traceback (most recent call last): File "/opt/conda/lib/python3.11/site-packages/mriqc/engine/plugin.py", line 64, in run_node result['result'] = node.run(updatehash=updatehash) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run result = self._run_interface(execute=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 1380, in _run_interface result = self._collate_results( ^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 1249, in _collate_results for i, nresult, err in nodes: File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/utils.py", line 94, in nodelist_runner result = node.run(updatehash=updatehash) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run result = self._run_interface(execute=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface return self._run_command(execute) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command raise NodeExecutionError(msg) nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node _UploadMetrics0. Traceback: Traceback (most recent call last): File "/opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py", line 537, in _make_request response = conn.getresponse() ^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/urllib3/connection.py", line 466, in getresponse httplib_response = super().getresponse() ^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/http/client.py", line 1390, in getresponse response.begin() File "/opt/conda/lib/python3.11/http/client.py", line 325, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/http/client.py", line 286, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/socket.py", line 706, in readinto return self._sock.recv_into(b) ^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/ssl.py", line 1314, in recv_into return self.read(nbytes, buffer) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/ssl.py", line 1166, in read return self._sslobj.read(len, buffer) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ TimeoutError: The read operation timed out The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/opt/conda/lib/python3.11/site-packages/requests/adapters.py", line 486, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py", line 847, in urlopen retries = retries.increment( ^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/urllib3/util/retry.py", line 470, in increment raise reraise(type(error), error, _stacktrace) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/urllib3/util/util.py", line 39, in reraise raise value File "/opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py", line 793, in urlopen response = self._make_request( ^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py", line 539, in _make_request self._raise_timeout(err=e, url=url, timeout_value=read_timeout) File "/opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py", line 370, in _raise_timeout raise ReadTimeoutError( urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='mriqc.nimh.nih.gov', port=443): Read timed out. (read timeout=15) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/conda/lib/python3.11/site-packages/nipype/interfaces/base/core.py", line 397, in run runtime = self._run_interface(runtime) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/mriqc/interfaces/webapi.py", line 141, in _run_interface response = upload_qc_metrics( ^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/mriqc/interfaces/webapi.py", line 244, in upload_qc_metrics response = requests.post( ^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/requests/api.py", line 115, in post return request("post", url, data=data, json=json, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/requests/api.py", line 59, in request return session.request(method=method, url=url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/requests/adapters.py", line 532, in send raise ReadTimeout(e, request=request) requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='mriqc.nimh.nih.gov', port=443): Read timed out. (read timeout=15) ```

Running with --no-sub fixes this.

What command did you use?

singularity run -e mriqc-24.0.0.simg ds002785 ds002785/derivatives/mriqc-24.0.0 participant --participant-label sub-0029 -w /lscratch/mphagen/work/ --omp-nthreads 8 --mem 10 --verbose-reports

What version of the software are you running?

24

How are you running this software?

Singularity

Is your data BIDS valid?

Yes

Are you reusing any previously computed results?

No

Please copy and paste any relevant log output.

Full output log ``` Subject sub-0029 Running task 30 Commandline: singularity run -e mriqc-24.0.0.simg ds002785 ds002785/derivatives/mriqc-24.0.0 participant --participant-label sub-0029 -w /lscratch/mphagen/work/ - -omp-nthreads 8 --mem 10 --verbose-reports ------------------------------------------------------------------ Running MRIQC version 24.1.0.dev0+g3fe90466.d20240417 ---------------------------------------------------------------- * BIDS dataset path: /scratch/users/mphagen/mriqc-protocol/ds002785. * Output folder: ds002785/derivatives/mriqc-24.0.0. * Analysis levels: ['participant']. ------------------------------------------------------------------ 2024-08-14 11:00:31 | IMPORTANT | mriqc | Building MRIQC's workflows... 2024-08-14 11:00:32 | IMPORTANT | mriqc | DataLad dataset identified, attempting to `datalad get` unavailable files. action summary: get (notneeded: 6) 2024-08-14 11:00:34 | IMPORTANT | mriqc | DataLad dataset identified, attempting to `datalad get` unavailable files. 2024-08-14 11:00:35 | IMPORTANT | mriqc | DataLad dataset identified, attempting to `datalad get` unavailable files. 2024-08-14 11:00:41 | IMPORTANT | mriqc | Workflow building finished (exit code 0). 2024-08-14 11:00:47 | WARNING | mriqc | IMPORTANT: Anonymized quality metrics (IQMs) will be submitted to MRIQC's metrics repository. Submission of IQ Ms can be disabled using the ``--no-sub`` argument. Please visit https://mriqc.readthedocs.io/en/latest/dsa.html to revise MRIQC's Data Sharing Agreement. 2024-08-14 11:01:36 | WARNING | nipype.interface | Insufficient number of low-b orientations (1) to safely calculate signal drift. 2024-08-14 11:01:38 | INFO | niworkflows | Estimating initial transform using AffineInitializer 2024-08-14 11:01:45 | INFO | niworkflows | Terminal outputs of initialization saved (/lscratch/mphagen/work/mriqc_wf/dwiMRIQC/SpatialNormalization/_in_fi le_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..dwi..sub-0029_dwi.nii.gz/EPI2MNI/stderr.nipype-init). 2024-08-14 11:01:45 | INFO | niworkflows | Loading settings from file /opt/conda/lib/python3.11/site-packages/niworkflows/data/boldref-mni_registration_p recise_000.json. 2024-08-14 11:01:45 | INFO | niworkflows | Retry #1, commandline: antsRegistration --collapse-output-transforms 1 --dimensionality 3 --float 0 --initial-moving-transform [ /lscratch/mphagen/work/mriqc_wf/dwiMRIQC/SpatialNormaliz ation/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..dwi..sub-0029_dwi.nii.gz/EPI2MNI/transform.mat, 0 ] --initialize-transforms-per-stag e 0 --interpolation LanczosWindowedSinc --output [ epi_to_mni, epi_to_mni_Warped.nii.gz ] --transform Rigid[ 0.05 ] --metric Mattes[ /templateflow/tpl-MNI152NLin2 009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-fMRIPrep_boldref.nii.gz, /lscratch/mphagen/work/mriqc_wf/dwiMRIQC/SpatialNormalization/_in_file_..scratch..users..mph agen..mriqc-protocol..ds002785..sub-0029..dwi..sub-0029_dwi.nii.gz/SharpenEPI/sub-0029_dwi_valid_average_corrected.nii.gz, 1, 56, Regular, 0.25 ] --convergence [ 10000x1000x100, 1e-06, 20 ] --smoothing-sigmas 4.0x2.0x1.0vox --shrink-factors 4x2x1 --use-histogram-matching 1 --masks [ /templateflow/tpl-MNI152NLin2009cAsym/tp l-MNI152NLin2009cAsym_res-02_desc-brain_mask.nii.gz, /lscratch/mphagen/work/mriqc_wf/dwiMRIQC/synthstrip_wf/_in_file_..scratch..users..mphagen..mriqc-protocol..ds 002785..sub-0029..dwi..sub-0029_dwi.nii.gz/synthstrip/clipped_corrected_desc-brain_mask.nii.gz ] --transform Affine[ 0.08 ] --metric Mattes[ /templateflow/tpl-MNI 152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-fMRIPrep_boldref.nii.gz, /lscratch/mphagen/work/mriqc_wf/dwiMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..dwi..sub-0029_dwi.nii.gz/SharpenEPI/sub-0029_dwi_valid_average_corrected.nii.gz, 1, 56, Regular, 0.25 ] --convergence [ 500x250x100, 1e-06, 20 ] --smoothing-sigmas 4.0x2.0x1.0vox --shrink-factors 8x4x2 --use-histogram-matching 1 --masks [ /templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-brain_mask.nii.gz, /lscratch/mphagen/work/mriqc_wf/dwiMRIQC/synthstrip_wf/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..dwi..sub-0029_dwi.nii.gz/synthstrip/clipped_corrected_desc-brain_mask.nii.gz ] --transform SyN[ 0.1, 3.0, 0.0 ] --metric CC[ /templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-fMRIPrep_boldref.nii.gz, /lscratch/mphagen/work/mriqc_wf/dwiMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..dwi..sub-0029_dwi.nii.gz/SharpenEPI/sub-0029_dwi_valid_average_corrected.nii.gz, 1, 4, None, 1 ] --convergence [ 100x30x20, 1e-06, 10 ] --smoothing-sigmas 3.0x2.0x1.0vox --shrink-factors 8x4x2 --use-histogram-matching 1 --masks [ /templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-brain_mask.nii.gz, /lscratch/mphagen/work/mriqc_wf/dwiMRIQC/synthstrip_wf/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..dwi..sub-0029_dwi.nii.gz/synthstrip/clipped_corrected_desc-brain_mask.nii.gz ] --winsorize-image-intensities [ 0.005, 0.995 ] --write-composite-transform 1 2024-08-14 11:02:04 | INFO | niworkflows | Successful spatial normalization (retry #1). 2024-08-14 11:02:04 | INFO | niworkflows | Report - setting fixed (/templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-fMRIPrep_boldref.nii.gz) and moving (/lscratch/mphagen/work/mriqc_wf/dwiMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..dwi..sub-0029_dwi.nii.gz/EPI2MNI/epi_to_mni_Warped.nii.gz) images 2024-08-14 11:02:04 | INFO | niworkflows | Generating visual report 2024-08-14 11:02:51 | INFO | niworkflows | Estimating initial transform using AffineInitializer 2024-08-14 11:03:07 | INFO | niworkflows | Terminal outputs of initialization saved (/lscratch/mphagen/work/mriqc_wf/anatMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..anat..sub-0029_T1w.nii.gz/SpatialNormalization/stderr.nipype-init). 2024-08-14 11:03:07 | INFO | niworkflows | Loading settings from file /opt/conda/lib/python3.11/site-packages/niworkflows/data/t1w-mni_registration_testing_000.json. 2024-08-14 11:03:07 | INFO | niworkflows | Retry #1, commandline: antsRegistration --collapse-output-transforms 1 --dimensionality 3 --float 0 --initial-moving-transform [ /lscratch/mphagen/work/mriqc_wf/anatMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..anat..sub-0029_T1w.nii.gz/SpatialNormalization/transform.mat, 0 ] --initialize-transforms-per-stage 0 --interpolation LanczosWindowedSinc --output [ ants_t1_to_mni, ants_t1_to_mni_Warped.nii.gz ] --transform Rigid[ 1.0 ] --metric Mattes[ /lscratch/mphagen/work/mriqc_wf/anatMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..anat..sub-0029_T1w.nii.gz/SpatialNormalization/fixed_masked.nii.gz, /lscratch/mphagen/work/mriqc_wf/anatMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..anat..sub-0029_T1w.nii.gz/SpatialNormalization/moving_masked.nii.gz, 1, 56, Random, 0.2 ] --convergence [ 20, 1e-07, 15 ] --smoothing-sigmas 4.0vox --shrink-factors 2 --use-histogram-matching 0 --transform Affine[ 1.0 ] --metric Mattes[ /lscratch/mphagen/work/mriqc_wf/anatMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..anat..sub-0029_T1w.nii.gz/SpatialNormalization/fixed_masked.nii.gz, /lscratch/mphagen/work/mriqc_wf/anatMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..anat..sub-0029_T1w.nii.gz/SpatialNormalization/moving_masked.nii.gz, 1, 56, Random, 0.1 ] --convergence [ 15, 1e-08, 5 ] --smoothing-sigmas 2.0vox --shrink-factors 1 --use-histogram-matching 1 --winsorize-image-intensities [ 0.005, 0.995 ] --write-composite-transform 1 2024-08-14 11:03:12 | INFO | niworkflows | Successful spatial normalization (retry #1). 2024-08-14 11:03:14 | INFO | niworkflows | Report - setting fixed (/lscratch/mphagen/work/mriqc_wf/anatMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..anat..sub-0029_T1w.nii.gz/SpatialNormalization/fixed_masked.nii.gz) and moving (/lscratch/mphagen/work/mriqc_wf/anatMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..anat..sub-0029_T1w.nii.gz/SpatialNormalization/ants_t1_to_mni_Warped.nii.gz) images 2024-08-14 11:03:14 | INFO | niworkflows | Generating visual report 2024-08-14 11:05:26 | WARNING | nipype.interface | Estimated signal variation in the background was too small (MAD=0.0, sigma=0.5239) 2024-08-14 11:05:26 | WARNING | nipype.interface | Estimated signal variation in the background was too small (MAD=0.0, sigma=0.5239) 2024-08-14 11:05:26 | WARNING | nipype.interface | Estimated signal variation in the background was too small (MAD=0.0, sigma=0.5239) 2024-08-14 11:06:12 | INFO | niworkflows | Estimating initial transform using AffineInitializer 2024-08-14 11:06:16 | INFO | niworkflows | Estimating initial transform using AffineInitializer 2024-08-14 11:06:17 | WARNING | nipype.workflow | Storing result file without outputs 2024-08-14 11:06:17 | WARNING | nipype.workflow | [Node] Error on "_UploadMetrics0" (/lscratch/mphagen/work/mriqc_wf/funcMRIQC/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..func..sub-0029_task-emomatching_acq-seq_bold.nii.gz/UploadMetrics/mapflow/_UploadMetrics0) 2024-08-14 11:06:17 | WARNING | nipype.workflow | Storing result file without outputs 2024-08-14 11:06:17 | WARNING | nipype.workflow | [Node] Error on "mriqc_wf.funcMRIQC.UploadMetrics" (/lscratch/mphagen/work/mriqc_wf/funcMRIQC/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..func..sub-0029_task-emomatching_acq-seq_bold.nii.gz/UploadMetrics) 2024-08-14 11:06:17 | ERROR | nipype.workflow | Node UploadMetrics.a1 (taskid=150) crashed: /scratch/users/mphagen/mriqc-protocol/ds002785/derivatives/mriqc-24.0.0/logs/crash-20240814-110617-mphagen-UploadMetrics.a1-9b22ba04-6602-4b06-abe0-9ba3f380cc26.txt 2024-08-14 11:06:20 | INFO | niworkflows | Loading settings from file /opt/conda/lib/python3.11/site-packages/niworkflows/data/boldref-mni_registration_precise_000.json. 2024-08-14 11:06:20 | INFO | niworkflows | Retry #1, commandline: antsRegistration --collapse-output-transforms 1 --dimensionality 3 --float 0 --initial-moving-transform [ /lscratch/mphagen/work/mriqc_wf/funcMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..func..sub-0029_task-emomatching_acq-seq_bold.nii.gz/EPI2MNI/transform.mat, 0 ] --initialize-transforms-per-stage 0 --interpolation LanczosWindowedSinc --output [ epi_to_mni, epi_to_mni_Warped.nii.gz ] --transform Rigid[ 0.05 ] --metric Mattes[ /templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-fMRIPrep_boldref.nii.gz, /lscratch/mphagen/work/mriqc_wf/funcMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..func..sub-0029_task-emomatching_acq-seq_bold.nii.gz/SharpenEPI/sub-0029_task-emomatching_acq-seq_bold_desc-realigned_valid_tstat_corrected.nii.gz, 1, 56, Regular, 0.25 ] --convergence [ 10000x1000x100, 1e-06, 20 ] --smoothing-sigmas 4.0x2.0x1.0vox --shrink-factors 4x2x1 --use-histogram-matching 1 --masks [ /templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-brain_mask.nii.gz, /lscratch/mphagen/work/mriqc_wf/funcMRIQC/synthstrip_wf/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..func..sub-0029_task-emomatching_acq-seq_bold.nii.gz/synthstrip/clipped_corrected_desc-brain_mask.nii.gz ] --transform Affine[ 0.08 ] --metric Mattes[ /templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-fMRIPrep_boldref.nii.gz, /lscratch/mphagen/work/mriqc_wf/funcMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..func..sub-0029_task-emomatching_acq-seq_bold.nii.gz/SharpenEPI/sub-0029_task-emomatching_acq-seq_bold_desc-realigned_valid_tstat_corrected.nii.gz, 1, 56, Regular, 0.25 ] --convergence [ 500x250x100, 1e-06, 20 ] --smoothing-sigmas 4.0x2.0x1.0vox --shrink-factors 8x4x2 --use-histogram-matching 1 --masks [ /templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-brain_mask.nii.gz, /lscratch/mphagen/work/mriqc_wf/funcMRIQC/synthstrip_wf/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..func..sub-0029_task-emomatching_acq-seq_bold.nii.gz/synthstrip/clipped_corrected_desc-brain_mask.nii.gz ] --transform SyN[ 0.1, 3.0, 0.0 ] --metric CC[ /templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-fMRIPrep_boldref.nii.gz, /lscratch/mphagen/work/mriqc_wf/funcMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..func..sub-0029_task-emomatching_acq-seq_bold.nii.gz/SharpenEPI/sub-0029_task-emomatching_acq-seq_bold_desc-realigned_valid_tstat_corrected.nii.gz, 1, 4, None, 1 ] --convergence [ 100x30x20, 1e-06, 10 ] --smoothing-sigmas 3.0x2.0x1.0vox --shrink-factors 8x4x2 --use-histogram-matching 1 --masks [ /templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-brain_mask.nii.gz, /lscratch/mphagen/work/mriqc_wf/funcMRIQC/synthstrip_wf/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..func..sub-0029_task-emomatching_acq-seq_bold.nii.gz/synthstrip/clipped_corrected_desc-brain_mask.nii.gz ] --winsorize-image-intensities [ 0.005, 0.995 ] --write-composite-transform 1 2024-08-14 11:06:25 | INFO | niworkflows | Terminal outputs of initialization saved (/lscratch/mphagen/work/mriqc_wf/funcMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..func..sub-0029_task-workingmemory_acq-seq_bold.nii.gz/EPI2MNI/stderr.nipype-init). 2024-08-14 11:06:25 | INFO | niworkflows | Loading settings from file /opt/conda/lib/python3.11/site-packages/niworkflows/data/boldref-mni_registration_precise_000.json. 2024-08-14 11:06:25 | INFO | niworkflows | Retry #1, commandline: antsRegistration --collapse-output-transforms 1 --dimensionality 3 --float 0 --initial-moving-transform [ /lscratch/mphagen/work/mriqc_wf/funcMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..func..sub-0029_task-workingmemory_acq-seq_bold.nii.gz/EPI2MNI/transform.mat, 0 ] --initialize-transforms-per-stage 0 --interpolation LanczosWindowedSinc --output [ epi_to_mni, epi_to_mni_Warped.nii.gz ] --transform Rigid[ 0.05 ] --metric Mattes[ /templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-fMRIPrep_boldref.nii.gz, /lscratch/mphagen/work/mriqc_wf/funcMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..func..sub-0029_task-workingmemory_acq-seq_bold.nii.gz/SharpenEPI/sub-0029_task-workingmemory_acq-seq_bold_desc-realigned_valid_tstat_corrected.nii.gz, 1, 56, Regular, 0.25 ] --convergence [ 10000x1000x100, 1e-06, 20 ] --smoothing-sigmas 4.0x2.0x1.0vox --shrink-factors 4x2x1 --use-histogram-matching 1 --masks [ /templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-brain_mask.nii.gz, /lscratch/mphagen/work/mriqc_wf/funcMRIQC/synthstrip_wf/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..func..sub-0029_task-workingmemory_acq-seq_bold.nii.gz/synthstrip/clipped_corrected_desc-brain_mask.nii.gz ] --transform Affine[ 0.08 ] --metric Mattes[ /templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-fMRIPrep_boldref.nii.gz, /lscratch/mphagen/work/mriqc_wf/funcMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..func..sub-0029_task-workingmemory_acq-seq_bold.nii.gz/SharpenEPI/sub-0029_task-workingmemory_acq-seq_bold_desc-realigned_valid_tstat_corrected.nii.gz, 1, 56, Regular, 0.25 ] --convergence [ 500x250x100, 1e-06, 20 ] --smoothing-sigmas 4.0x2.0x1.0vox --shrink-factors 8x4x2 --use-histogram-matching 1 --masks [ /templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-brain_mask.nii.gz, /lscratch/mphagen/work/mriqc_wf/funcMRIQC/synthstrip_wf/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..func..sub-0029_task-workingmemory_acq-seq_bold.nii.gz/synthstrip/clipped_corrected_desc-brain_mask.nii.gz ] --transform SyN[ 0.1, 3.0, 0.0 ] --metric CC[ /templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-fMRIPrep_boldref.nii.gz, /lscratch/mphagen/work/mriqc_wf/funcMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..func..sub-0029_task-workingmemory_acq-seq_bold.nii.gz/SharpenEPI/sub-0029_task-workingmemory_acq-seq_bold_desc-realigned_valid_tstat_corrected.nii.gz, 1, 4, None, 1 ] --convergence [ 100x30x20, 1e-06, 10 ] --smoothing-sigmas 3.0x2.0x1.0vox --shrink-factors 8x4x2 --use-histogram-matching 1 --masks [ /templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-brain_mask.nii.gz, /lscratch/mphagen/work/mriqc_wf/funcMRIQC/synthstrip_wf/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..func..sub-0029_task-workingmemory_acq-seq_bold.nii.gz/synthstrip/clipped_corrected_desc-brain_mask.nii.gz ] --winsorize-image-intensities [ 0.005, 0.995 ] --write-composite-transform 1 2024-08-14 11:06:40 | INFO | niworkflows | Successful spatial normalization (retry #1). 2024-08-14 11:06:40 | INFO | niworkflows | Report - setting fixed (/templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-fMRIPrep_boldref.nii.gz) and moving (/lscratch/mphagen/work/mriqc_wf/funcMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..func..sub-0029_task-emomatching_acq-seq_bold.nii.gz/EPI2MNI/epi_to_mni_Warped.nii.gz) images 2024-08-14 11:06:40 | INFO | niworkflows | Generating visual report 2024-08-14 11:06:45 | INFO | niworkflows | Successful spatial normalization (retry #1). 2024-08-14 11:06:45 | INFO | niworkflows | Report - setting fixed (/templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-fMRIPrep_boldref.nii.gz) and moving (/lscratch/mphagen/work/mriqc_wf/funcMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..func..sub-0029_task-workingmemory_acq-seq_bold.nii.gz/EPI2MNI/epi_to_mni_Warped.nii.gz) images 2024-08-14 11:06:45 | INFO | niworkflows | Generating visual report Finished tasks 30 with exit code 1 ```

Additional information / screenshots

No response

oesteban commented 2 months ago

Thanks!

Can you paste the contents of /scratch/users/mphagen/mriqc-protocol/ds002785/derivatives/mriqc-24.0.0/logs/crash-20240814-110617-mphagen-UploadMetrics.a1-9b22ba04-6602-4b06-abe0-9ba3f380cc26.txt? This is the 'crashfile' corresponding to the particular error.

However, the workflow crashed for another reason -- the metrics were not uploaded, that's clear, but the workflow will go on unless you are running on 'strict' mode (and you are not). So there must be another error under /scratch/users/mphagen/mriqc-protocol/ds002785/derivatives/mriqc-24.0.0/logs/

jbwexler commented 1 month ago

I just ran a bunch of datasets with 24.0.1 and got the above error for all of them. Perhaps the internet was down at the time? From what I can tell, there were no other errors and I'm not running in strict mode:

(main) login4.frontera(1051)$ ls /scratch1/03201/jbwexler/openneuro_derivatives/derivatives/mriqc/ds004478-mriqc/logs/
crash-20240904-092631-jbwexler-UploadMetrics.a0-e9a2ac19-0251-4d7b-acba-3de08a1dd9ae.txt
crash-20240904-092634-jbwexler-UploadMetrics.a0-7ab1dd29-225d-4af7-83bd-dbaf8c8fb2a1.txt
crash-20240904-092634-jbwexler-UploadMetrics.a0-c8c9e83e-560b-4c94-b815-ee997cb78c4b.txt
crash-20240904-092644-jbwexler-UploadMetrics.a0-2d3d5279-dfd1-44f8-9311-9d86e8c55f21.txt
crash-20240904-092714-jbwexler-UploadMetrics.a0-193566fd-bdf4-4ef1-a4b4-3c460449d393.txt
crash-20240904-092718-jbwexler-UploadMetrics.a0-e79a29d4-8979-4af0-b91d-9f011cbed8a3.txt
mriqc-20240904-092224_a57ba336-81bb-45bb-b2d9-3858aa74ca17.log
mriqc-20240904-092225_66f5102a-eb8c-4c2e-b3f3-f143ea15ca3a.log
mriqc-20240904-092228_f8df30d4-4022-43e9-81b6-d6a7dea20837.log
mriqc-20240904-092230_b7475734-3be8-46f8-8721-1c200776456e.log
mriqc-20240904-092231_8f8ac21a-85f6-4e07-b3fb-98d74d71cbf3.log
mriqc-20240904-092236_342684f6-b222-48fb-bee7-3135c537be65.log

Here's an example command:

code/containers/scripts/singularity_cmd run --bind /tmp:/node_tmp --bind /scratch1/03201/jbwexler/mriqc_home:/home/mriqc code/containers/images/bids/bids-mriqc--24.0.1.sing sourcedata/raw /scratch1/03201/jbwexler/openneuro_derivatives/derivatives/mriqc/ds004478-mriqc participant --participant-label '01' -w '/node_tmp/work_dir/mriqc/ds004478_sub-01' -vv --nprocs 11 --ants-nthreads 8 --verbose-reports --dsname ds004478 --mem_gb 30
oesteban commented 1 month ago

Can you paste the content of one of those crash files?

jbwexler commented 1 month ago

Yep, here's an example:

Node: mriqc_wf.anatMRIQC.UploadMetrics
Working directory: /node_tmp/work_dir/mriqc/ds004478_sub-01/mriqc_wf/anatMRIQC/a63125fa58866c73d5b1e2e2859283294faa562b/UploadMetrics

Node inputs:

auth_token = 8sSYVI0XjFqacEMZ8wF4
email = <undefined>
endpoint = https://mriqc.nimh.nih.gov:443/api/v1
in_iqms = /scratch1/03201/jbwexler/openneuro_derivatives/derivatives/mriqc/ds004478-mriqc/sub-01/anat/sub-01_T1w.json
modality = undefined
strict = False

Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/site-packages/mriqc/engine/plugin.py", line 64, in run_node
    result['result'] = node.run(updatehash=updatehash)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node UploadMetrics.

Traceback:
        Traceback (most recent call last):
          File "/opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py", line 537, in _make_request
            response = conn.getresponse()
                       ^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/urllib3/connection.py", line 466, in getresponse
            httplib_response = super().getresponse()
                               ^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/http/client.py", line 1390, in getresponse
            response.begin()
          File "/opt/conda/lib/python3.11/http/client.py", line 325, in begin
            version, status, reason = self._read_status()
                                      ^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/http/client.py", line 286, in _read_status
            line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/socket.py", line 706, in readinto
            return self._sock.recv_into(b)
                   ^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/ssl.py", line 1314, in recv_into
            return self.read(nbytes, buffer)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/ssl.py", line 1166, in read
            return self._sslobj.read(len, buffer)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        TimeoutError: The read operation timed out

        The above exception was the direct cause of the following exception:

        Traceback (most recent call last):
          File "/opt/conda/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
            resp = conn.urlopen(
                   ^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py", line 847, in urlopen
            retries = retries.increment(
                      ^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/urllib3/util/retry.py", line 470, in increment
            raise reraise(type(error), error, _stacktrace)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/urllib3/util/util.py", line 39, in reraise
            raise value
          File "/opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py", line 793, in urlopen
            response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py", line 539, in _make_request
            self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
          File "/opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py", line 370, in _raise_timeout
            raise ReadTimeoutError(
        urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='mriqc.nimh.nih.gov', port=443): Read timed out. (read timeout=15)

        During handling of the above exception, another exception occurred:

        Traceback (most recent call last):
          File "/opt/conda/lib/python3.11/site-packages/nipype/interfaces/base/core.py", line 397, in run
            runtime = self._run_interface(runtime)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/mriqc/interfaces/webapi.py", line 148, in _run_interface
            response, payload = upload_qc_metrics(
                                ^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/mriqc/interfaces/webapi.py", line 272, in upload_qc_metrics
            response = requests.post(
                       ^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/requests/api.py", line 115, in post
            return request("post", url, data=data, json=json, **kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/requests/api.py", line 59, in request
            return session.request(method=method, url=url, **kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
            resp = self.send(prep, **send_kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
            r = adapter.send(request, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/requests/adapters.py", line 532, in send
            raise ReadTimeout(e, request=request)
        requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='mriqc.nimh.nih.gov', port=443): Read timed out. (read timeout=15)
mckenziephagen commented 1 month ago
`crash-20240814-110617-mphagen-UploadMetrics.a1-9b22ba04-6602-4b06-abe0-9ba3f380cc26.txt` ```Node: mriqc_wf.funcMRIQC.UploadMetrics Working directory: /lscratch/mphagen/work/mriqc_wf/funcMRIQC/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..func..sub-0029_task-emomatchi ng_acq-seq_bold.nii.gz/UploadMetrics Node inputs: auth_token = 8sSYVI0XjFqacEMZ8wF4 email = endpoint = https://mriqc.nimh.nih.gov:443/api/v1 in_iqms = ['/scratch/users/mphagen/mriqc-protocol/ds002785/derivatives/mriqc-24.0.0/sub-0029/func/sub-0029_task-emomatching_acq-seq_bold.json'] strict = False Traceback (most recent call last): File "/opt/conda/lib/python3.11/site-packages/mriqc/engine/plugin.py", line 64, in run_node result['result'] = node.run(updatehash=updatehash) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run result = self._run_interface(execute=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 1380, in _run_interface result = self._collate_results( ^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 1249, in _collate_results for i, nresult, err in nodes: File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/utils.py", line 94, in nodelist_runner result = node.run(updatehash=updatehash) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run result = self._run_interface(execute=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface ^^^^^^^^^^^^^^^^^^^ return self._run_command(execute) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command raise NodeExecutionError(msg) nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node _UploadMetrics0. Traceback: Traceback (most recent call last): File "/opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py", line 537, in _make_request response = conn.getresponse() ^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/urllib3/connection.py", line 466, in getresponse httplib_response = super().getresponse() ^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/http/client.py", line 1390, in getresponse response.begin() File "/opt/conda/lib/python3.11/http/client.py", line 325, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/http/client.py", line 286, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/socket.py", line 706, in readinto return self._sock.recv_into(b) ^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/ssl.py", line 1314, in recv_into return self.read(nbytes, buffer) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/ssl.py", line 1166, in read return self._sslobj.read(len, buffer) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ TimeoutError: The read operation timed out The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/opt/conda/lib/python3.11/site-packages/requests/adapters.py", line 486, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py", line 847, in urlopen retries = retries.increment( ^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/urllib3/util/retry.py", line 470, in increment raise reraise(type(error), error, _stacktrace) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/urllib3/util/util.py", line 39, in reraise raise value File "/opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py", line 793, in urlopen response = self._make_request( ^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py", line 539, in _make_request self._raise_timeout(err=e, url=url, timeout_value=read_timeout) File "/opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py", line 370, in _raise_timeout raise ReadTimeoutError( urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='mriqc.nimh.nih.gov', port=443): Read timed out. (read timeout=15) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/conda/lib/python3.11/site-packages/nipype/interfaces/base/core.py", line 397, in run runtime = self._run_interface(runtime) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/mriqc/interfaces/webapi.py", line 141, in _run_interface response = upload_qc_metrics( ^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/mriqc/interfaces/webapi.py", line 244, in upload_qc_metrics response = requests.post( ^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/requests/api.py", line 115, in post return request("post", url, data=data, json=json, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/requests/api.py", line 59, in request return session.request(method=method, url=url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/requests/adapters.py", line 532, in send raise ReadTimeout(e, request=request) requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='mriqc.nimh.nih.gov', port=443): Read timed out. (read timeout=15) ```

As for the other crashfile, I found this:

`crash-20240814-114516-mphagen-EPI2MNI .a0-f02fd06b-9c09-4b39-b35c-5cd905232ae8.txt` ``` Node: mriqc_wf.dwiMRIQC.SpatialNormalization.EPI2MNI Working directory: /lscratch/mphagen/work/mriqc_wf/dwiMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..dwi..sub- 0029_dwi.nii.gz/EPI2MNI Node inputs: compress_report = auto explicit_masking = False flavor = precise float = False initial_moving_transform = lesion_mask = moving = boldref moving_image = /lscratch/mphagen/work/mriqc_wf/dwiMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..dwi..sub-0029 _dwi.nii.gz/SharpenEPI/sub-0029_dwi_valid_average_corrected.nii.gz moving_mask = /lscratch/mphagen/work/mriqc_wf/dwiMRIQC/synthstrip_wf/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..dwi..sub-0029_dwi.nii .gz/synthstrip/clipped_corrected_desc-brain_mask.nii.gz num_threads = 8 orientation = RAS out_report = report.svg File "/opt/conda/lib/python3.11/site-packages/mriqc/engine/plugin.py", line 64, in run_node reference = boldref reference_image = /templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-fMRIPrep_boldref.nii.gz reference_mask = /templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-brain_mask.nii.gz settings = template = MNI152NLin2009cAsym template_resolution = template_spec = Traceback (most recent call last): File "/opt/conda/lib/python3.11/site-packages/mriqc/engine/plugin.py", line 64, in run_node result['result'] = node.run(updatehash=updatehash) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run result = self._run_interface(execute=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface return self._run_command(execute) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command raise NodeExecutionError(msg) nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node EPI2MNI. Traceback: Traceback (most recent call last): File "/opt/conda/lib/python3.11/site-packages/nipype/interfaces/base/core.py", line 398, in run runtime = self._post_run_hook(runtime) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/niworkflows/interfaces/reportlets/registration.py", line 82, in _post_run_hook return super()._post_run_hook(runtime) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/nipype/interfaces/mixins/reporting.py", line 50, in _post_run_hook self._generate_report() File "/opt/conda/lib/python3.11/site-packages/niworkflows/interfaces/reportlets/base.py", line 102, in _generate_report plot_registration( File "/opt/conda/lib/python3.11/site-packages/niworkflows/viz/utils.py", line 388, in plot_registration svg = extract_svg(display, compress=compress) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/niworkflows/viz/utils.py", line 142, in extract_svg image_svg = svg_compress(image_svg, compress) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/niworkflows/viz/utils.py", line 66, in svg_compress pout = subprocess.run( ^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/subprocess.py", line 571, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command 'svgo -i - -o - -q -p 3 --pretty' returned non-zero exit status 135. ```

However, this might not be representative - this is the only crashfile for EPI2MNI in the directory, and this error was more widespread than this one participant. The other crashfiles are mostly for ComputeQI2, so I'll see if those are related to this problem.

oesteban commented 1 month ago

The first error in the UploadMetrics node was addressed in #1336, which was released with 24.0.2. Basically, yes, when the server is down you get a timeout and it wasn't handled gracefully (now it should). Please reopen this issue if 24.0.2 crashes again this way.

@mckenziephagen for the second error in EPI2MNI let's open a new issue.