search

nipy / nipype

Workflows and interfaces for neuroimaging packages
https://nipype.readthedocs.org/en/latest/
Other
748 stars 530 forks source link

File(exists=True) in output specs can cause failures after unused outputs are cleaned up #3493

Open effigies opened 2 years ago

effigies commented 2 years ago

Summary

If an output_spec contains a trait File(exists=True), and that output field is not passed as an input to another node, then the file will get cleaned up by remove_unnecessary_outputs but still be set in the traits. When re-loading a node working directory, this then causes a trait failure:

         ***********************************                                                                                                 
220718-18:01:54,394 nipype.workflow ERROR:                                                                                                                                                                                                                                                
         could not run node: nibabies_21_1_wf.single_subject_470437_1mo_wf.func_preproc_ses_1mo_task_rest_acq_AP_run_02_wf.subcortical_mni_alignment_wf.applyxfm_roi                                                  
220718-18:01:54,394 nipype.workflow INFO:                                                                                                    
         crashfile: /out/sub-470437/ses-1mo/log/20220718-172738_8d289d13-1f67-4ead-93a7-d590eb97a25f/crash-20220718-175440-root-_applyxfm_roi18-9b41d34f-48ea-4453-b226-60f0625e806e.txt
220718-18:01:54,394 nipype.workflow ERROR:                                                                                                   
         could not run node: nibabies_21_1_wf.single_subject_470437_1mo_wf.func_preproc_ses_1mo_task_rest_acq_PA_run_04_wf.subcortical_mni_alignment_wf.applyxfm_roi
220718-18:01:54,394 nipype.workflow INFO:                                                                                                    
         crashfile: /out/sub-470437/ses-1mo/log/20220718-172738_8d289d13-1f67-4ead-93a7-d590eb97a25f/crash-20220718-175440-root-_applyxfm_roi
18-5ef037cc-028f-4499-bfc1-7850b5900e63.txt                                                                                                                                                                                                                                               
220718-18:01:54,394 nipype.workflow ERROR:                                                                                                                                                                                                                                                
         could not run node: nibabies_21_1_wf.single_subject_470437_1mo_wf.func_preproc_ses_1mo_task_rest_acq_PA_run_05_wf.subcortical_mni_al                                                                                                                                             
ignment_wf.applyxfm_roi                                                                                                                                                                                                                                                                   
220718-18:01:54,394 nipype.workflow INFO:                                                                                                    
         crashfile: /out/sub-470437/ses-1mo/log/20220718-172738_8d289d13-1f67-4ead-93a7-d590eb97a25f/crash-20220718-175440-root-_applyxfm_roi0-5ec94b31-2de2-400e-872c-dbefe686a5a3.txt
220718-18:01:54,394 nipype.workflow ERROR:                                                                                                   
         could not run node: nibabies_21_1_wf.single_subject_470437_1mo_wf.func_preproc_ses_1mo_task_rest_acq_PA_run_04_wf.subcortical_mni_alignment_wf.applyxfm_roi
220718-18:01:54,394 nipype.workflow INFO:                                                                                                    
         crashfile: /out/sub-470437/ses-1mo/log/20220718-172738_8d289d13-1f67-4ead-93a7-d590eb97a25f/crash-20220718-175452-root-_applyxfm_roi17-4ba136ca-5ae1-4f59-8a8c-490f6a9b6344.txt
220718-18:01:54,394 nipype.workflow INFO:                                                                                                    
         ***********************************                                                                                                                                                                                                                                              
220718-18:01:54,472 nipype.workflow CRITICAL:                                                                                                
         nibabies failed: 4 raised. Re-raising first.                                                                                        
RuntimeError: Traceback (most recent call last):                                                                                             
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node                                   
    result["result"] = node.run(updatehash=updatehash)                                                                                       
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 524, in run                                            
    result = self._run_interface(execute=True)                                                                                               
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 642, in _run_interface                                 
    return self._run_command(execute)                                                                                                        
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 750, in _run_command                                   
    raise NodeExecutionError(                                                                                                                
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node _applyxfm_roi18.                                      

Traceback (most recent call last):                                                                                                           
  File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/core.py", line 454, in aggregate_outputs                               
    setattr(outputs, key, val)                                                                                                               
  File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate                            
    value = super(File, self).validate(objekt, name, value, return_pathlike=True)                                                            
  File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate                            
    self.error(objekt, name, str(value))                                                                                                     
  File "/opt/conda/lib/python3.9/site-packages/traits/base_trait_handler.py", line 74, in error                                              
    raise TraitError(                                                                                                                        
traits.trait_errors.TraitError: The 'out_matrix_file' trait of a FLIRTOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '/scratch/nibabies_21_1_wf/single_subject_470437_1mo_wf/func_preproc_ses_1mo_task_rest_acq_AP_run_02_wf/subcor
tical_mni_alignment_wf/applyxfm_roi/mapflow/_applyxfm_roi18/vol0000_xform-00000_merged_flirt.mat' <class 'str'> was specified.                                                                                                                                                            

During handling of the above exception, another exception occurred:                                                                          

Traceback (most recent call last):                                                                                                           
  File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/core.py", line 401, in run                                                                                                                                                                                          
    outputs = self.aggregate_outputs(runtime)                                                                                                
  File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/fsl/preprocess.py", line 742, in aggregate_outputs                          
    outputs = super(FLIRT, self).aggregate_outputs(                                                                                          
  File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/core.py", line 461, in aggregate_outputs                               
    raise FileNotFoundError(msg)                                                                                                             
FileNotFoundError: No such file or directory '/scratch/nibabies_21_1_wf/single_subject_470437_1mo_wf/func_preproc_ses_1mo_task_rest_acq_AP_run_02_wf/subcortical_mni_alignment_wf/applyxfm_roi/mapflow/_applyxfm_roi18/vol0000_xform-00000_merged_flirt.mat' for output 'out_matrix_file' 
of a ApplyXFM interface

I can see a few solutions:

1) Purge exists=True from output_specs. 2) Monkey-patch File(exists=True) in output specs to be simply File() 3) Catch the error and replace the str value with Undefined 4) If a file is going to be removed by remove_unnecessary_outputs, then set its value to Undefined before saving it.

Platform details:

{'commit_hash': '%h',
 'commit_source': 'archive substitution',
 'networkx_version': '2.8.2',
 'nibabel_version': '3.2.2',
 'nipype_version': '1.8.1',
 'numpy_version': '1.22.4',
 'pkg_path': '/opt/conda/lib/python3.9/site-packages/nipype',
 'scipy_version': '1.8.1',
 'sys_executable': '/opt/conda/bin/python',
 'sys_platform': 'linux',
 'sys_version': '3.9.12 | packaged by conda-forge | (main, Mar 24 2022, '
                '23:25:59) \n'
                '[GCC 10.3.0]',
 'traits_version': '6.3.2'}

Execution environment

Choose one

effigies commented 1 year ago

This error is becoming increasingly annoying. Almost every single processing error is now primarily reported as a failure to populate an output spec. See, for yet another example, https://github.com/nipreps/fmriprep/issues/3006.