🐛 [User-reported Bug] ICA AROMA crashes with 'DataFrame' object has no attribute 'append'. Did you mean: '_append' error

Describe the bug

Node: cpac_113622100_1.create_aroma_169.aroma_wf
Working directory: /mnt/Friar/rs/BIDS_DATA/derivatives/cpac1.8.6proc001/outputs/working/pipeline_APPCore_2/cpac_113622100_1/create_aroma_169/_scan_1/aroma_wf

Node inputs:

TR = <undefined>
args = <undefined>
denoise_type = nonaggr
dim = <undefined>
environ = {}
feat_dir = <undefined>
fnirt_warp_file = <undefined>
in_file = <undefined>
mask = <undefined>
mat_file = <undefined>
melodic_dir = <undefined>
motion_parameters = <undefined>
out_dir = .

Traceback (most recent call last):
  File "/usr/share/fsl/6.0/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/code/CPAC/pipeline/nipype_pipeline_engine/engine.py", line 443, in run
    return super().run(updatehash)
  File "/usr/share/fsl/6.0/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/usr/share/fsl/6.0/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/usr/share/fsl/6.0/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node aroma_wf.

Cmdline:
    ICA_AROMA.py -den nonaggr -i /mnt/Friar/rs/BIDS_DATA/derivatives/cpac1.8.6proc001/outputs/working/pipeline_APPCore_2/cpac_113622100_1/warp_ts_to_T1template_169/_scan_1/func_concat_warp_ts_to_T1template_169/sub-113622100_ses-1_task-rest_run-1_bold_resample_calc_tshift_0_volreg_tcat_calc_maths_0_trans_tcat.nii.gz -m /mnt/Friar/rs/BIDS_DATA/derivatives/cpac1.8.6proc001/outputs/working/pipeline_APPCore_2/cpac_113622100_1/create_aroma_169/_scan_1/bet_aroma/sub-113622100_ses-1_task-rest_run-1_bold_resample_calc_tshift_0_volreg_tcat_calc_maths_0_trans_tcat_brain_mask.nii.gz -mc /mnt/Friar/rs/BIDS_DATA/derivatives/cpac1.8.6proc001/outputs/working/pipeline_APPCore_2/cpac_113622100_1/create_aroma_169/_scan_1/par_mcflirt/sub-113622100_ses-1_task-rest_run-1_bold_resample_calc_tshift_0_volreg_tcat_calc_maths_0_trans_tcat_mcf.nii.gz.par -o /mnt/Friar/rs/BIDS_DATA/derivatives/cpac1.8.6proc001/outputs/working/pipeline_APPCore_2/cpac_113622100_1/create_aroma_169/_scan_1/aroma_wf
Stdout:

    ------------------------------- RUNNING ICA-AROMA ------------------------------- 
    --------------- 'ICA-based Automatic Removal Of Motion Artifacts' --------------- 

    Step 1) MELODIC
    Step 2) Automatic classification of the components
      - registering the spatial maps to MNI
      - extracting the CSF & Edge fraction features
      - extracting the Maximum RP correlation feature
      - extracting the High-frequency content feature
      - classification
    Found 1 file(s)
    Found 44 head motion-related components in a total of 44 components.
Stderr:
    Traceback (most recent call last):
      File "/opt/ICA-AROMA/ICA_AROMA.py", line 214, in <module>
        classification_plots.classification_plot(os.path.join(outDir, 'classification_overview.txt'),
      File "/opt/ICA-AROMA/classification_plots.py", line 60, in classification_plot
        df = df.append(df3, ignore_index=True)
      File "/usr/share/fsl/6.0/lib/python3.10/site-packages/pandas/core/generic.py", line 5989, in __getattr__
        return object.__getattribute__(self, name)
    AttributeError: 'DataFrame' object has no attribute 'append'. Did you mean: '_append'?
Traceback:
    Traceback (most recent call last):
      File "/usr/share/fsl/6.0/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 453, in aggregate_outputs
        setattr(outputs, key, val)
      File "/usr/share/fsl/6.0/lib/python3.10/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate
        value = super(File, self).validate(objekt, name, value, return_pathlike=True)
      File "/usr/share/fsl/6.0/lib/python3.10/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
        self.error(objekt, name, str(value))
      File "/usr/share/fsl/6.0/lib/python3.10/site-packages/traits/base_trait_handler.py", line 74, in error
        raise TraitError(
    traits.trait_errors.TraitError: The 'nonaggr_denoised_file' trait of an ICA_AROMAOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '/mnt/Friar/rs/BIDS_DATA/derivatives/cpac1.8.6proc001/outputs/working/pipeline_APPCore_2/cpac_113622100_1/create_aroma_169/_scan_1/aroma_wf/denoised_func_data_nonaggr.nii.gz' <class 'str'> was specified.

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/code/CPAC/pipeline/nipype_pipeline_engine/monkeypatch.py", line 43, in run
        outputs = self.aggregate_outputs(runtime)
      File "/usr/share/fsl/6.0/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 460, in aggregate_outputs
        raise FileNotFoundError(msg)
    FileNotFoundError: No such file or directory '/mnt/Friar/rs/BIDS_DATA/derivatives/cpac1.8.6proc001/outputs/working/pipeline_APPCore_2/cpac_113622100_1/create_aroma_169/_scan_1/aroma_wf/denoised_func_data_nonaggr.nii.gz' for output 'nonaggr_denoised_file' of a ICA_AROMA interface

To reproduce

No response

Preconfig

[ ] default
[ ] abcd-options
[ ] anat-only
[ ] blank
[ ] ccs-options
[ ] fmriprep-options
[ ] fx-options
[ ] monkey
[ ] monkey-ABCD
[ ] ndmg
[ ] nhp-macaque
[ ] preproc
[ ] rbc-options
[ ] rodent

Custom pipeline configuration

%YAML 1.1
---
# CPAC Pipeline Configuration YAML file
# Version 1.8.7.dev1
#
# http://fcp-indi.github.io for more info.
#
# Tip: This file can be edited manually with a text editor for quick modifications.
FROM: blank

pipeline_setup:

  # Name for this pipeline configuration - useful for identification.
  # This string will be sanitized and used in filepaths
  pipeline_name: APPCore_2
  output_directory:

    # Directory where C-PAC should write out processed data, logs, and crash reports.
    # - If running in a container (Singularity/Docker), you can simply set this to an arbitrary
    #   name like '/outputs', and then map (-B/-v) your desired output directory to that label.
    # - If running outside a container, this should be a full path to a directory.
    path: /mnt/Friar/rs/BIDS_DATA/derivatives/cpac1.8.6proc001/outputs/output

  system_config:

    # Prior to running a pipeline C-PAC makes a rough estimate of a worst-case-scenario maximum concurrent memory usage with high-resoltion data, raising an exception describing the recommended minimum memory allocation for the given configuration.
    # Turning this option off will allow pipelines to run without allocating the recommended minimum, allowing for more efficient runs at the risk of out-of-memory crashes (use at your own risk)
    raise_insufficient: Off

    # The maximum amount of memory each participant's workflow can allocate.
    # Use this to place an upper bound of memory usage.
    # - Warning: 'Memory Per Participant' multiplied by 'Number of Participants to Run Simultaneously'
    #   must not be more than the total amount of RAM.
    # - Conversely, using too little RAM can impede the speed of a pipeline run.
    # - It is recommended that you set this to a value that when multiplied by
    #   'Number of Participants to Run Simultaneously' is as much RAM you can safely allocate.
    maximum_memory_per_participant: 9.0

    # The maximum amount of cores (on a single machine) or slots on a node (on a cluster/grid)
    # to allocate per participant.
    # - Setting this above 1 will parallelize each participant's workflow where possible.
    #   If you wish to dedicate multiple cores to ANTS-based anatomical registration (below),
    #   this value must be equal or higher than the amount of cores provided to ANTS.
    # - The maximum number of cores your run can possibly employ will be this setting multiplied
    #   by the number of participants set to run in parallel (the 'Number of Participants to Run
    #   Simultaneously' setting).
    max_cores_per_participant: 3

    # The number of cores to allocate to ANTS-based anatomical registration per participant.
    # - Multiple cores can greatly speed up this preprocessing step.
    # - This number cannot be greater than the number of cores per participant.
    num_ants_threads: 3

    # The number of cores to allocate to processes that use OpenMP.
    num_OMP_threads: 3

  working_directory:

    # Directory where C-PAC should store temporary and intermediate files.
    # - This directory must be saved if you wish to re-run your pipeline from where you left off (if not completed).
    # - NOTE: As it stores all intermediate files, this directory can grow to become very
    #   large, especially for data with a large amount of TRs.
    # - If running in a container (Singularity/Docker), you can simply set this to an arbitrary
    #   name like '/work', and then map (-B/-v) your desired output directory to that label.
    # - If running outside a container, this should be a full path to a directory.
    # - This can be written to '/tmp' if you do not intend to save your working directory.
    path: /mnt/Friar/rs/BIDS_DATA/derivatives/cpac1.8.6proc001/outputs/working

  log_directory:
    path: /mnt/Friar/rs/BIDS_DATA/derivatives/cpac1.8.6proc001/outputs/log

  outdir_ingress:
    run: On

  Debugging:

    # Verbose developer messages.
    verbose: On

anatomical_preproc:
  run: On
  brain_extraction:
    run: On

    # using: ['3dSkullStrip', 'BET', 'UNet', 'niworkflows-ants', 'FreeSurfer-ABCD', 'FreeSurfer-BET-Tight', 'FreeSurfer-BET-Loose', 'FreeSurfer-Brainmask']
    # this is a fork option
    using: [BET]
    FSL-BET:

      # Switch "On" to crop out neck regions before generating the mask (default: Off).
      Robustfov: On

  # Non-local means filtering via ANTs DenoiseImage
  non_local_means_filtering:

    # this is a fork option
    run: [On]

  # N4 bias field correction via ANTs
  n4_bias_field_correction:

    # this is a fork option
    run: [On]

segmentation:

  # Automatically segment anatomical images into white matter, gray matter,
  # and CSF based on prior probability maps.
  run: On

registration_workflows:
  anatomical_registration:
    run: On
    registration:
      FSL-FNIRT:

        # The resolution to which anatomical images should be transformed during registration.
        # This is the resolution at which processed anatomical files will be output.
        # specifically for monkey pipeline
        ref_resolution: 1mm

    # The resolution to which anatomical images should be transformed during registration.
    # This is the resolution at which processed anatomical files will be output.
    resolution_for_anat: 1mm

  functional_registration:
    coregistration:

      # functional (BOLD/EPI) registration to anatomical (structural/T1)
      run: On
      boundary_based_registration:

        # this is a fork point
        #   run: [On, Off] - this will run both and fork the pipeline
        run: [On]

    func_registration_to_template:

      # these options modify the application (to the functional data), not the calculation, of the
      # T1-to-template and EPI-to-template transforms calculated earlier during registration
      # apply the functional-to-template (T1 template) registration transform to the functional data
      run: On
      output_resolution:

        # The resolution (in mm) to which the preprocessed, registered functional timeseries outputs are written into.
        # NOTE:
        #   selecting a 1 mm or 2 mm resolution might substantially increase your RAM needs- these resolutions should be selected with caution.
        #   for most cases, 3 mm or 4 mm resolutions are suggested.
        # NOTE:
        #   this also includes the single-volume 3D preprocessed functional data,
        #   such as the mean functional (mean EPI) in template space
        func_preproc_outputs: 4mm

        # The resolution (in mm) to which the registered derivative outputs are written into.
        # NOTE:
        #   this is for the single-volume functional-space outputs (i.e. derivatives)
        #   thus, a higher resolution may not result in a large increase in RAM needs as above
        func_derivative_outputs: 4mm

functional_preproc:
  run: On
  slice_timing_correction:

    # Interpolate voxel time courses so they are sampled at the same time points.
    # this is a fork point
    #   run: [On, Off] - this will run both and fork the pipeline
    run: [On]

    # use specified slice time pattern rather than one in header
    tpattern: altplus

  motion_estimates_and_correction:
    run: On
    motion_correction:

      # using: ['3dvolreg', 'mcflirt']
      # Forking is currently broken for this option.
      # Please use separate configs if you want to use each of 3dvolreg and mcflirt.
      # Follow https://github.com/FCP-INDI/C-PAC/issues/1935 to see when this issue is resolved.
      using: [3dvolreg]

  func_masking:
    run: On

    # using: ['AFNI', 'FSL', 'FSL_AFNI', 'Anatomical_Refined', 'Anatomical_Based', 'Anatomical_Resampled', 'CCS_Anatomical_Refined']
    # FSL_AFNI: fMRIPrep-style BOLD mask. Ref: https://github.com/nipreps/niworkflows/blob/a221f612/niworkflows/func/util.py#L246-L514
    # Anatomical_Refined: 1. binarize anat mask, in case it is not a binary mask. 2. fill holes of anat mask 3. init_bold_mask : input raw func → dilate init func brain mask 4. refined_bold_mask : input motion corrected func → dilate anatomical mask 5. get final func mask
    # Anatomical_Based: Generate the BOLD mask by basing it off of the anatomical brain mask. Adapted from DCAN Lab's BOLD mask method from the ABCD pipeline.
    # Anatomical_Resampled: Resample anatomical brain mask in standard space to get BOLD brain mask in standard space. Adapted from DCAN Lab's BOLD mask method from the ABCD pipeline. ("Create fMRI resolution standard space files for T1w image, wmparc, and brain mask […] don't use FLIRT to do spline interpolation with -applyisoxfm for the 2mm and 1mm cases because it doesn't know the peculiarities of the MNI template FOVs")
    # CCS_Anatomical_Refined: Generate the BOLD mask by basing it off of the anatomical brain. Adapted from the BOLD mask method from the CCS pipeline.
    # this is a fork point
    using: [Anatomical_Refined]

  generate_func_mean:

    # Generate mean functional image
    run: On

  normalize_func:

    # Normalize functional image
    run: On

  coreg_prep:

    # Generate sbref
    run: On

nuisance_corrections:
  2-nuisance_regression:

    # this is a fork point
    #   run: [On, Off] - this will run both and fork the pipeline
    run: [On]

    # Select which nuisance signal corrections to apply
    Regressors:
      - Name: nGSRySCRUB
        Motion:
          include_delayed: On
          include_squared: On
          include_delayed_squared: On
        aCompCor:
          summary:
            method: DetrendPC
            components: 5
          tissues:
          - WhiteMatter
          - CerebrospinalFluid
          extraction_resolution: 2
        GlobalSignal:
          summary: Mean
        CerebrospinalFluid:
          summary: Mean
          extraction_resolution: 2
          erode_mask: On
        Censor:
          method: Kill
          number_of_previous_trs_to_censor: 0
          number_of_subsequent_trs_to_censor: 0
          thresholds:
          - type: FD_J
            value: 0.25
        PolyOrt:
          degree: 2
        Bandpass:
          bottom_frequency: 0.01
          top_frequency: 0.1
          method: default

  1-ICA-AROMA:

    # this is a fork point
    #   run: [On, Off] - this will run both and fork the pipeline
    run: [On]

# OUTPUTS AND DERIVATIVES
# -----------------------
post_processing:
  spatial_smoothing:
    run: On

    # Smooth the derivative outputs.
    # Set as ['nonsmoothed'] to disable smoothing. Set as ['smoothed', 'nonsmoothed'] to get both.
    #
    # Options:
    #     ['smoothed', 'nonsmoothed']
    output: [smoothed, nonsmoothed]

  z-scoring:

    # z-score standardize the derivatives. This may be needed for group-level analysis.
    # Set as ['raw'] to disable z-scoring. Set as ['z-scored', 'raw'] to get both.
    #
    # Options:
    #     ['z-scored', 'raw']
    output: [z-scored, raw]

### Run command

cpac --image fcpindi/c-pac --tag latest \
             run /mnt/Friar/rs/BIDS_DATA/ /mnt/Friar/rs/BIDS_DATA/derivatives/cpac1.8.6proc001/outputs participant \
             --data_config_file /mnt/Friar/rs/BIDS_DATA/derivatives/cpac1.8.6proc001/configs/dataconfig_${sub}.yml \
             --pipeline-file /mnt/Friar/rs/BIDS_DATA/derivatives/cpac1.8.6proc001/configs/pipeline_config_APP_core_2.yml \
             --participant-label ${sub}

Expected behavior

not crash :)

Acceptance criteria

Finish ICA Aroma

Screenshots

No response

C-PAC version

v1.8.7

Container platform

Docker

Docker and/or Singularity version(s)

No response

Additional context

No response

FCP-INDI / C-PAC

🐛 [User-reported Bug] ICA AROMA crashes with 'DataFrame' object has no attribute 'append'. Did you mean: '_append' error #2117

Describe the bug

To reproduce

Preconfig

Custom pipeline configuration

Expected behavior

Acceptance criteria

Screenshots

C-PAC version

Container platform

Docker and/or Singularity version(s)

Additional context