search

juaml / junifer

Forschungszentrum Jülich Neuroimaging Feature Extractor
https://juaml.github.io/junifer
GNU Affero General Public License v3.0
14 stars 13 forks source link

[BUG]: Merge parcellations fail with extra dimensions #331

Closed fraimondo closed 4 months ago

fraimondo commented 4 months ago

Is there an existing issue for this?

Current Behavior

Running a simple FC marker with two parcellations fails:

/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/data/parcellations.py:1785: RuntimeWarning: The parcellations have different resolutions!Resampling all parcellations to the first one in the list.
  warn_with_log(
Traceback (most recent call last):
  File "/home/fraimondo/miniconda3/envs/neurodc/bin/junifer", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/api/cli.py", line 223, in run
    api_run(
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/api/functions.py", line 176, in run
    mc.fit(datagrabber_object[t_element])
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/markers/collection.py", line 101, in fit
    m_value = marker.fit_transform(data, storage=self._storage)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/pipeline/pipeline_step_mixin.py", line 229, in fit_transform
    return self._fit_transform(input=input, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/markers/base.py", line 201, in _fit_transform
    t_out = self.compute(input=t_input, extra_input=extra_input)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/markers/functional_connectivity/functional_connectivity_base.py", line 138, in compute
    aggregation = self.aggregate(input, extra_input=extra_input)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/markers/functional_connectivity/functional_connectivity_parcels.py", line 102, in aggregate
    return parcel_aggregation.compute(input, extra_input=extra_input)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/markers/parcel_aggregation.py", line 191, in compute
    parcellation_img, labels = get_parcellation(
                               ^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/data/parcellations.py", line 348, in get_parcellation
    resampled_parcellation_img, labels = merge_parcellations(
                                         ^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/data/parcellations.py", line 1805, in merge_parcellations
    parc_data[parc_data == 0] += t_parc_data[parc_data == 0]
                                 ~~~~~~~~~~~^^^^^^^^^^^^^^^^
IndexError: too many indices for array: array is 3-dimensional, but 4 were indexed

Expected Behavior

to run withour error.

Steps To Reproduce

  1. Install latest (--pre) junifer from pip.
  2. run this yaml:
workdir: /tmp

datagrabber:
  kind: DataladAOMICID1000
  types: 
    - BOLD
    - BOLD_confounds
    - BOLD_mask
preprocess:
    kind: fMRIPrepConfoundRemover
    detrend: true
    standardize: true
    strategy:
        motion: full
        wm_csf: full
        global_signal: full
    low_pass: 0.08
    high_pass: 0.01
    masks:
      - inherit
      - compute_epi_mask
      - threshold: 0
markers:
  - name: parccortical-Schaefer100x17-TianxS2x3T
    kind: FunctionalConnectivityParcels
    parcellation: 
        - Schaefer100x17
        - TianxS2x3TxMNInonlinear2009cAsym
    cor_method: correlation
    cor_method_params:
      empirical: true
    masks:
      - inherit

storage:
  kind: HDF5FeatureStorage
  uri: "/data/group/riseml/fraimondo/ds003097_FC/ds003097_FC.hdf5"
queue:
  jobname: aomic_fc
  kind: HTCondor
  env:
    kind: conda
    name: neurodc
  mem: 8G
  disk: 5G
  verbose: info
  pre_run: |
    source /data/group/appliedml/tools/afni_23.1.10/afni.sh
    source /data/group/appliedml/tools/ants_2.5.0/ants.sh
    source /data/group/appliedml/tools/fsl_6.0.4-patched2/fsl.sh

Environment

junifer:
  version: 0.0.5.dev27
python:
  version: 3.11.8
  implementation: CPython
dependencies:
  click: 8.1.7
  numpy: 1.26.4
  scipy: 1.11.4
  datalad: 0.19.6
  pandas: 2.1.4
  nibabel: 5.2.1
  nilearn: 0.10.2
  sqlalchemy: 2.0.29
  ruamel.yaml: 0.17.40
  httpx: 0.26.0
  tqdm: 4.66.1
  templateflow: 24.2.0
  looseversion: None
system:
  platform: Linux-6.6.13+bpo-amd64-x86_64-with-glibc2.36
environment:
  LC_CTYPE: en_US.UTF-8
  PATH: 
    /home/fraimondo/miniconda3/envs/neurodc/bin:/home/fraimondo/miniconda3/condabin:/home/fraimondo/.dotfiles/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/X11R6/bin:/usr/local/games:/usr/games

Relevant log output

Activating neurodc
Sourcing pre_run.sh
Script dir from afni.sh: /data/group/appliedml/tools/afni_23.1.10
Path after sourcing afni.sh: /data/group/appliedml/tools/afni_23.1.10/binaries:/home/fraimondo/miniconda3/envs/neurodc/bin:/home/fraimondo/miniconda3/condabin:/home/fraimondo/miniconda3/envs/neurodc/bin:/home/fraimondo/miniconda3/condabin:/home/fraimondo/.dotfiles/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/X11R6/bin:/usr/local/games:/usr/games
Script dir from ants.sh: /data/group/appliedml/tools/ants_2.5.0
Path after sourcing ants.sh: /data/group/appliedml/tools/ants_2.5.0/binaries:/data/group/appliedml/tools/afni_23.1.10/binaries:/home/fraimondo/miniconda3/envs/neurodc/bin:/home/fraimondo/miniconda3/condabin:/home/fraimondo/miniconda3/envs/neurodc/bin:/home/fraimondo/miniconda3/condabin:/home/fraimondo/.dotfiles/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/X11R6/bin:/usr/local/games:/usr/games
Script dir from fsl.sh: /data/group/appliedml/tools/fsl_6.0.4-patched2
Path after sourcing fsl.sh: /data/group/appliedml/tools/fsl_6.0.4-patched2/binaries:/data/group/appliedml/tools/ants_2.5.0/binaries:/data/group/appliedml/tools/afni_23.1.10/binaries:/home/fraimondo/miniconda3/envs/neurodc/bin:/home/fraimondo/miniconda3/condabin:/home/fraimondo/miniconda3/envs/neurodc/bin:/home/fraimondo/miniconda3/condabin:/home/fraimondo/.dotfiles/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/X11R6/bin:/usr/local/games:/usr/games
Running junifer run /home/fraimondo/dev/scratch/neurodc/junifer_jobs/aomic_fc/config.yaml --verbose info --element sub-0014 in conda environment
2024-04-12 22:25:36,051 - JUNIFER - INFO - ===== Lib Versions =====
2024-04-12 22:25:36,051 - JUNIFER - INFO - numpy: 1.26.4
2024-04-12 22:25:36,051 - JUNIFER - INFO - scipy: 1.11.4
2024-04-12 22:25:36,051 - JUNIFER - INFO - pandas: 2.1.4
2024-04-12 22:25:36,051 - JUNIFER - INFO - nilearn: 0.10.2
2024-04-12 22:25:36,051 - JUNIFER - INFO - nibabel: 5.2.1
2024-04-12 22:25:36,051 - JUNIFER - INFO - junifer: 0.0.5.dev27
2024-04-12 22:25:36,051 - JUNIFER - INFO - ========================
2024-04-12 22:25:36,051 - JUNIFER - INFO - Parsing yaml file: /home/fraimondo/dev/scratch/neurodc/junifer_jobs/aomic_fc/config.yaml
2024-04-12 22:25:36,058 - JUNIFER - INFO - `datadir` is None, creating a temporary directory
2024-04-12 22:25:36,058 - JUNIFER - INFO - `datadir` set to /tmp/tmpgd5w41ln/datadir
2024-04-12 22:25:36,289 - JUNIFER - INFO - Validating Marker Collection
2024-04-12 22:25:36,289 - JUNIFER - INFO - DataGrabber output type: ['BOLD', 'BOLD_confounds', 'BOLD_mask']
2024-04-12 22:25:36,289 - JUNIFER - INFO - Validating Data Reader:
2024-04-12 22:25:36,289 - JUNIFER - INFO - Data Reader output type: ['BOLD', 'BOLD_confounds', 'BOLD_mask']
2024-04-12 22:25:36,289 - JUNIFER - INFO - Validating Preprocessor: fMRIPrepConfoundRemover
2024-04-12 22:25:36,289 - JUNIFER - INFO - Preprocess output type: ['BOLD']
2024-04-12 22:25:36,289 - JUNIFER - INFO - Validating Marker: parccortical-Schaefer100x17-TianxS2x3T
2024-04-12 22:25:36,386 - JUNIFER - INFO - Marker output type: ['matrix']
2024-04-12 22:25:36,386 - JUNIFER - INFO - Validating storage for parccortical-Schaefer100x17-TianxS2x3T
2024-04-12 22:27:39,404 - JUNIFER - INFO - Getting element sub-0014
2024-04-12 22:28:45,106 - JUNIFER - INFO - Fitting pipeline
2024-04-12 22:28:45,106 - JUNIFER - INFO - Reading BOLD from /tmp/tmpgd5w41ln/datadir/derivatives/fmriprep/sub-0014/func/sub-0014_task-moviewatching_space-MNI152NLin2009cAsym_desc-preproc_bold.nii.gz
2024-04-12 22:28:45,106 - JUNIFER - INFO - BOLD is type NIFTI
2024-04-12 22:28:45,107 - JUNIFER - INFO - Reading BOLD_confounds from /tmp/tmpgd5w41ln/datadir/derivatives/fmriprep/sub-0014/func/sub-0014_task-moviewatching_desc-confounds_regressors.tsv
2024-04-12 22:28:45,107 - JUNIFER - INFO - BOLD_confounds is type TSV
2024-04-12 22:28:45,123 - JUNIFER - INFO - Reading BOLD_mask from /tmp/tmpgd5w41ln/datadir/derivatives/fmriprep/sub-0014/func/sub-0014_task-moviewatching_space-MNI152NLin2009cAsym_desc-brain_mask.nii.gz
2024-04-12 22:28:45,123 - JUNIFER - INFO - BOLD_mask is type NIFTI
2024-04-12 22:28:45,124 - JUNIFER - INFO - Preprocessing data with fMRIPrepConfoundRemover
2024-04-12 22:28:45,124 - JUNIFER - INFO - Preprocessing BOLD
2024-04-12 22:28:47,902 - JUNIFER - INFO - No `t_r` specified, using t_r from NIfTI header
2024-04-12 22:28:47,903 - JUNIFER - INFO - Read t_r from NIfTI header: 2.200000047683716
2024-04-12 22:28:48,266 - JUNIFER - INFO - Cleaning image using nilearn
2024-04-12 22:28:54,670 - JUNIFER - INFO - Fitting marker parccortical-Schaefer100x17-TianxS2x3T
2024-04-12 22:28:54,778 - JUNIFER - INFO - Computing BOLD
2024-04-12 22:28:54,779 - JUNIFER - INFO - Fetching one of Schaefer parcellations.
2024-04-12 22:28:54,779 - JUNIFER - INFO - Parcellation parameters:
2024-04-12 22:28:54,779 - JUNIFER - INFO -  resolution: 3.0
2024-04-12 22:28:54,779 - JUNIFER - INFO -  n_rois: 100
2024-04-12 22:28:54,779 - JUNIFER - INFO -  yeo_networks: 17
2024-04-12 22:28:54,781 - JUNIFER - INFO - Loading parcellation /home/fraimondo/junifer/data/parcellations/schaefer_2018/Schaefer2018_100Parcels_17Networks_order_FSLMNI152_2mm.nii.gz
2024-04-12 22:28:54,816 - JUNIFER - INFO - Found existing xfm file for MNI152NLin6Asym to MNI152NLin2009cAsym at /home/fraimondo/junifer/data/xfms/MNI152NLin6Asym_to_MNI152NLin2009cAsym/MNI152NLin6Asym_to_MNI152NLin2009cAsym_Composite.h5
2024-04-12 22:28:55,773 - JUNIFER - INFO - Downloading template MNI152NLin2009cAsym in resolution 2
2024-04-12 22:28:55,892 - JUNIFER - INFO - antsApplyTransforms command to be executed:
antsApplyTransforms -d 3 -e 3 -n 'GenericLabel[NearestNeighbor]' -i /tmp/tmpol3919xi/parcellationsupbm3qm_/Schaefer100x17.nii.gz -r /tmp/tmpol3919xi/parcellationsupbm3qm_/MNI152NLin2009cAsym_T1w_3.0.nii.gz -t /home/fraimondo/junifer/data/xfms/MNI152NLin6Asym_to_MNI152NLin2009cAsym/MNI152NLin6Asym_to_MNI152NLin2009cAsym_Composite.h5 -o /tmp/tmpyk7pw003/parcellationskeha8hp0/Schaefer100x17_warped_from_MNI152NLin6Asym_to_MNI152NLin2009cAsym.nii.gz
2024-04-12 22:29:00,187 - JUNIFER - INFO - antsApplyTransforms command succeeded with the following output:
b'/tmp/tmpol3919xi/parcellationsupbm3qm_/Schaefer100x17.nii.gz is a file\n/tmp/tmpol3919xi/parcellationsupbm3qm_/MNI152NLin2009cAsym_T1w_3.0.nii.gz is a file\n/home/fraimondo/junifer/data/xfms/MNI152NLin6Asym_to_MNI152NLin2009cAsym/MNI152NLin6Asym_to_MNI152NLin2009cAsym_Composite.h5 is a file\n/tmp/tmpyk7pw003/parcellationskeha8hp0/Schaefer100x17_warped_from_MNI152NLin6Asym_to_MNI152NLin2009cAsym.nii.gz is a prefix\nSingularity args: --bind /tmp/tmpol3919xi/parcellationsupbm3qm_:/data/mount_1 --bind /tmp/tmpol3919xi/parcellationsupbm3qm_:/data/mount_2 --bind /home/fraimondo/junifer/data/xfms/MNI152NLin6Asym_to_MNI152NLin2009cAsym:/data/mount_3 --bind /tmp/tmpyk7pw003/parcellationskeha8hp0:/data/mount_4\nCorrected args for ANTS: antsApplyTransforms -d 3 -e 3 -n GenericLabel[NearestNeighbor] -i /data/mount_1/Schaefer100x17.nii.gz -r /data/mount_2/MNI152NLin2009cAsym_T1w_3.0.nii.gz -t /data/mount_3/MNI152NLin6Asym_to_MNI152NLin2009cAsym_Composite.h5 -o /data/mount_4/Schaefer100x17_warped_from_MNI152NLin6Asym_to_MNI152NLin2009cAsym.nii.gz\nRunning command: /data/group/appliedml/tools/ants_2.5.0/singularity_cmd exec --cleanenv --bind /tmp/tmpol3919xi/parcellationsupbm3qm_:/data/mount_1 --bind /tmp/tmpol3919xi/parcellationsupbm3qm_:/data/mount_2 --bind /home/fraimondo/junifer/data/xfms/MNI152NLin6Asym_to_MNI152NLin2009cAsym:/data/mount_3 --bind /tmp/tmpyk7pw003/parcellationskeha8hp0:/data/mount_4 /data/group/appliedml/tools/ants_2.5.0/ants_v2.5.0.sif antsApplyTransforms -d 3 -e 3 -n GenericLabel[NearestNeighbor] -i /data/mount_1/Schaefer100x17.nii.gz -r /data/mount_2/MNI152NLin2009cAsym_T1w_3.0.nii.gz -t /data/mount_3/MNI152NLin6Asym_to_MNI152NLin2009cAsym_Composite.h5 -o /data/mount_4/Schaefer100x17_warped_from_MNI152NLin6Asym_to_MNI152NLin2009cAsym.nii.gz\n'
2024-04-12 22:29:00,249 - JUNIFER - INFO - Fetching one of Tian parcellations.
2024-04-12 22:29:00,249 - JUNIFER - INFO - Parcellation parameters:
2024-04-12 22:29:00,249 - JUNIFER - INFO -  resolution: 3.0
2024-04-12 22:29:00,249 - JUNIFER - INFO -  scale: 2
2024-04-12 22:29:00,249 - JUNIFER - INFO -  space: MNI152NLin2009cAsym
2024-04-12 22:29:00,249 - JUNIFER - INFO -  magneticfield: 3T
2024-04-12 22:29:00,251 - JUNIFER - INFO - Loading parcellation /home/fraimondo/junifer/data/parcellations/Tian2020MSA_v1.1/3T/Subcortex-Only/Tian_Subcortex_S2_3T_2009cAsym.nii.gz
2024-04-12 22:29:00,334 - JUNIFER - WARNING - The parcellations have different resolutions!Resampling all parcellations to the first one in the list.

Anything else?

No response

synchon commented 4 months ago

@fraimondo Can you check this branch out and see if the issue is solved?

codecov[bot] commented 4 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 100.00%. Comparing base (a13e587) to head (54e6fb8).

Additional details and impacted files [![Impacted file tree graph](https://app.codecov.io/gh/juaml/junifer/pull/331/graphs/tree.svg?width=650&height=150&src=pr&token=5H21JuZXMw&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml)](https://app.codecov.io/gh/juaml/junifer/pull/331?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) ```diff @@ Coverage Diff @@ ## main #331 +/- ## ========================================= Coverage 100.00% 100.00% ========================================= Files 1 1 Lines 1 1 ========================================= Hits 1 1 ``` | [Flag](https://app.codecov.io/gh/juaml/junifer/pull/331/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | Coverage Δ | | |---|---|---| | [docs](https://app.codecov.io/gh/juaml/junifer/pull/331/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | `100.00% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#carryforward-flags-in-the-pull-request-comment) to find out more.
github-actions[bot] commented 4 months ago

PR Preview Action v1.4.7 :---: Preview removed because the pull request was closed. 2024-05-06 10:44 UTC

fraimondo commented 4 months ago 
workdir: /tmp

datagrabber:
  kind: DataladAOMICID1000
  types: 
    - BOLD
    - BOLD_confounds
    - BOLD_mask
preprocess:
    kind: fMRIPrepConfoundRemover
    detrend: true
    standardize: true
    strategy:
        motion: full
        wm_csf: full
        global_signal: full
    low_pass: 0.08
    high_pass: 0.01
    masks:
      - inherit
      - compute_epi_mask
      - threshold: 0
markers:
  - name: parccortical-Schaefer100x17-TianxS2x3T
    kind: FunctionalConnectivityParcels
    parcellation: 
        - Schaefer100x17
        - TianxS2x3TxMNInonlinear2009cAsym
    cor_method: correlation
    cor_method_params:
      empirical: true
    masks:
      - inherit

storage:
  kind: HDF5FeatureStorage
  uri: "/data/group/riseml/fraimondo/ds003097_FC/ds003097_FC.hdf5"
queue:
  jobname: aomic_fc
  kind: HTCondor
  env:
    kind: conda
    name: neurodc
  mem: 8G
  disk: 5G
  verbose: info
  pre_run: |
    source /data/group/appliedml/tools/afni_23.1.10/afni.sh
    source /data/group/appliedml/tools/ants_2.5.0/ants.sh
    source /data/group/appliedml/tools/fsl_6.0.4-patched2/fsl.sh

Yes, it works now.