🐛 User-provided brain masks are not being read in to the resource list

nimzodisaster commented 1 month ago

Describe the bug

I am supplying my own brain masks e.g. data config:

- anat:
    T1w: /mnt/Friar/rs/BIDS_DATA/APP/sub-107990100/ses-1/anat/sub-107990100_ses-1_T1w.nii.gz
    freesurfer_dir: null
  brain_mask: /mnt/Friar/rs/BIDS_DATA/APP/sub-107990100/ses-1/anat/sub-107990100_ses-1_T1w_desc-brain_mask.nii.gz
  site: APP
  subject_id: '107990100'
  unique_id: '1'

but the brain masks are not being read in to the resource list.

C-PAC failed to start
Traceback (most recent call last):
...
...
    self.node, self.out = strat_pool.get_data(resource, **kwargs)
  File "/code/CPAC/pipeline/engine.py", line 390, in get_data
    elif quick_single or len(self.get(resource)) == 1:
  File "/code/CPAC/pipeline/engine.py", line 364, in get
    raise LookupError(
LookupError: When trying to connect one of the node blocks ['register_ANTs_anat_to_template', 'register_FSL_anat_to_template'] to workflow 'cpac_108486100_1' after node block 'brain_extraction_T2': 

[!] C-PAC says: None of the listed resources are in the resource pool:

  ['space-T1w_desc-brain_mask', 'space-longitudinal_desc-brain_mask', 'space-T1w_desc-acpcbrain_mask']

My pipeline config file is set like:

    brain_extraction:

    run: Off

    # using: ['3dSkullStrip', 'BET', 'UNet', 'niworkflows-ants', 'FreeSurfer-ABCD', 'FreeSurfer-BET-Tight', 'FreeSurfer-BET-Loose', 'FreeSurfer-Brainmask']
    # this is a fork option
    using: ['BET']

The pipeline functions do not seem to allow for a key like 'Custom-brain-atlas'. The path and pattern of the mask name seems correct in the data config.

# Add the provided brain mask to the resource pool if it exists
if 'brain_mask' in rpool.data_config:
    brain_mask_path = rpool.data_config['brain_mask']
    rpool.add('space-T1w_desc-brain_mask', brain_mask_path)
    logger.info(f"Added provided brain mask to resource pool: {brain_mask_path}")

To reproduce

Provide your own brain mask in data config, set brain extraction to Off.

Preconfig

[ ] default
[ ] abcd-options
[ ] anat-only
[ ] blank
[ ] ccs-options
[ ] fmriprep-options
[ ] fx-options
[ ] monkey
[ ] monkey-ABCD
[ ] ndmg
[ ] nhp-macaque
[ ] preproc
[ ] rbc-options
[ ] rodent

Custom pipeline configuration

brain_extraction:

run: Off

# using: ['3dSkullStrip', 'BET', 'UNet', 'niworkflows-ants', 'FreeSurfer-ABCD', 'FreeSurfer-BET-Tight', 'FreeSurfer-BET-Loose', 'FreeSurfer-Brainmask']
# this is a fork option
using: ['BET']

Run command

No response

Expected behavior

Load provided resources, check resources prior to any step.

Acceptance criteria

use provided resources

Screenshots

No response

C-PAC version

No response

Container platform

Docker

Docker and/or Singularity version(s)

Docker Latest

Additional context

Proposed change to cpac_pipeline.py in MAIN branch. Check for mask in resource pool diff:

1013a1014,1019

if not rpool.check_rpool('space-T1w_desc-brain_mask'):
    anat_brain_mask_blocks = [

    ]
    pipeline_blocks += anat_brain_blocks

nimzodisaster commented 1 month ago

It seems T2 brain masking and T1 brain masking takes different approaches..

nimzodisaster commented 1 month ago

in anat_preproc, add a dummy node to just return the resource.

@nodeblock(
    name="brain_mask_custom",
    switch=[
    ["anatomical_preproc", "brain_extraction", "run"],
    ["anatomical_preproc", "run"],
    ],
    option_key=["anatomical_preproc", "brain_extraction", "using"],
    option_val="custom",
    inputs=["space-T1w_desc-brain_mask"],
    outputs=["space-T1w_desc-brain_mask"],
)
def brain_mask_custom(wf, cfg, strat_pool, pipe_num, opt=None):
    # Retrieve the data from strat_pool
    node, out = strat_pool.get_data('space-T1w_desc-brain_mask')

    # Return the workflow and the retrieved data
    outputs = {
        'space-T1w_desc-brain_mask': (node, out)
    }

    return (wf, outputs)

nimzodisaster commented 1 month ago

add custom as a valid option in schema

valid_options = {
    'acpc': {
        'target': ['brain', 'whole-head']
    },
    'brain_extraction': {
        'using': ['3dSkullStrip', 'BET', 'UNet', 'niworkflows-ants',
                  'FreeSurfer-BET-Tight', 'FreeSurfer-BET-Loose',
                  'FreeSurfer-ABCD', 'FreeSurfer-Brainmask','custom']

nimzodisaster commented 1 month ago

add to pipeline.py the option for custom

from CPAC.anat_preproc.anat_preproc import (
    freesurfer_reconall,
    freesurfer_abcd_preproc,
    anatomical_init,
    acpc_align_head,
    acpc_align_head_with_mask,
    acpc_align_brain,
    acpc_align_brain_with_mask,
    registration_T2w_to_T1w,
    non_local_means,
    n4_bias_correction,
    t1t2_bias_correction,
    brain_mask_afni,
    brain_mask_fsl,
    brain_mask_niworkflows_ants,
    brain_mask_unet,
    brain_mask_freesurfer,
    brain_mask_freesurfer_abcd,
    brain_mask_freesurfer_fsl_tight,
    brain_mask_freesurfer_fsl_loose,
    brain_mask_acpc_afni,
    brain_mask_acpc_fsl,
    brain_mask_acpc_niworkflows_ants,
    brain_mask_acpc_unet,
    brain_mask_acpc_freesurfer,
    brain_mask_acpc_freesurfer_abcd,
    correct_restore_brain_intensity_abcd,
    brain_mask_acpc_freesurfer_fsl_tight,
    brain_mask_acpc_freesurfer_fsl_loose,
    brain_extraction_temp,
    brain_extraction,
    anatomical_init_T2,
    acpc_align_head_T2,
    acpc_align_head_with_mask_T2,
    acpc_align_brain_T2,
    acpc_align_brain_with_mask_T2,
    non_local_means_T2,
    n4_bias_correction_T2,
    brain_mask_T2,
    brain_mask_acpc_T2,
    brain_extraction_temp_T2,
    brain_extraction_T2,
    brain_mask_custom
)
...
...
    anat_brain_mask_blocks = [
        [brain_mask_afni,
        brain_mask_fsl,
        brain_mask_niworkflows_ants,
        brain_mask_unet,
        brain_mask_freesurfer_abcd,
        brain_mask_freesurfer,
        brain_mask_freesurfer_fsl_tight,
        brain_mask_freesurfer_fsl_loose,
        brain_mask custom]
    ]
    pipeline_blocks += anat_brain_mask_blocks

    ...
    ...
        if rpool.check_rpool('space-T1w_desc-brain_mask'):
        anat_brain_mask_blocks = [
            brain_extraction
        ]
        pipeline_blocks += anat_brain_blocks

and to the import at the top!

nimzodisaster commented 1 month ago

And after those changes, space-T1w_desc-brain_mask is not in the resource pool...so I guess, I went the long way around to finding that out.

    raise LookupError(
LookupError: When trying to connect one of the node blocks ['register_ANTs_anat_to_template', 'register_FSL_anat_to_template'] to workflow 'cpac_101939200_1' after node block 'brain_extraction_T2': 

[!] C-PAC says: None of the listed resources are in the resource pool:

  ['space-T1w_desc-brain_mask', 'space-longitudinal_desc-brain_mask', 'space-T1w_desc-acpcbrain_mask']

nimzodisaster commented 1 month ago

I have placed files directly in the output folder

./outputs/pipeline_APP-core-pipeline/107925100_ses-1/anat/sub-107925100_ses-1_desc-brain_T1w.nii.gz ./outputs/pipeline_APP-core-pipeline/107925100_ses-1/anat/sub-107925100_ses-1_desc-brain_T1w.nii.gz

but I still get varieties of :


    connect, fetched =self.get(resource,
  File "/code/CPAC/pipeline/engine.py", line 364, in get
    raise LookupError(
LookupError: When trying to connect one of the node blocks ['tissue_seg_fsl_fast', 'tissue_seg_ants_prior', 'tissue_seg_freesurfer'] to workflow 'cpac_101939200_1' after node block 'correct_restore_brain_intensity_abcd': 

[!] C-PAC says: None of the listed resources are in the resource pool:

  ['desc-brain_T1w', 'space-longitudinal_desc-brain_T1w']

Options:
- You can enable a node block earlier in the pipeline which produces these resources. Check the 'outputs:' field in a node block's documentation.
- You can directly provide this required data by pulling it from another BIDS directory using 'source_outputs_dir:' in the pipeline configuration, or by placing it directly in your C-PAC output directory.
- If you have done these, and you still get this message, please let us know through any of our support channels at: https://fcp-indi.github.io/```

shnizzedy commented 1 month ago

Is

anatomical_preproc:
  brain_extraction:
    run: Off
    using: ['BET']

your whole pipeline config?

Since C-PAC v1.8.5 unspecified values come from the "blank" preconfig, which includes https://github.com/FCP-INDI/C-PAC/blob/96db8b0b65ab1d5f55fb3b895855af34d72c17e4/CPAC/resources/configs/pipeline_config_blank.yml#L184-L185, so that could be why the provided files aren't being read in.

(with

C-PAC version

No response

― https://github.com/FCP-INDI/C-PAC/issues/2112#issue-2296372967

I'm assuming you're running latest?)

In your log directory, there should be some timestamped cpac_pipeline_config* files. You can check the one without "min" in the name and see if it includes

pipeline_setup:
  outdir_ingress:
    run: Off

and if turning it on solves the issue.

nimzodisaster commented 1 month ago

Sorry for the incomplete information. This is in my config, and see attached files containing the timestamped configs and failed to start error.

pipeline_setup:
  outdir_ingress:
    run: On

Here are the contents of the out directory: I pre-populated it with the brain mask, trying to adhere to the formatting, I do not have json files in there though.

mothership:/mnt/Friar/rs/BIDS_DATA/derivatives/cpac1.8.6proc001/outputs/outputs/pipeline_utputs/pipeline_APP-core-pipeline/101939200_ses-1/anat$ ls -a
drwxrwxr-x 2 jlee38 jlee38    4096 May 22 14:45 ./
drwxrwxr-x 3 jlee38 jlee38    4096 May 22 14:45 ../
-rw-rw-r-- 1   jlee38 jlee38 3154047 May 22 14:46 sub-101939200_ses-1_desc-brain_T1w.nii.gz
-rw-r--r-- 1     jlee38 jlee38  157666 May 22 14:45 sub-101939200_ses-1_T1w_desc-brain_mask.nii.gz

I've uploaded the contents of the log file directory for one case as text files: cpac_data_config_b530090e_2024-05-22T21-59-30Z.yml.txt cpac_pipeline_config_b530090e_2024-05-22T21-59-30Z.yml.txt cpac_pipeline_config_b530090e_2024-05-22T21-59-30Z_min.yml.txt failedToStart.log.txt pypeline.log.txt

shnizzedy commented 1 month ago

If this doesn't fix the problem, I'll try to replicate and see what's going on here, but it might simply be the BIDS-entity-without-a-key T1w in the middle of the mask's filename is throwing a wrench in the pattern matching. Can you try renaming the mask sub-101939200_ses-1_space-T1w_desc-brain_mask.nii.gz and updating the data config accordingly?

nimzodisaster commented 1 month ago

ugh..I meant to name them with the space key, but I guess. I was tired. Unfortunately, renaming it correctly didn't change the lookup error:

LookupError: When trying to connect one of the node blocks ['register_ANTs_anat_to_template', 'register_FSL_anat_to_template'] to workflow 'cpac_101939200_1' after node block 'brain_extraction_T2': 
[!] C-PAC says: None of the listed resources are in the resource pool:
  ['space-T1w_desc-brain_mask', 'space-longitudinal_desc-brain_mask', 'space-T1w_desc-acpcbrain_mask']

This is all coming about because I discovered [FreeSurfer's new brain extraction tool] SynthStrip (https://surfer.nmr.mgh.harvard.edu/docs/synthstrip/), which is just phenomenally consistent, accurate, and fast.

nimzodisaster commented 1 month ago

I just figured out that I spent the last 50 hours hacking away at CPAC code in my local repository, or retrying latest docker, trying to get past my issue ONLY TO FIND OUT IT WAS A PERMISSIONS ISSUE WITH DOCKER.

FCP-INDI / C-PAC