search

neuropoly / data-management

Repo that deals with datalad aspects for internal use
4 stars 0 forks source link

PR: add extra derivatives to spine-generic-processed #93

Closed sandrinebedard closed 3 years ago

sandrinebedard commented 3 years ago

Description

I added automatic spinal cord segmentations that were manually QC-ed on the branch sb/add_auto_sc_seg to the derivatives of datasets/spine-generic-processed .

sandrinebedard commented 3 years ago

@kousu if could take a look, thanks!

kousu commented 3 years ago

The upload looks okay to me, all the files are there, etc. However bids-validator has some opinions:

nguenther@data:~/datasets/spine-generic-processed$ git branch
  git-annex
  master
* sb/add_auto_sc_seg
nguenther@data:~/datasets/spine-generic-processed$ bids-validator .
bids-validator@1.6.1

    1: [ERR] Files with such naming scheme are not part of BIDS specification. This error is most commonly caused by typos in file names that make them not BIDS compatible. Please consult the specification and make sure your files are named correctly. If this is not a file naming issue (for example when including files not yet covered by the BIDS specification) you should include a ".bidsignore" file in your dataset (see https://github.com/bids-standard/bids-validator#bidsignore for details). Please note that derived (processed) data should be placed in /derivatives folder and source data (such as DICOMS or behavioural logs in proprietary formats) should be placed in the /sourcedata folder. (code: 1 - NOT_INCLUDED)
        ./README.md
            Evidence: README.md
        ./sub-amu01/anat/sub-amu01_acq-MToff_MTS.json
            Evidence: sub-amu01_acq-MToff_MTS.json
        ./sub-amu01/anat/sub-amu01_acq-MToff_MTS.nii.gz
            Evidence: sub-amu01_acq-MToff_MTS.nii.gz
        ./sub-amu01/anat/sub-amu01_acq-MTon_MTS.json
            Evidence: sub-amu01_acq-MTon_MTS.json
        ./sub-amu01/anat/sub-amu01_acq-MTon_MTS.nii.gz
            Evidence: sub-amu01_acq-MTon_MTS.nii.gz
        ./sub-amu01/anat/sub-amu01_acq-T1w_MTS.json
            Evidence: sub-amu01_acq-T1w_MTS.json
        ./sub-amu01/anat/sub-amu01_acq-T1w_MTS.nii.gz
            Evidence: sub-amu01_acq-T1w_MTS.nii.gz
        ./sub-amu02/anat/sub-amu02_acq-MToff_MTS.json
            Evidence: sub-amu02_acq-MToff_MTS.json
        ./sub-amu02/anat/sub-amu02_acq-MToff_MTS.nii.gz
            Evidence: sub-amu02_acq-MToff_MTS.nii.gz
        ./sub-amu02/anat/sub-amu02_acq-MTon_MTS.json
            Evidence: sub-amu02_acq-MTon_MTS.json
        ... and 1542 more files having this issue (Use --verbose to see them all).

    Please visit https://neurostars.org/search?q=NOT_INCLUDED for existing conversations about this issue.

    2: [ERR] The number of volumes in this scan does not match the number of volumes in the corresponding .bvec and .bval files. (code: 29 - VOLUME_COUNT_MISMATCH)
        ./sub-amu01/dwi/sub-amu01_acq-b0_dwi.nii.gz
        ./sub-amu01/dwi/sub-amu01_rec-average_dwi.nii.gz
        ./sub-amu02/dwi/sub-amu02_acq-b0_dwi.nii.gz
        ./sub-amu02/dwi/sub-amu02_rec-average_dwi.nii.gz
        ./sub-amu03/dwi/sub-amu03_acq-b0_dwi.nii.gz
        ./sub-amu03/dwi/sub-amu03_rec-average_dwi.nii.gz
        ./sub-amu04/dwi/sub-amu04_acq-b0_dwi.nii.gz
        ./sub-amu04/dwi/sub-amu04_rec-average_dwi.nii.gz
        ./sub-amu05/dwi/sub-amu05_acq-b0_dwi.nii.gz
        ./sub-amu05/dwi/sub-amu05_rec-average_dwi.nii.gz
        ... and 275 more files having this issue (Use --verbose to see them all).

    Please visit https://neurostars.org/search?q=VOLUME_COUNT_MISMATCH for existing conversations about this issue.

    3: [ERR] Sub label contain an Illegal Character hyphen or underscore. Please edit the filename as per BIDS spec. (code: 62 - SUBJECT_VALUE_CONTAINS_ILLEGAL_CHARACTER)
        ./sub-oxfordOhba02/dwi/mask_sub-oxfordOhba02_dwi_dwi_mean.nii.gz
            Evidence: sub name contains illegal character:/sub-oxfordOhba02/dwi/mask_sub-oxfordOhba02_dwi_dwi_mean.nii.gz
        ./sub-oxfordOhba02/dwi/sub-oxfordOhba02_dwi_b0.nii.gz
            Evidence: sub name contains illegal character:/sub-oxfordOhba02/dwi/sub-oxfordOhba02_dwi_b0.nii.gz
        ./sub-oxfordOhba02/dwi/sub-oxfordOhba02_dwi_dwi.nii.gz
            Evidence: sub name contains illegal character:/sub-oxfordOhba02/dwi/sub-oxfordOhba02_dwi_dwi.nii.gz
        ./sub-oxfordOhba02/dwi/sub-oxfordOhba02_dwi_dwi_mean_centerline.csv
            Evidence: sub name contains illegal character:/sub-oxfordOhba02/dwi/sub-oxfordOhba02_dwi_dwi_mean_centerline.csv

    Please visit https://neurostars.org/search?q=SUBJECT_VALUE_CONTAINS_ILLEGAL_CHARACTER for existing conversations about this issue.

    1: [WARN] Not all subjects contain the same files. Each subject should contain the same number of files with the same naming unless some files are known to be missing. (code: 38 - INCONSISTENT_SUBJECTS)
        ./sub-amu01/dwi/sub-amu01_run-01_dwi.bval
            Evidence: Subject: sub-amu01; Missing file: sub-amu01_run-01_dwi.bval
        ./sub-amu01/dwi/sub-amu01_run-01_dwi.bvec
            Evidence: Subject: sub-amu01; Missing file: sub-amu01_run-01_dwi.bvec
        ./sub-amu01/dwi/sub-amu01_run-01_dwi.json
            Evidence: Subject: sub-amu01; Missing file: sub-amu01_run-01_dwi.json
        ./sub-amu01/dwi/sub-amu01_run-01_dwi.nii.gz
            Evidence: Subject: sub-amu01; Missing file: sub-amu01_run-01_dwi.nii.gz
        ./sub-amu01/dwi/sub-amu01_run-02_dwi.bval
            Evidence: Subject: sub-amu01; Missing file: sub-amu01_run-02_dwi.bval
        ./sub-amu01/dwi/sub-amu01_run-02_dwi.bvec
            Evidence: Subject: sub-amu01; Missing file: sub-amu01_run-02_dwi.bvec
        ./sub-amu01/dwi/sub-amu01_run-02_dwi.json
            Evidence: Subject: sub-amu01; Missing file: sub-amu01_run-02_dwi.json
        ./sub-amu01/dwi/sub-amu01_run-02_dwi.nii.gz
            Evidence: Subject: sub-amu01; Missing file: sub-amu01_run-02_dwi.nii.gz
        ./sub-amu01/dwi/sub-amu01_run-03_dwi.bval
            Evidence: Subject: sub-amu01; Missing file: sub-amu01_run-03_dwi.bval
        ./sub-amu01/dwi/sub-amu01_run-03_dwi.bvec
            Evidence: Subject: sub-amu01; Missing file: sub-amu01_run-03_dwi.bvec
        ... and 3664 more files having this issue (Use --verbose to see them all).

    Please visit https://neurostars.org/search?q=INCONSISTENT_SUBJECTS for existing conversations about this issue.

    2: [WARN] Not all subjects/sessions/runs have the same scanning parameters. (code: 39 - INCONSISTENT_PARAMETERS)
        ./sub-amu01/dwi/sub-amu01_dwi.nii.gz
        ./sub-amu01/dwi/sub-amu01_rec-average_dwi.nii.gz
        ./sub-amu02/dwi/sub-amu02_dwi.nii.gz
        ./sub-amu02/dwi/sub-amu02_rec-average_dwi.nii.gz
        ./sub-amu03/dwi/sub-amu03_dwi.nii.gz
        ./sub-amu03/dwi/sub-amu03_rec-average_dwi.nii.gz
        ./sub-amu04/dwi/sub-amu04_dwi.nii.gz
        ./sub-amu04/dwi/sub-amu04_rec-average_dwi.nii.gz
        ./sub-amu05/dwi/sub-amu05_dwi.nii.gz
        ./sub-amu05/dwi/sub-amu05_rec-average_dwi.nii.gz
        ... and 569 more files having this issue (Use --verbose to see them all).

    Please visit https://neurostars.org/search?q=INCONSISTENT_PARAMETERS for existing conversations about this issue.

        Summary:                   Available Tasks:        Available Modalities: 
        4557 Files, 14.79GB                                T1w                   
        267 - Subjects                                     T2star                
        1 - Session                                        T2w                   
                                                           dwi                   

    If you have any questions, please post on https://neurostars.org/tags/bids.

I don't think all (or even any?) of these were caused by your branch; I'll double-check in a moment.

kousu commented 3 years ago

I'm going to get @jcohenadad's second opinion on this in our 1-on-1 today.

sandrinebedard commented 3 years ago

I don't think all (or even any?) of these were caused by your branch; I'll double-check in a moment.

No I don't think any of them are about the derivatives I added in the branch! But I think that when we created the dataset, we didn't run Bids validator, so we still could address that.

From what I see, 

3: [ERR] Sub label contain an Illegal Character hyphen or underscore. Please edit the filename as per BIDS spec. (code: 62 - SUBJECT_VALUE_CONTAINS_ILLEGAL_CHARACTER)
        ./sub-oxfordOhba02/dwi/mask_sub-oxfordOhba02_dwi_dwi_mean.nii.gz
            Evidence: sub name contains illegal character:/sub-oxfordOhba02/dwi/mask_sub-oxfordOhba02_dwi_dwi_mean.nii.gz
        ./sub-oxfordOhba02/dwi/sub-oxfordOhba02_dwi_b0.nii.gz
            Evidence: sub name contains illegal character:/sub-oxfordOhba02/dwi/sub-oxfordOhba02_dwi_b0.nii.gz
        ./sub-oxfordOhba02/dwi/sub-oxfordOhba02_dwi_dwi.nii.gz
            Evidence: sub name contains illegal character:/sub-oxfordOhba02/dwi/sub-oxfordOhba02_dwi_dwi.nii.gz
        ./sub-oxfordOhba02/dwi/sub-oxfordOhba02_dwi_dwi_mean_centerline.csv
            Evidence: sub name contains illegal character:/sub-oxfordOhba02/dwi/sub-oxfordOhba02_dwi_dwi_mean_centerline.csv`

Some of those files should just not be there and should be deleted, maybe we could compare with spine-generic/data-multi-subject to compare if the errors were introduced in this processed version of the dataset or not!

kousu commented 3 years ago

Okay! I think this is probably good to merge this then!

jcohenadad commented 3 years ago

@alexfoias can you pls inform how to ignore a bunch of files not passing the validator

@kousu @sandrinebedard indeed, this spine-generic BIDS data are not passing the new qMRI rules anymore, this is something we need to update at some point-- @alexfoias can you pls open an issue on the spine-generic repos (or find if already opened)

kousu commented 3 years ago

Alright it's merged! Thanks @sandrinebedard. Sorry for the wait!

upload-log.txt