sandrinebedard commented 2 years ago

Context

Currenlty, we have a processed version of this dataset (spine-generic-processed) where we have derivatives, but some are duplicated. See #121.

This PR moves the derivatives from spine-generic-processed to data-multi-subjects.

The derivatives are now BIDS compatible, they inlcude:

spinal cord segmentation (under derivatives/labels)
softsegmentation (under derivatives/lables_softseg) -->https://github.com/sct-pipeline/contrast-agnostic-softseg-spinalcord

Related issues

fixes #121

What is missing

since the processing script of the derivatives is not merged yet, in the README.txt under derivatives/labels_softseg, the link needs to be updated. #124
Add a readme for segmentations in /derivatives/labels to explain what are the processing steps and link to the processing script #124
Delete spine-generic-processed

sandrinebedard commented 1 year ago

@mguaypaq , could you take a look at the PR. Thank you!

mguaypaq commented 1 year ago

Sure, I'm pretty busy today but I'll take a look tomorrow.

sandrinebedard commented 1 year ago

I had some extra modifications to add, but I was unable to push to git annex:

(base) sebeda@joplin:~/data_nvme_sebeda/data-multi-subject/derivatives/labels/sub-perform02$ git push origin git-annex:git-annex
Username for 'https://github.com': sandrinebedard
Password for 'https://sandrinebedard@github.com':
To https://github.com/spine-generic/data-multi-subject
 ! [rejected]            git-annex -> git-annex (non-fast-forward)
error: failed to push some refs to 'https://github.com/spine-generic/data-multi-subject'
hint: Updates were rejected because a pushed branch tip is behind its remote
hint: counterpart. Check out this branch and integrate the remote changes
hint: (e.g. 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details

kousu commented 1 year ago

@sandrinebedard please try

git annex sync --no-content --only-annex --cleanup # upload mirror contents
git push

The wiki docs seem to have had a bad merge at some point (this point). The last step, git push origin git-annex:git-annex, is redundant. And also impossible, it turns out, if you're collaborating with others who've also been uploading to the annex.

The lab manual docs have the same instructions but without the final step. I've updated the wiki to match.

sandrinebedard commented 1 year ago

Thanks @kousu ! It seams t have worked!

sandrinebedard commented 1 year ago

@mguaypaq I will need finally to modify some extra data. I think you can wait to check this PR until then. I'll ping you when it's ready!

sandrinebedard commented 1 year ago

Hello @mguaypaq ! I've finished the extra modification, you can now take a look, thank you!

mguaypaq commented 1 year ago

It looks like the following 45 files are missing from amazon:

derivatives/labels_softseg/sub-beijingGE02/anat/sub-beijingGE02_T1w_softseg.nii.gz
derivatives/labels_softseg/sub-beijingGE02/anat/sub-beijingGE02_T2star_softseg.nii.gz
derivatives/labels_softseg/sub-beijingGE02/anat/sub-beijingGE02_T2w_softseg.nii.gz
derivatives/labels_softseg/sub-beijingGE02/dwi/sub-beijingGE02_rec-average_dwi_softseg.nii.gz
derivatives/labels_softseg/sub-beijingVerio01/anat/sub-beijingVerio01_T1w_softseg.nii.gz
derivatives/labels_softseg/sub-beijingVerio01/anat/sub-beijingVerio01_T2star_softseg.nii.gz
derivatives/labels_softseg/sub-beijingVerio01/anat/sub-beijingVerio01_T2w_softseg.nii.gz
derivatives/labels_softseg/sub-beijingVerio02/anat/sub-beijingVerio02_T1w_softseg.nii.gz
derivatives/labels_softseg/sub-beijingVerio02/anat/sub-beijingVerio02_T2star_softseg.nii.gz
derivatives/labels_softseg/sub-beijingVerio02/anat/sub-beijingVerio02_T2w_softseg.nii.gz
derivatives/labels_softseg/sub-beijingVerio03/anat/sub-beijingVerio03_T1w_softseg.nii.gz
derivatives/labels_softseg/sub-beijingVerio03/anat/sub-beijingVerio03_T2star_softseg.nii.gz
derivatives/labels_softseg/sub-beijingVerio03/anat/sub-beijingVerio03_T2w_softseg.nii.gz
derivatives/labels_softseg/sub-beijingVerio03/dwi/sub-beijingVerio03_rec-average_dwi_softseg.nii.gz
derivatives/labels_softseg/sub-ubc01/anat/sub-ubc01_acq-MTon_MTS_softseg.nii.gz
derivatives/labels_softseg/sub-ubc01/anat/sub-ubc01_T1w_softseg.nii.gz
derivatives/labels_softseg/sub-ubc01/anat/sub-ubc01_T2star_softseg.nii.gz
derivatives/labels_softseg/sub-ubc01/anat/sub-ubc01_T2w_softseg.nii.gz
derivatives/labels_softseg/sub-ubc01/dwi/sub-ubc01_rec-average_dwi_softseg.nii.gz
derivatives/labels_softseg/sub-ubc03/anat/sub-ubc03_acq-MTon_MTS_softseg.nii.gz
derivatives/labels_softseg/sub-ubc03/anat/sub-ubc03_T2star_softseg.nii.gz
derivatives/labels_softseg/sub-ubc03/anat/sub-ubc03_T2w_softseg.nii.gz
derivatives/labels_softseg/sub-ubc05/anat/sub-ubc05_acq-MTon_MTS_softseg.nii.gz
derivatives/labels_softseg/sub-ubc05/anat/sub-ubc05_T1w_softseg.nii.gz
derivatives/labels_softseg/sub-ubc05/anat/sub-ubc05_T2star_softseg.nii.gz
derivatives/labels_softseg/sub-ubc05/anat/sub-ubc05_T2w_softseg.nii.gz
derivatives/labels_softseg/sub-ubc05/dwi/sub-ubc05_rec-average_dwi_softseg.nii.gz
derivatives/labels_softseg/sub-ucdavis03/anat/sub-ucdavis03_acq-MTon_MTS_softseg.nii.gz
derivatives/labels_softseg/sub-ucdavis03/anat/sub-ucdavis03_T1w_softseg.nii.gz
derivatives/labels_softseg/sub-ucdavis03/anat/sub-ucdavis03_T2star_softseg.nii.gz
derivatives/labels_softseg/sub-ucdavis03/anat/sub-ucdavis03_T2w_softseg.nii.gz
derivatives/labels_softseg/sub-ucdavis03/dwi/sub-ucdavis03_rec-average_dwi_softseg.nii.gz
derivatives/labels_softseg/sub-ucdavis05/anat/sub-ucdavis05_acq-MTon_MTS_softseg.nii.gz
derivatives/labels_softseg/sub-ucdavis05/anat/sub-ucdavis05_T1w_softseg.nii.gz
derivatives/labels_softseg/sub-ucdavis05/anat/sub-ucdavis05_T2star_softseg.nii.gz
derivatives/labels_softseg/sub-ucdavis05/anat/sub-ucdavis05_T2w_softseg.nii.gz
derivatives/labels_softseg/sub-ucdavis05/dwi/sub-ucdavis05_rec-average_dwi_softseg.nii.gz
derivatives/labels_softseg/sub-ucdavis06/anat/sub-ucdavis06_acq-MTon_MTS_softseg.nii.gz
derivatives/labels_softseg/sub-ucdavis06/anat/sub-ucdavis06_T1w_softseg.nii.gz
derivatives/labels_softseg/sub-ucdavis06/anat/sub-ucdavis06_T2star_softseg.nii.gz
derivatives/labels_softseg/sub-ucdavis06/anat/sub-ucdavis06_T2w_softseg.nii.gz
derivatives/labels_softseg/sub-ucdavis07/anat/sub-ucdavis07_acq-MTon_MTS_softseg.nii.gz
derivatives/labels_softseg/sub-ucdavis07/anat/sub-ucdavis07_T1w_softseg.nii.gz
derivatives/labels_softseg/sub-ucdavis07/anat/sub-ucdavis07_T2star_softseg.nii.gz
derivatives/labels_softseg/sub-ucdavis07/anat/sub-ucdavis07_T2w_softseg.nii.gz

The file metadata already exists on your branch sb/121-move-derivatives, so it should be enough to upload the file contents to amazon and update the git-annex branch, like this (taken from step 6 of the wiki instructions):

git annex enableremote amazon public=no
git annex copy derivatives/labels_softseg/sub-{beijingGE02,beijingVerio01,beijingVerio02,beijingVerio03,ubc01,ubc03,ubc05,ucdavis03,ucdavis05,ucdavis06,ucdavis07}/*/*.nii.gz --to amazon
git annex enableremote amazon public=yes
git annex sync --no-content --only-annex

sandrinebedard commented 1 year ago

OK I've added them to amazon!

spine-generic / data-multi-subject

Move derivatives from spine-generic-processed #123

Context

Related issues

What is missing