Open jcohenadad opened 3 years ago
I don't think BIDS has really thought through this use case yet. Derivative works are easier to handle in a separate repository, with a hyperlink between the two, instead of making a giant repo containing everything. That seems like the fundamental source of friction to me.
We could handle this with git subtree
, but then the subdataset wouldn't be BIDS-compatible? (and the contents would still be physically duplicated, but at least easy to have git
verify the synchronization).
We could also just make it a completely separate repo and put a note spine-generic/data-multi-subject
with the link to the corrected files. That also wouldn't be BIDS-compatible, but maybe we don't need to be BIDS compatible?
I think the datalad philosophy is good to internalize here:
so, in this case, I would say the derivatives/
folder fits in as the 'preprocessed data' repo, in the middle there. In this diagram, the lines are git submodule
URLs, but we could get essentially the same workflow either with git subtree
or with just writing down a link to the more-raw dataset in the README.
Any dataset might be a derivative of any other dataset, recursively, ad infinitum. So a single derivatives/
folder is never going to be able to model every processing or analysis you might do with a dataset. It should be possible to accomplish all the same work while avoiding derivatives/
entirely, and thus avoiding having our hands tied by it.
The manuals segmentations generated in spine-generic-processed
are:
(I can not edit your issue 😕)
Just some context about the spine-generic-processed
dataset:
The reason for spine-generic-processed
was that the derivatives created are generated from the resampled and reoriented images (the current derivatives in data-multi-subject also had non BIDS compatible suffix)
I copied the *_csfseg-manual.nii.gz
and *_gmseg-manual.nii.gz
since it isa futur plan to also add all GM seg in spine-generic-processed
for sct_deepseg GM seg training
Thank you for your feedback @sandrinebedard and @kousu. I agree it is unrealistic to have a single derivatives/ folder that will accommodate all related research for the next 50 years 😅 . So what we should probably do, for the benefits of the community, is "simply" publish the spine-generic-processed
as a new BIDS directory. It is similar to the scheme described in https://github.com/spine-generic/data-multi-subject/issues/94#issuecomment-891314347, but not exactly the same because we won't have the sub-module features (which is OK for now... we are still learning to get acquainted with all these technologies).
It sounds like the big worry here is denormalization. So, can we just not denormalize? Instead of spine-generic-processed
, rename it to, say, spine-generic-derivatives
or data-multi-subject-derivatives
, delete every duplicate of a file here, and probably just forget using the derivatives/
folder. Then there's only one unique place for everything.
There could still be some sync issues, but we can detect them by comparing the git timestamps of data-multi-subject
to the corresponding timestamps in data-multi-subject-derivatives
.
There are currently this many overlapping images between the two:
$ fdupes -1 -r data-multi-subject spine-generic-processed/ | grep .nii.gz | wc -l
1130
(I cut out .bval, .bvec and .json files because many of them are themselves duplicated within the datasets, and would overestimate the problem)
There's already been quite a lot of drift between the two datasets:
Some files were added or never copied (I can't tell without looking deeper):
and here's the files that are new, from Sandrine et cie.'s work
(full set, too large to fit in a comment: right.txt)
I don't understand BIDS! Why are some things in the latter set under derivatives/
and some not?
Anyway I think we could probably replace spine-generic-processed
with just the set of files in this last list. Then the two datasets would collectively be normalized.
Anyway I think we could probably replace spine-generic-processed with just the set of files in this last list. Then the two datasets would collectively be normalized.
sorry i'm not sure i understand. I think we need to set up a meeting to discuss this-- GH has its limitations when it comes to explaining stuff, especially given that we don't all come from the same background (misunderstandings often happen, and it would be problematic in this case as it could negatively impact research projects down the line). Let's make sure we are all on the same page.
We should replace spine-generic-processed
with its subset of files listed in https://github.com/spine-generic/data-multi-subject/files/7611055/right.txt.
because doing that achieves https://en.wikipedia.org/wiki/Database_normalization
A derivatives to this dataset was created for the contrast-agnostic-sofseg project. The derived dataset is called
spine-generic-processed
. It is currently hosted in the internal server at NeuroPoly Lab. However, at some point, we will make it public (tagging @kousu ).From the
spine-generic-processed
derived dataset, the following manual segmentations were created:Also, the following manual segmentations were duplicated from this repository, to the derived internal repository:
*_gmseg-manual.nii.gz
: GM manual segmentations.The problem is that, by having duplicated data at two different locations, if at some point someone wishes to update the manual segmentation on one of the two database, they won't be in sync anymore, causing possible conflicts and unnecessary future efforts (eg: one researcher would manually correct GM of
sub-ubc02
while it has already been done on the other dataset).We should absolutely avoid this situation. Few ideas:
spine-generic-processed/derivatives/labels/sub-amu01/anat/sub-amu01_T2star_seg-manual.nii.gz