OpenNeuro dataset preparation

[yarikoptic]

TODOs/aspects/concers

dups

• [x] remove __dup files we didn't want to deface dups
  • [x] func spacetop_prep/datalad/remove_dups_func.sh
  • [x] fmap: compare time and delete earlier
    • [x] jq .IntendedFor sub-/ses-/fmap/__dupjson
  • [x] anat: compare images and determine quality

    de-identify (redacted branch)

• [x] _scans.tsv ended up in git, not git-annex with original dates.Sanitize the dates
  • [x] shift by some number of dates following https://bids-specification.readthedocs.io/en/stable/common-principles.html#units instructions
  • [x] double check that no dates in side car json files.

    ❯ grep '\<20[12][0-9]\>' */**/*.json */**/*.tsv | grep -v -e ConversionSoftwa -e InstanceCreationTime -e ContentTime -e AcquisitionTime -e LargestImagePixelValue -e SAR -e WipMemBlock -e '\.bval' -e WindowCenter

    came empty

  • [ ] potentially "graft" in a new history. A related question asked to git-annex forum
  • [ ] remove sourcedata and .heudiconv (there is a version on yariks laptop)
• [ ] deface the anatomicals (we have bidsonym bids-app now -- give it a shot or find what Yarik used)
• [x] make sure that we have all participants converted: a few were found in the spacetop .tsv but not convert. Done in convert-missing-subjects branch on rolando: ongoing 7/10?
  • [x] 4 just need conversion: 0105 ses 01, 0103 ses 02, 0128 ses 04, 0126 ses 04
  • [x] ??? need figuring out where they were converted to (had incorrect subject metadata)
  • [x] rewrite commits so we have fake dates in _scans.tsv for them as well
• [x] integrate behavioral (_events.tsv /.json) data
• [ ] (maybe) rename tasks (would entail renaming files) → rename_social.sh

ultimately, share

• [ ] create empty dataset using openneuro CLI → we have a code for this. we just need to implement
• [ ] datalad push or upload otherwise (depends on how we deal with _scans files)
• [ ] "publish" on openneuro (make it public)

Phenotype

• [ ] populate participants.tsv
• [x] BIDS behavioral
  • [ ] Via behavioral script, we were able to see which folders weren't converted.
  • [ ] Once missing conversion is done...datalad run -m "DEV: convert task-faces data to BIDS" python ./code/spacetop-prep/spacetop_prep/events/events_faces.py
• [ ] BIDS phenotype