Comments on Data Processing [ReproIn/Datalad] Tutorial

[vassiki]

• [ ] It would be helpful to get people to git log immediately after creating a datalad dataset, to start exploring the commit history before any data is installed. Also good to make a note to check the log frequently.
• [ ] When we use heudiconv to convert DICOMS to NIFTI and get the data into BIDS format, do we need to change the singularity hub url?
• [ ] It would be helpful to ask people to explore the contents of .datalad as we create and install datasets and environments, only if it doesn't distract from the workflow too much.
• [ ] I always forget the difference between Reproin and Heudiconv, is it just me or is it worthwhile to add a note about each?
• [ ] Clearly instruct not to run example heudiconv command, only run the one in the task solution.
• [ ] Does every datalad command create a commit message? Is this because we are always automatically committing to git-annex?
• [ ] Reason for using inputs/rawdata nomenclature for both datalad datasets in the tutorial?
• [ ] How to access which scripts are available with datalad run-procedure? Also might be unclear about this command...
• [ ] In GLM analysis, reference to OHBM Yoda poster should be removed
• [ ] For the shell script to generate events and the fsl script, should we change url?
• [ ] Hitting roadblock at exercise for creating first level design.
• [ ] VM came with datalad version 0.10.0rc5, on updating to datalad 0.11.0, I got an error about the git annex version (error message below)

[yarikoptic]

Additional remarks

• [ ] Warnings due to all the entry points
  • a WARNING is issued

    (section2) vagrant@nitrcce:~/lesson-03-01/localizer_scans$ datalad containers-add heudiconv --url ~/images/heudiconv.simg \
    >       --call-fmt 'singularity exec {img} {cmd}'
    [WARNING] Failed to load entrypoint neuroimaging: The 'nose>=1.3.4; extra == "full"' distribution was not found and is required by datalad [__init__.py:resolve:774] 
    save(ok): /home/vagrant/lesson-03-01/localizer_scans (dataset)                  
    containers_add(ok): /home/vagrant/lesson-03-01/localizer_scans/.datalad/environments/heudiconv/image (file)
    action summary:
    containers_add (ok: 1)
    save (ok: 1)

• another WARNING

  (section2) vagrant@nitrcce:~/lesson-03-01/localizer_scans$ datalad containers-list
  [WARNING] Failed to load entrypoint neuroimaging: The 'Pillow; extra == "full"' distribution was not found and is required by datalad [__init__.py:resolve:774] 
  containers(ok): /home/vagrant/lesson-03-01/localizer_scans/.datalad/environments/heudiconv/image (file)

  • and one more

    datalad containers-run -m "Convert sub-02 DICOMs into BIDS" \
    --container-name heudiconv \
    heudiconv -f reproin -s 02 -c dcm2niix -b -l "" --minmeta -a . \
    -o /tmp/heudiconv.sub-02 --files inputs/rawdata/dicoms

• [ ] YODA comes from nowhere . may be expand on it at the top instead of at the "procedure" step

[yarikoptic]

• [x] One datalad command doesn't work (bug in datalad AFAIK):

  datalad run \
  -m "FSL FEAT analysis config script" \
  --output sub-02/1stlvl_design.fsf \
  bash -c 'sed -e "s,##BASEPATH##,{pwd},g" -e "s,##SUB##,sub-02,g" \
      code/ffa_design.fsf > {outputs}'

This one is a datalad issue, so we could upgrade

sudo apt-get update; sudo apt-get install -y git-annex-standalone; pip install --upgrade datalad[full] datalad-neuroimaging datalad-container

[djarecka]

@yarikoptic (cc: @satra ): yes, I have some issue with this datalad run command, not sure how we run this last time (because I believe we did run the script:

(section2) vagrant@nitrcce:~/lesson-03-01/glm_analysis$ datalad run -m "FSL FEAT analysis config script" --output sub-02/1stlvl_design.fsf bash -c 'sed -e "s,##BASEPATH##,{pwd},g" -e "s,##SUB##,sub-02,g" code/ffa_design.fsf > {outputs}'
[WARNING] No matching files found for 'sub-02/1stlvl_design.fsf' 
[INFO   ] == Command start (output follows) ===== 
bash: -c: line 0: syntax error near unexpected token `newline'
bash: -c: line 0: `sed -e "s,##BASEPATH##,/home/vagrant/lesson-03-01/glm_analysis,g" -e "s,##SUB##,sub-02,g" code/ffa_design.fsf > '
[INFO   ] == Command exit (modification check follows) ===== 
[INFO   ] The command had a non-zero exit code. If this is expected, you can save the changes with 'datalad save -r -F.git/COMMIT_EDITMSG .' 
Failed to run 'bash -c \'sed -e "s,##BASEPATH##,/home/vagrant/lesson-03-01/glm_analysis,g" -e "s,##SUB##,sub-02,g" code/ffa_design.fsf > \'' under '/home/vagrant/lesson-03-01/glm_analysis'. Exit code=1. out= err=(sect

[yarikoptic]

note that upgrade of datalad causes another issue, see #11