Section 3: Neuroimaging workflows

[satra]

13:00-14:30 Neuroimaging Workflows Dorota Jarecka and Satrajit Ghosh, MIT, United States, Camille Maumet, INRIA, France

• [ ] talk 1: ReproFlow: Reusable scripts and environments, PROV
• [ ] Exercise: Run, rinse, and repeat
• [ ] talk 2: ReproEnv: Virtual machines/ContainersReproPaper, NIDM components
• [ ] Exercise: Create different environments
• [ ] [talk 3: ReproTest: Variability sources (analysis models, operating systems, software versions)]
• [ ] Exercise: Run analysis with different environments

[satra]

preview of talk1 + exercise1: @djarecka @mih @yarikoptic

• talk 1: Workflows and provenance

  • scripts and workflows
    • a script/workflow is a function
    • GUIs, bash scripts, python scripts, dataflow frameworks
    • reuse over recreate
    • modularity
    • map some use cases to workflow types
  • what and why provenance
    • what is provenance?
    • what to track?
    • how to track?
    • map some use cases to provenance details

Exercise:

1. rerun the FSL GLM workflow.
   • clear out the working directory
   • rerun the analysis and check for differences
     • what does it mean to check for differences
   • can the workflow be re-used on a different dataset?
     • what are the requirements?
2. use reprozip to rerun the workflow
   • Q for @mih - can datalad run-command use a reprozip command? what directories does it monitor (if any)?
3. [optional] what does the corresponding nipype workflow look like?
   • what are the differences?

[yarikoptic]

datalad run monitors the dataset (and its subdatasets), so as long as you produce output within that dataset somewhere -- should be recorded by datalad run

[djarecka]

talk2: containers

• what are containers?
  • isolate env
  • encapsulate env
• why do you use?
  • reproducibility
  • collaboration
  • changing hardware
  • save experiments
  • access to ready to use env
• types of containers:
  • VM: VirtualBox, AWS...
  • Containers: Docker, Singularity
• how to chose?
  • access to hardware
  • where is your data
  • docker image currently the most portable
  • CPU/GPU ?
  • basic difference between VM and Containers:
  • emulate whole computer system vs sharing the host system’s kernel
  • containers are lightweight and fast
  • using and installing docker depends on your host OS
  • why did we choose VM and Singularity/Docker for the training
• what does it mean do use container/Docker:
  • using "computer" within "computer"
  • it can be used as a part of the bash command
  • interactive session
  • mounting sources from our host computer

Exercises In Docker, Singularity or both? (example of command written in docker language)

• using Docker
  • docker pull
  • docker images, docker rmi
  • docker run (+ [-it, -p, -v, --rm])
  • docker ps
• creating new env
  • neurodocker
  • docker build (?)

[djarecka]

@satra - I didn't want to spam #15 , so moving here. All images are Singularaity, so I guess it won't help me with Docker. I'm also not completely sure, what are the results of your discussion with datalad team, I'm showing Docker, and they're using Singularity, is that right?

[satra]

you can generate docker with exactly those neurodocker commands. just replace singularity/Singularity with docker/Dockerfile

but i do think any instruction slides should have both docker and singularity options, even if the focus is on one.

@yarikoptic - i think the plan is to use singularity containers, correct?

[mih]

Using singularity containers in the datalad context simplifies things a bit. However, I do not plan to talk much about containers, but instead just use them (by name and URL), and otherwise refer to and/or rely on what @djarecka is presenting first.

[djarecka]

@satra - I know that I can use very similar command to create Dockerfile, but the whole idea of cheating was about creating quickly the image (not Dockerfile) if I have already some layers within VM.

Having Singularity image doesn't not help with building a similar Docker image, Docker still has to create all layers. Do you think I can add one Docker image to VM?

[satra]

@djarecka - as i wrote in #15 - we should add those containers internally to the VM. but i would build those in the same way with neurodocker inside the VM. perhaps i am missing something.

[djarecka]

@satra - oh, ok, I just wasn't sure if it's ok to add an additional Docker images to the VM (or we're trying to keep it as small as possible). It would be great if I have also one T1 image, so I can run bet within the container.

I will update #15