Pydesperate times call for Pydrastic measures

[effigies]

Project info

Title: Pydesperate times call for Pydrastic measures

Project lead: Dorota Jarecka (MM: @dorota) and Chris Markiewicz (T: @effigies, MM: @markiewicz)

Project collaborators:

Registered Brainhack Global 2020 Event: Montreal

Project Description: Nipype is a Python library that provides a uniform interface to existing neuroimaging software and facilitates interaction between these packages within a single workflow. It forms the basis of widely-used pipelines such as C-PAC and fMRIPrep.

Pydra is a new workflow engine, written from scratch for Nipype 2. Pydra is reaching the maturity needed to justify upgrading workflows from Nipype 1, and it is time to build out the remainder of the ecosystem.

In this hackathon, we will focus on building tasks (interfaces) for neuroimaging tools that can be coordinated by Pydra, and workflows, sequences of tasks that meet real needs in neuroimaging.

If you're familiar with Nipype 1 and would like to flesh out a set of tasks (e.g., AFNI), we have a tasks template that you can use. If you would rather focus on a workflow, we encourage you to use the niflow manager to begin to create a workflow.

Please read the contribution guide for tips on getting started and our policies on acknowledging contributions.

Data to use: This project is not focused on any specific dataset, but participants are welcome to bring their own data or public data for testing and demonstrations. For MRI-based workflows, we can recommend OpenNeuro ds000114 for different tasks and modalities.

Link to project repository/sources: https://github.com/nipype/pydra

Goals for Brainhack Global 2020:

• New tasks giving Pydra users access to common neuroimaging functions
• Example workflows that can be run independently or included in larger workflows
• Improved documentation for new users
  • Migration guide for users of other tools, including Nipype 1
• Creating a dashboard for Pydra
• Improving Slurm execution of the workflows
• Datalad-aware data fetching and export
• Creating Pydra command-line interface (CLI)
• Dask integration

Good first issues:

1. Running and improving tutorial
2. Improving Documentation
3. Helper functions to promote Python functions to Pydra tasks
4. Cli for reading the results
5. Good first issues (to see live)

Skills:

• Required: Some Python
• Nice to have: experience with writing workflows, e.g. with Nipype 1, Bash, Snakemake, Makefiles, CWL
• Recommended: Reasonable comfort with git

Tools/Software/Methods to Use:

• Python 3.7+ (we recommend Anaconda to manage Python environments)
• Whatever tools you want to coordinate (FreeSurfer, AFNI, Julia, etc)

Communication channels:

• Chat: https://mattermost.brainhack.org/brainhack/channels/nipype
• Video: (See channel header)

Project labels

• Type of project:

  documentation, #method_development, #pipeline_development

• Project development status:

  2_releases_existing

• Topic of the projet:

  reproducible_scientific_methods, #workflows

• Tools used in the project:

  Nipype, #AFNI, #ANTs, #BIDS, #Brainstorm, #Datalad, #DIPY, #FieldTrip, #Freesurfer,

  FSL, #MNE, #MRtrix, #SPM

• Tools skill level required to enter the project (more than one possible):

  comfortable, #expert, #familiar

• Programming language used in the project:

  Python, #containerization, #documentation, #shell_scripting, #Unix_command_line,

• Modalities involved in the project (if any):

  behavioral, #DWI, #ECG, #ECOG, #EEG, #eye_tracking, #fMRI, #fNIRS, #MEG, #MRI, #PET, #TDCS, #TMS

• Git skills required to enter the project (more than one possible):

  1_commit_push, #2_branches_PRs, #3_continuous_integration

• [x] I added all of the labels I want an associate to my project

Project Submission

Submission checklist

Once the issue is submitted, please check items in this list as you add under ‘Additional project info’

• [x] Link to your project: could be a code repository, a shared document, etc.
• [x] Goals for Brainhack Global 2020: describe what you want to achieve during this brainhack.
• [x] Flesh out at least 2 “good first issues”: those are tasks that do not require any prior knowledge about your project, could be defined as issues in a GitHub repository, or in a shared document.
• [x] Skills: list skills that would be particularly suitable for your project. We ask you to include at least one non-coding skill. Use the issue labels for this purpose.
• [x] Chat channel: A link to a chat channel that will be used during the Brainhack Global 2020 event. This can be an existing channel or a new one. We recommend using the Brainhack space on Mattermost.

Optionally, you can also include information about:

• [ ] Number of participants required.
• [ ] Twitter-sized summary of your project pitch.
• [ ] Provide an image of your project for the Brainhack Global 2020 website.

We would like to think about how you will credit and onboard new members to your project. If you’d like to share your thoughts with future project participants, you can include information about:

• [ ] Specify how you will acknowledge contributions (e.g. listing members on a contributing page).
• [ ] Provide links to onboarding documents if you have some:

[PeerHerholz]

Ah, sorry @effigies, I think I forgot to mention that the things under Project Labels should have a # for the github bot to find it IIRC.

[complexbrains]

I am on it now editing the issue. Will be ready in a minute !

[complexbrains]

Ok it seems like the issue is ready to be published. I will publish it now but you can make edit and changes as necessary as you want throughout the time! Thank you for submitting the project! :tada:

[htwangtw]

My aim was to understand how to use pydra to create a workflow with python functions, using some code from nilearn to downsample imaging data. Here's the script demo. Passing Nifti1 image is not recommanded. Now I am working on writing some pydra FSL interface, starting with manual creation with fslmaths. Now understood the process, I will try to use the converter for more efficient interface creation.

[effigies]

I've been keeping some notes as we've worked through issues this week. Probably won't be able to make the wrap-up session.

[harveyaa]

I've been working on integrating pydra into nixtract https://github.com/danjgale/nixtract, a command line tool for timeseries generation from preprocessed fMRI data. I've run into an issue with passing a custom class FunctionalImage as input to a task, modifying it and returning it as an output, I haven't succeeded in finding a workaround yet but I will continue with the project after brainhack and will keep in touch :)

[axiezai]

Finally merged the first mrtrix3 pydra task at https://github.com/nipype/pydra-mrtrix3, mrconvert is a fundamental tool in mrtrix3 that manages all sorts of image conversions. The next step is to create a script that translates nipype interfaces for mrtrix3 over while creating unit tests. Will be taking inspirations from https://github.com/nipype/pydra-fsl :)