Processing pipelines for the NDA imaging data

[mgxd]

Goal: To develop a set of processing pipelines that will be run behind the scenes on the NDAR data, and will recompute as new data streams in. The provenance information from the pipelines will be stored as NIDM and can be tracked in relation to the output.

Impact: Run common pipelines to maximize re-use of derived data.

Participants: TBA (MIT)

Deliverable: A set of Nipype workflows that can be run in the cloud.

Future development: Nipype workflows will be maintained as part of a funded R01.

[obenshaindw]

The NDA team are proposing some participants for each issue along with a list of available resources (internal and external), and possible limitations / needs.

NDA Point Person(s)

David Obenshain @obenshaindw

Available Resources

• NITRC-CE-based compute clusters, managed by StarCluster
• Some existing imaging workflows
  • C-PAC Pipeline
  • NITRC Pipeline
  • LONI NDAR Workflow
• NDA Web Service for S3 Tokens

Comments / Limitations / Needs

• Easy deployment of all dependencies; existing pipelines were somewhat difficult to get running on our own environment.
• How to manage processing cluster in AWS and scale appropriately, what is the best approach?
• How to support extension of pipelines while still maintaining provenance.
• Is Nipype limited to imaging data, or can we use the framework for other types (i.e., EEG, Eye Tracking, Omics?)
• Should returning of results from a pipeline be a base requirement?