search

nipy / nipype

Workflows and interfaces for neuroimaging packages
https://nipype.readthedocs.org/en/latest/
Other
745 stars 528 forks source link

Optimize data transfer between pure python nodes #86

Open chrisgorgo opened 13 years ago

chrisgorgo commented 13 years ago

Currently when two pure python nodes (like nipy GLM fit and contrast estimate) are connected (A->B) node A saves the data to file and node B reads it. The reading phase could be skipped by sending the data directly to node B. This, however, creates some problems:

  1. It will not work with "timestamp" hashing.
  2. Big chunks of data will have to be kept in memory until receiving nodes will be executed.
djarecka commented 6 years ago

@chrisfilo - do you have any new comments on this issue?

chrisgorgo commented 6 years ago

Nope.

satra commented 6 years ago

this may be supported as part of efficiency implementation in 2.0 workflow engine, but will require some complex logic to implement properly. currently low priority.