dat-ecosystem-archive / gasket

Build cross platform data pipelines [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
191 stars 19 forks source link

conditional process trigger #22

Closed cdaringe closed 8 years ago

cdaringe commented 8 years ago

hi all:

i'm interested in integrating gasket into our neuro imaging analysis project.

TWO PREGUNTAS!

1 we have a pipeline need as such:

=> client runs taskA => client runs taskB ( system watcher picks up on Bs output ) ........=> remote computer aggregates taskB results (some ML routines w/ other client results) => client runs taskC (using result from remote computation)

2 i'm assuming gasket is stateless. e.g. taskB fails. system comes back up, tries again, and replays from taskA. correct?

discussion.

i was considering making my own pipeliner (which may be required if it's too out of scope for gasket to achieve these goals), but of course, i'd rather chat it out first!

in the end i'm going to have to produce something that has the following effect:

    "example": [
      {
        "command": "echo hello world",
        "type": "pipe"
      },
      // now i intercept the flow here, and store some state about my progress through the pipeline... proceed :)
      {
        "command": "transform-uppercase",
        "type": "cmd",
      },
// once again i intercept the flow, and do some remote data computation/synchronization
      {
        "command": "crazy-async-xform",  // doesn't run immediately.  gasket sets up a listener because of `onEvent` then waits.  my app then emits the event, feeds the data in, and it proceeds
        "type": "cmd",
        "onEvent": "doCrazyXform"
      },
    ]