search

dat-ecosystem-archive / gasket

Build cross platform data pipelines [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
191 stars 19 forks source link

progress API #14

Open max-mapper opened 9 years ago

max-mapper commented 9 years ago

when doing long pipelines like this:

"search-ncbi": [
    "dat cat",
    "grep Guillardia",
    "tool-stream extractProperty assemblyid",
    "bionode-ncbi download assembly -",
    "tool-stream collectMatch status completed",
    "tool-stream extractProperty uid",
    "bionode-ncbi link assembly bioproject -",
    "tool-stream extractProperty destUID",
    "bionode-ncbi link bioproject sra -",
    "tool-stream extractProperty destUID",
    "grep 35526",
    "bionode-ncbi download sra -",
    "tool-stream collectMatch status completed",
    "tee > metadata.json"
  ],

it would be really nice to get some progress events so we can render a progress bar or something....

currently it just sits there while all the subcommands run

DEBUG=* is a little better, but its pretty verbose

I'm not sure if we need a progress bar or a 'percentage done' necessarily, just some sort of output that lets users know whats going on under the hood inside the pipeline