search

grailbio / reflow

A language and runtime for distributed, incremental data processing in the cloud
Apache License 2.0
964 stars 52 forks source link

-local mode for runbatch? #75

Closed olgabot closed 5 years ago

olgabot commented 6 years ago

Hello, I'd like to run some jobs using local compute, and I see there's a -local option for reflow run but no option for runbatch. Is adding this feature in the roadmap? Thank you! Olga

mariusae commented 6 years ago

Batch mode in Reflow, as it exists today, is a bit of an incongruous add-on; you're running into some of the edge cases! I'm currently re-working the scheduling mechanism in Reflow so that the same scheduler can support both "batch" and "non-batch" workloads (i.e., removes the distinction). One of the fall-outs from this work is that batch mode becomes a much simpler veneer on top of the Reflow evaluator, and isn't a special snowflake like it is today. Along with that will come more consistency, including local mode availability...

mariusae commented 5 years ago

Reflow now has the genbatch command, which translates a batch into a reflow module, which can then be run as any other, including in local mode.