feature/pipeline - Githubissues

hofstadter-io / cuetils

CLI and library for diff, patch, and ETL operations on CUE, JSON, and Yaml

BSD 3-Clause "New" or "Revised" License

78 stars 6 forks source link

feature/pipeline #35

Closed verdverm closed 2 years ago

verdverm commented 2 years ago

implements the pipeline command and library
based on cue/flow DAG engine, tasks found by attributes
- [x] @pipeline(name,[tags]) and @task([task/type],[tags])
supports many ops
- [x] cuetils/structural
- [x] @log (named @print currently)
- [x] cue/tools/...
- [x] api requests
- [x] api server (?)
- [x] db conn / call
- [ ] background commands (bg, fg, kill)

questions:

how to compose pipelines and still find tasks?
getting the final value?
is a pipeline a task as well?
progress reporting via chans?

later / follow up issues:

drawing the pipeline?

verdverm commented 2 years ago

did a bunch of other stuff, will be writing docs to cover the features & tasks later.

For now, see the examples/flow directory