ssec-jhu / dplutils

Distributed(Data) Pipeline Uitilities
BSD 3-Clause "New" or "Revised" License
1 stars 0 forks source link

better handling of yaml config, including from cli #64

Closed amitschang closed 1 month ago

amitschang commented 5 months ago

There is a way to get config from yaml in executor, but there is no hookup in the cli utils. Also, the current config does not also deal with context, which feels awkward. we might want something like:

---
context:
  ctxvar: someval
  ctxarr:
  - 1
  - 2
taskconfig:
  task:
     kwargs: ...

qualifying it like this opens up possibility of pipelines defined entirely in yaml, where the task graph and executor also set there.