search

DAGWorks-Inc / hamilton

Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage and metadata. Runs and scales everywhere python does.
https://hamilton.dagworks.io/en/latest/
BSD 3-Clause Clear License
1.56k stars 92 forks source link

@config.custom for clearer/more powerful config parsing #143

Open elijahbenizzy opened 1 year ago

elijahbenizzy commented 1 year ago

Is your feature request related to a problem? Please describe. @config gives no insight/structure into what config items are required, as it takes in the whole thing. Its meant to be internal but exposed externally.

Describe the solution you'd like @config.custom would take in the same argument as resolver in @resolve

Describe alternatives you've considered Keeping the current one, not supporting custom functions, keeping the current one as a fallback.

Additional context See Peter Robinson's comments here: https://hamilton-opensource.slack.com/archives/C03M33QB4M8/p1682076384844779

elijahbenizzy commented 1 year ago

Timeboxed this but there's some weirdness about how we enforce the config that made me not want to deal with this yet...

elijahbenizzy commented 1 year ago

(intended) fix for the immediate issue here, this is a follow-up.