WPRDC / wprdc-etl

MIT License
8 stars 3 forks source link

Remove configuration file handling from pipeline #55

Closed saylorsd closed 7 years ago

saylorsd commented 8 years ago

@bsmithgall The CLI command used a config file which I haven't accounted for yet, so #56 will fail the build test. I was thinking of parsing the config file within the scripts code and then sending them as individual arguments to the pipeline. Maybe along the lines of moving the pipeline.set_config_from_file() code to pipeline.scripts and then writing a new function in pipeline to handle a dict of arbitrary settings?

Or.... now that I type this all out, maybe it would make more sense to keep a lot of the old config file stuff in pipeline, but rework it a bit so that using a file is optional. I think I'll close #56 and try that.