search

coursera / dataduct

DataPipeline for humans.
Other
252 stars 83 forks source link

Configs cannot be easily stored in repo #153

Open warhammerkid opened 8 years ago

warhammerkid commented 8 years ago

I would like to store some of the configs for Dataduct inside the repo with all the custom step definitions and other pipelines. Unfortunately Config only appears to read the first file in the priority list, which means that I would also need to store all my credentials in that file, and I don't really like the idea of storing credentials in the repo.

Is there some way to achieve what I want without making any patches to the code? How do you use this at Coursera?

sb2nov commented 8 years ago

@warhammerkid currently we have a fab step internally that creates the config using a file in a private repo. But this is something we should fix soon and split the config into Config + Credentials and just merge the two YAML files in the config loader. I'll make a PR to fix this soon.

warhammerkid commented 8 years ago

Thank you so much - my Python is really rusty :smile:

kpx-dev commented 8 years ago

@warhammerkid I'm new to dataduct and pipeline in general but I think this is what you are looking for?

export DATADUCT_CONFIG_PATH=config/dataduct.yml && dataduct pipeline validate kada/rds_etl.yaml

I gitignore the config/dataduct.yml file