coursera / dataduct

DataPipeline for humans.
Other
252 stars 82 forks source link

Is it possible to use variables in SQL command or file? #267

Open seanadkinson opened 7 years ago

seanadkinson commented 7 years ago

Hi there, thanks for creating this product.

I'd like to re-use the same SQL command across different pipelines, providing a simple variable to change how it works. Take for example:

-   step_type: extract-rds
    host_name: db
    database: mydb
    sql: |
        SELECT id, firstName, lastName
        FROM user u
        WHERE customer = '$CUSTOMER';

Is there a way for me to share this step across pipelines, and to provide $CUSTOMER?

Thank you!

soobrosa commented 7 years ago

We solved this in our ETL framework with Ruby. https://github.com/wunderlist/night-shift/