coursera / dataduct

DataPipeline for humans.
Other
252 stars 83 forks source link

How to use @scheduledstartTime in sql-command? #192

Closed workanandr closed 8 years ago

workanandr commented 8 years ago

I have tried with both the following code, but they are not working. Any help is appreciated. Thanks.

step_type: sql-command
    command: |
        unload ('select * from tmp_tbl2') to
        's3://mybucket/data/#{format(@scheduledStartTime, 'YYYY-MM-dd-HH-mm-ss')}/'
        credentials 'aws_access_key_id=xxxxxxx;aws_secret_access_key=yyyyyy';

step_type: sql-command
    command: |
        unload ('select * from tmp_tbl2') to
        's3://mybucket/data/#{@scheduledStartTime}/'
        credentials 'aws_access_key_id=xxxxxxx;aws_secret_access_key=yyyyyy';
sb2nov commented 8 years ago

Hi there are basically two ways you can pass arguments to SQL commands Using the sql-command

-   type: sql-command
    script_arguments:
    -   '#{format(minusDays(@scheduledStartTime,1),"YYYY-MM-dd")}'
    command: |
        DELETE FROM prod.page_views WHERE page_view_dt = ?;

and then otherwise if you're using create-update-sql step then

-   step_type: create-update-sql
    script_arguments:
    -   '#{format(minusDays(@scheduledStartTime,1),"YYYY-MM-dd")}'
    command: |
        DELETE FROM prod.page_views WHERE page_view_dt = %s;
    table_definition: tables/prod.page_views.sql