coursera / dataduct

DataPipeline for humans.
Other
252 stars 83 forks source link

Unable to find SQL command file in specified S3 location #212

Open donigian opened 8 years ago

donigian commented 8 years ago

I'm getting the following error:

The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: 129BF648B7185xxx)

Using this configuration: ➜ dataduct cat example_sql_command.yaml

name: example_sql_command
frequency: one-time
load_time: 01:00  # Hour:Min in UTC

description: Example for the sql_command step

steps:
-   step_type: sql-command
    command: SELECT count(*) FROM lookup.test_dp2;

Does the validate or activate dataduct command create the sql command file in S3 (s3://xxx/yyy/xxx/yyy/src/example_sql_command/version_20160103071xxx/SqlCommandStep0/file)?

I don't see a SQL command file in S3 though the permissions for both DataPipelineDefaultRole & DataPipelineDefaultResourceRole include:

...
"s3:Get*",
"s3:List*",
"s3:Put*",
...

Using these configs ~/.dataduct/dataduct.cfg

redshift:
    CLUSTER_ID: xxx
    DATABASE_NAME: xxx
    HOST: xxx
    PASSWORD: xxx
    USERNAME: xxx
    PORT: 5439
logging:
    CONSOLE_DEBUG_LEVEL: INFO
    FILE_DEBUG_LEVEL: DEBUG
    LOG_DIR: ~/.dataduct
    LOG_FILE: dataduct.log
etl:
    REGION: us-east-1
    S3_ETL_BUCKET: xxx
    S3_BASE_PATH: xxx
    ROLE: DataPipelineDefaultRole
    RESOURCE_ROLE: DataPipelineDefaultResourceRole
mysql:
    host_alias_1:
        HOST: FILL_ME_IN
        PASSWORD: FILL_ME_IN
        USERNAME: FILL_ME_IN
ec2:
    INSTANCE_TYPE: m1.small
    ETL_AMI: ami-05355a6c
    SECURITY_GROUP_IDS: xxx
    SUBNET_ID: xxx
emr:
    MASTER_INSTANCE_TYPE: m1.large
    NUM_CORE_INSTANCES: 1
    CORE_INSTANCE_TYPE: m1.large
    CLUSTER_AMI: 3.1.0