search

coursera / dataduct

DataPipeline for humans.
Other
252 stars 83 forks source link

Artifacts not getting uploaded to s3 bucket on validate command #221

Open abhishekmahawar opened 8 years ago

abhishekmahawar commented 8 years ago

Hi, I have a requirement to schedule a pipeline at a frequency of 30 minutes. Since, dataduct supports only three values of frequency(one time, daily, hourly), I thought of not activating the pipeline but just validating it and then manually update the schedule from AWS console to run every 30 minutes. But looks like validate command is not uploading the required artifacts to s3 and so the actual execution of pipeline fails.

talhatec commented 7 years ago

Hey, were you able to get the S3 uploads working? im getting similar issues.

sb2nov commented 7 years ago

https://github.com/coursera/dataduct/blob/master/dataduct/pipeline/schedule.py#L39 just adding a new field here should be able to fix this.