search

coursera / dataduct

DataPipeline for humans.
Other
252 stars 82 forks source link

Add support for preconditions in ShellCommandActivity, allow custom s3 paths in preconditions #245

Open p5k6 opened 8 years ago

p5k6 commented 8 years ago

reference PR #244 Background: we're setting up an event-driven pipeline, where our Ops team delivers a file to a set s3 location, and should be run when delivered. Adding support for preconditions in ShellCommandActivity allows us to build a custom step which begins execution when a path (using S3KeyExists) is available in s3.

I'll post up an example (probably tomorrow) which hopefully will explain/show a bit better :)

p5k6 commented 8 years ago

here's the custom step and an example script that show how we've set up an event driven pipeline. When our ops team delivers to s3://my-s3-bucket-name/app/qa/full/#{format(@scheduledStartTime, 'YYYY-MM-dd')}/, then the pipeline kicks off.

This does only work where dataduct has pr #244 built (our staging branch has merged it for us)