search

coursera / dataduct

DataPipeline for humans.
Other
252 stars 82 forks source link

Support for new schedule types on-demand and timeseries #237

Open kewlkiev opened 8 years ago

kewlkiev commented 8 years ago

Couldn't find any feature in dataduct for http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-concepts-schedules.html#dp-concepts-ondemand

Which can activate a pipeline on trigger, rather than on a schedule.

A design which I discussed with @everanurag was to have current schedule.py being renamed to cronschedule.py and this in turn inherits schedule.py. which will be parent object for ondemand,cron and timeseries.

Let me know what you guys think?

kewlkiev commented 8 years ago

Currently Dataduct's steps are very tightly coupled to schedule, which means for feature like on-demand which requires no schedule reference in any step, It involves changing validation in lots of places if I make schedule object as None. @idralyuk @everanurag , can you please raise a pull request for the interim fix.