Cron in the Cloud (for data scraping and processing) - Githubissues

rufuspollock / ideas

Ideas for (tech) stuff to research, build or work on.

https://rufuspollock.com/

50 stars 4 forks source link

Cron in the Cloud (for data scraping and processing) #81

Open rufuspollock opened 10 years ago

rufuspollock commented 10 years ago

Various possible existing implementations:

scraperwiki
(hack) travis
jenkins
Original Proposal

From: http://ideas.okfn.org/ideas/377/cron-in-the-cloud/ (2011)

Want a service where I can give it a piece of code (including a pointer to git or mercurial repo) and have it run that code on a regular basis.

Wishlist of features:

Language: python, js (node.js)
Notification of run and results
Store results (locally or remotely (files and databases))
Logging support (possibly using XUnit or TAP)
Workflow support (do this after that has run successfully)
Spin up workers on demand

This is result of long-running discussions of Friedrich Lindenberg and Rufus Pollock and is almost certainly not original. (See also this post: http://notebook.okfn.org/2011/04/18/25k-spending-etl-process/)

gr33ndata commented 10 years ago

One other existing implementation is morph.io