snowplow / factotum

A system to programmatically run data pipelines
http://snowplowanalytics.com/blog/2016/04/09/introducing-factotum-data-pipeline-runner/
221 stars 12 forks source link

Add converter from Makefile to a Factotum factfile #5

Open alexanderdean opened 8 years ago

alexanderdean commented 8 years ago

We need this so we can start migrating makefiles over to Factotum.

$ factotum job make2factotum FILE

where FILE is a minimalistic Makefile, containing only the following elements:

Example of a minimalistic Makefile that will be convertable:

pipeline=acme

done: run-web-sql
    /notify-v0.2.0.sh $(pipeline) "Completed successfully"

start:
    /notify-v0.2.0.sh $(pipeline) "Started"
check-lock: start
    /check-lock.sh $(pipeline)
emr-etl-runner: check-lock
    /r73/emr-etl-runner-r73-rc2.sh $(pipeline) && /notify-v0.2.0.sh $(client) "Ran EmrEtlRunner"
storage-loader: emr-etl-runner
    /storage-loader-r73-rc2.sh $(pipeline) && /notify-v0.2.0.sh $(pipeline) "Ran StorageLoader"
run-dedupe-sql: storage-loader
    /sql-runner-0.2.0-lock.sh $(pipeline) dedupe
run-web-sql: run-dedupe-sql
    /sql-runner-0.2.0-lock.sh $(pipeline) web
alexanderdean commented 8 years ago

Depends on https://github.com/snowplow/makefile-rs

alexanderdean commented 8 years ago

Bringing forwards...