Placeholder for feature parity with EmrEtlRunner

snowplow / dataflow-runner

Run templatable playbooks of Hadoop/Spark/et al jobs on Amazon EMR

http://snowplowanalytics.com

19 stars 8 forks source link

Placeholder for feature parity with EmrEtlRunner #11

Closed alexanderdean closed 7 years ago

alexanderdean commented 7 years ago

Particularly around robustness.

This is a bit of an unfair ticket - it involves:

Doing a detailed review of the EmrEtlRunner EMR invocation and monitoring code
Identifying any missing features in Dataflow Runner
Adding tickets into 0.2.0
Closing this ticket and de-scheduling it from the 0.2.0 milestone

BenFradet commented 7 years ago

So far:

EBS: #4
Apps: #6
Ability to inspect step status and act on it (I don't really know it'll translate to df-runner yet)

alexanderdean commented 7 years ago

Ability to inspect step status and act on it (I don't really know it'll translate to df-runner yet)

I think this is fiddly but doable, but will need further thought...

BenFradet commented 7 years ago

Having given some thought to the last point, I'm not sure it really applies to df-runner since it seems really snowplow specific. What do you think @alexanderdean ?

alexanderdean commented 7 years ago

Hmm - I hear you. Let's break out the last point into a separate ticket "Explore options around ..." and put it into 0.3.0. This way gives us some more time to consider different options and whether we must have this in Dataflow Runner or if there is another way...