snowplow / dataflow-runner

Run templatable playbooks of Hadoop/Spark/et al jobs on Amazon EMR
http://snowplowanalytics.com
19 stars 8 forks source link

Add steps' start/finish timestamps to log messages #82

Closed spenes closed 2 years ago

spenes commented 2 years ago

With transient clusters, start/finish timestamps of cluster steps can be deduced from log timestamp previously. But currently it is not possible because all the step logs are displayed after cluster is terminated and timestamp of the step log doesn't reflect actual time step is completed. In order to get this information back, we will include start/finish timestamps of steps to log messages.