snowplow / dataflow-runner

Run templatable playbooks of Hadoop/Spark/et al jobs on Amazon EMR
http://snowplowanalytics.com
19 stars 8 forks source link

Logs are not reported in case of non-0 exit codes #31

Closed BenFradet closed 6 years ago

BenFradet commented 7 years ago

Despite the fact that the logs are outputted to stderr when running locally, they do not seem to be picked up by factotum in case of an error (non-0 exit codes).

alexanderdean commented 7 years ago

Originally reported by @alexanderdean

alexanderdean commented 6 years ago

Let's keep this in actually.

BenFradet commented 6 years ago

Ok so, factotum does pick up the logs:

[ec2-user@ip-172-31-19-39 df-runner]$ factotum run df-runner.factfile 
Task 'echo alpha' was started at 2018-01-24 14:47:47.268593587 UTC
Task 'echo alpha' stdout:
alpha
Task 'echo alpha': succeeded after 0.0s
Task 'dataflow-runner' was started at 2018-01-24 14:47:47.268640177 UTC
Task 'dataflow-runner': succeeded after 0.0s
2/2 tasks run in 0.0s
Task 'dataflow-runner' stderr:
time="2018-01-24T14:47:47Z" level=warning msg="lock already held at test-lock"
lock already held at test-lock
Factotum job finished early as a task ('dataflow-runner') requested an early finish. The following tasks were not run: .

And they also appear in factotum.log:

2018-01-24T14:47:47Z INFO factotum::factotum::executor::execution_strategy - task 'dataflow-runner' stderr:
'time="2018-01-24T14:47:47Z" level=warning msg="lock already held at test-lock"
lock already held at test-lock'
2018-01-24T14:47:47Z INFO factotum::factotum::executor - 'dataflow-runner' returned 17 in Duration { secs: 0, nanos: 7560347 }

so the problem doesn't appear to be fixable from df-runner.

BenFradet commented 6 years ago

Works in console, closing as not an issue:

https://console.snowplowanalytics.com/20d9c178-15a6-4049-a77d-e0a22a22609f/scheduled-jobs/com.snowplowanalytics/a2e2672c3cc1e5ef701bfd0abcfeb8b50657aa2c255dc434b66823e092f3514a/dataflow-runner

alexanderdean commented 6 years ago

Thanks for digging into this.