search

Open-EO / openeo-geopyspark-driver

OpenEO driver for GeoPySpark (Geotrellis)
Apache License 2.0
25 stars 4 forks source link

regression: YARN diagnostics not returned in job error logs #729

Open bossie opened 3 months ago

bossie commented 3 months ago

292 enabled logging of YARN errors in addition to in-job errors but they no longer show up when fetching a job's logs in OpenEO.

For example, this integration test failed but according to the logs, there are no errors:

AssertionError: Assertion for batch job <BatchJob job_id='j-240314c2fb984b22a906da442c2a0b31'> failed:
 Job failed
Job status: error
Kibana logs: https://kibana-infra.vgt.vito.be/app/kibana#/discover?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-7d,to:now))&_a=(columns:!(message,levelname,filename),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'592a42f0-e665-11ec-8cc4-3747d5233c59',key:levelname,negate:!f,params:(query:ERROR),type:phrase),query:(match:(levelname:(query:ERROR,type:phrase))))),index:'592a42f0-e665-11ec-8cc4-3747d5233c59',interval:auto,query:(language:kuery,query:'job_id%20:%20%22j-240314c2fb984b22a906da442c2a0b31%22%20'),sort:!(!('@timestamp',desc)))

Job error logs: []

The missing log line ended up in the new openeo-terrascope-jobtracker-000001 index: yarn_diagnostics_new_index