jdries
commented
3 years ago Indeed, this is also our experience, and it should improve. There are some ongoing things/ideas that we have to improve the situation:
- For UDF's, we added local debugging: https://open-eo.github.io/openeo-python-client/udf.html#example-downloading-a-datacube-and-executing-an-udf-locally
- Whenever someone logs a concrete error message that is hard to understand, we always try to improve it. This unfortunately has to happen on a case-by-case basis.
- We also want to expose more of our internal logging, but can't give guarantees that this will be digestible to anyone not familiar with backend internals.
- Batch job internals can be accessed through the Spark-UI, accessible through a (free) Terrascope VM: https://docs.terrascope.be/#/Developers/Hadoop/AccessingSparkLogs
- One other idea, is to allow full client-side evaluation of small jobs, using a pure-Python library. This would allow developers to bring their workflows to the point that they at least run. Unfortunately also requires more work.
Jaapel
Due to the nature of the openEO project, it is hard to debug OpenEO projects, as execution happens only on the server-side. On the client side, errors are only returned when the workflow is not set-up correctly.
Monitoring and visibility of errors in the remote backend (like pod logs) or a more extensive logging suite would benefit the speed at which we can develop the aquamonitor on OpenEO. Definitively when working with the jobs API, if the server runs into an error of any kind, a generic error is returned without any way to do debugging.