Closed jdries closed 1 year ago
@jdries When I test, I do get a nice error message. Not the unclear message. Example code:
import openeo
eoconn = openeo.connect("openeo-dev.vito.be").authenticate_oidc()
udf_code = """
from openeo.udf import UdfData, StructuredData
def transform(data: UdfData) -> UdfData:
raise Exception("This error message should be visible to user")
res = [
StructuredData(description="res", data=[x * x for x in sd.data], type="list")
for sd in data.get_structured_data_list()
]
data.set_structured_data_list(res)
"""
datacube = eoconn.load_collection(
"SENTINEL1_GRD",
spatial_extent={"west": 16.06, "south": 48.06, "east": 16.15, "north": 48.15},
temporal_extent=["2017-03-01", "2017-04-01"],
bands=["VV", "VH"]
)
udf = openeo.UDF(udf_code)
rescaled_cube = datacube.apply(process=udf)
rescaled_cube.download("test.tiff")
Output end with this nice error message:
Is there a way I can better test this ticket? I have the feeling that the SparkLiStener was not needed to make this error message.
That's good already, maybe also try with batch jobs?
When running with a batch job, there are many logs, where the thrown error is shown 17 times. But they get logged as info and debug level, while the SparkListener logs I added as error level. So Maybe my change did not have an impact
Snippet used to run as batch job:
...
rescaled_cube = datacube.apply(process=udf)
# rescaled_cube.download("test.tiff")
# Creating a new job at the back-end by sending the datacube information.
job = rescaled_cube.create_job()
# Starts the job and waits until it finished to download the result.
job.start_and_wait()
job.get_results().download_files("output")
Batch job on creo just gave me a better message, like the one below. There's still a lot of less usefull info in the stack traces. I'm wondering if we can maybe put the most important part first. In this case, this would be: FileNotFoundError: [Errno 2] No such file or directory: '/eodata/auxdata/SRTMGL1/dem/N64E024.SRTMGL1.hgt.zip
The part below org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException
is also quite useless.
{
"id": "[1675450345603, 653605]",
"time": "2023-02-03T18:52:25.603Z",
"level": "error",
"message": "LogErrorSparkListener.onTaskEnd(...) error: org.apache.spark.api.python.PythonException: Traceback (most recent call last):\n File \"/usr/local/spark/python/lib/pyspark.zip/pyspark/worker.py\", line 619, in main\n process()\n File \"/usr/local/spark/python/lib/pyspark.zip/pyspark/worker.py\", line 611, in process\n serializer.dump_stream(out_iter, outfile)\n File \"/usr/local/spark/python/lib/pyspark.zip/pyspark/serializers.py\", line 132, in dump_stream\n for obj in iterator:\n File \"/usr/local/spark/python/lib/pyspark.zip/pyspark/util.py\", line 74, in wrapper\n return f(*args, **kwargs)\n File \"/opt/openeo/lib/python3.8/site-packages/epsel.py\", line 44, in wrapper\n return _FUNCTION_POINTERS[key](*args, **kwargs)\n File \"/opt/openeo/lib/python3.8/site-packages/epsel.py\", line 37, in first_time\n return f(*args, **kwargs)\n File \"/opt/openeo/lib/python3.8/site-packages/openeo/util.py\", line 362, in wrapper\n return f(*args, **kwargs)\n File \"/opt/openeo/lib/python3.8/site-packages/openeogeotrellis/collections/s1backscatter_orfeo.py\", line 794, in process_product\n dem_dir_context = S1BackscatterOrfeo._get_dem_dir_context(\n File \"/opt/openeo/lib64/python3.8/site-packages/openeogeotrellis/collections/s1backscatter_orfeo.py\", line 258, in _get_dem_dir_context\n dem_dir_context = S1BackscatterOrfeo._creodias_dem_subset_srtm_hgt_unzip(\n File \"/opt/openeo/lib64/python3.8/site-packages/openeogeotrellis/collections/s1backscatter_orfeo.py\", line 664, in _creodias_dem_subset_srtm_hgt_unzip\n with zipfile.ZipFile(zip_filename, 'r') as z:\n File \"/usr/lib64/python3.8/zipfile.py\", line 1251, in __init__\n self.fp = io.open(file, filemode)\nFileNotFoundError: [Errno 2] No such file or directory: '/eodata/auxdata/SRTMGL1/dem/N64E024.SRTMGL1.hgt.zip'\n\n\tat org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:545)\n\tat org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:703)\n\tat org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:685)\n\tat org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:498)\n\tat org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)\n\tat scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)\n\tat scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)\n\tat scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)\n\tat scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:511)\n\tat scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)\n\tat scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)\n\tat scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)\n\tat scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:511)\n\tat scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)\n\tat scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:489)\n\tat org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:140)\n\tat org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)\n\tat org.apache.spark.scheduler.Task.run(Task.scala:131)\n\tat org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)\n\tat org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1462)\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\n"
},
Non-batch jobs can show a nice and to-the point error message. Would it be possible to show the same kind of error for batch-jobs? Considering the more complex job pipeline.
Adding more logs to the already enormous output might still keep it hard to use for users that don't have acces to Kibana.
Errors that happen before running in the executors gets shown nicely in batch and non-batch job requests.
FYI: verbose log printing by python client for batch jobs got handled here: https://github.com/Open-EO/openeo-python-client/issues/332
Errors now look like this to the user:
OpenEO batch job failed: UDF Exception during Spark execution: File "<string>", line 8, in transform File "<string>", line 7, in function_in_transform File "<string>", line 4, in function_in_root Exception: This error message should be visible to user
Visible to the user, even in for example https://editor.openeo.org/?server=https%3A%2F%2Fopeneo-dev.vito.be
Errors for executors show up towards the user as:
Can we translate this into something better, perhaps by using a spark listener?
The full stack trace is in elasticsearch, it could also be an option to filter out relevant messages from stack traces. This is full log content: