Open-EO / openeo-geopyspark-driver

OpenEO driver for GeoPySpark (Geotrellis)
Apache License 2.0
25 stars 4 forks source link

Truncate long UDF errors in middle instead of end. #764

Closed EmileSonneveld closed 2 months ago

EmileSonneveld commented 2 months ago

The most interesting part of the UDF trace are at the beginning and end. This error got truncated at the end, making it useless.

OpenEO batch job failed: UDF Exception during Spark execution:   File "/opt/openeo/lib/python3.8/site-packages/openeo/udf/run_code.py", line 180, in run_udf_code
    result_cube = func(cube=data.get_datacube_list()[0], context=data.user_context)
  File "<string>", line 282, in apply_datacube
  File "<string>", line 235, in delineate
  File "tmp/venv_model/fielddelineation/utils/delineation.py", line 59, in _apply_delineation
    preds = run_prediction(
  File "tmp/venv_model/vito_lot_delineation/inference/main.py", line 33, in main
    semantic = model.forward_process(inp)
  File "tmp/venv_model/vito_lot_delineation/models/MultiHeadResUnet3D/main.py", line 99, in forward_process
    return self.model(x)
  File "tmp/venv_static/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "tmp/venv_model/vito_lot_delineation/models/MultiHeadResUnet3D/model/main.py", line 205, in forward
    memory.append(layer(memory[-1]))
  File "tmp/venv_static/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "tmp/venv_model/vito_lot_delineation/models/MultiHeadResUnet3D/model/layers.py", line 105, in forward
    x = self.down(x)
  File "tmp/venv_static/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "tmp/venv_model/vito_lot_delineation/models/MultiHeadResUnet3D/model/modules.py", line 367, in forward
    return self.f(x)
  File "tmp/venv_static/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "tmp/venv_model/vito_lot_delineation/models/MultiHeadResUnet3D/model/modules.py", line 82, in forward
    x = self.conv(x)  # Perform the convolution
  File "tmp/venv_static/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "tmp/venv_static/torch/nn/modules/conv.py", line 607, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "tmp/ve...

Trunking lines in the middle and printing ... skipped stack frames ... should be more handy

EmileSonneveld commented 1 month ago

Python snippet to show truncated error:

url = "https://openeo-staging.dataspace.copernicus.eu/"
connection = openeo.connect(url).authenticate_oidc()

spatial_extent_tap = {
    "east": 5.08,
    "north": 51.22,
    "south": 51.215,
    "west": 5.07,
}

datacube = connection.load_collection(
    "SENTINEL2_L2A",
    spatial_extent=spatial_extent_tap,
    temporal_extent=["2023-06-01", "2023-06-20"],
    bands=['B04', 'B03', 'B02'],
    properties={"eo:cloud_cover": lambda v: v <= 95},
)

UDF_code = """
import xarray
from openeo.udf import XarrayDataCube

def apply_datacube(cube: XarrayDataCube, context: dict) -> XarrayDataCube:
    def recursive_crash(counter_arg):
        def internal(counter):
            if counter <= 0:
                raise Exception("recursive_crash Exception!")
            else:
                recursive_crash(counter - 1)
        internal(counter_arg)
    recursive_crash(200)

    return XarrayDataCube(cube.get_array())
"""
datacube = datacube.apply_dimension(dimension="t", code=UDF_code, runtime="Python")
datacube.download()