Open-EO / openeo-geopyspark-driver

OpenEO driver for GeoPySpark (Geotrellis)
Apache License 2.0
26 stars 4 forks source link

Improve signal noise ratio of batch job logs #538

Open soxofaan opened 1 year ago

soxofaan commented 1 year ago

I was trying out a user snippet for https://discuss.eodc.eu/t/printing-or-logging-from-udf/421/8 to help with getting to inspect logs. It's cumbersome to get to these logs, even with filtering, because there is a lot of logging that is not very relevant for the user.

e.g. even after filtering down to INFO level I still get this (no inspect logs above the fold): image

I have to use the filter widget to find my inspect logs: image

soxofaan commented 1 year ago

FYI the job is very simple, just a load_collection and apply with UDF:

cube = connection.load_collection(
    "SENTINEL2_L2A",
    spatial_extent={"west": 5.11, "south": 51.11, "east": 5.12, "north": 51.12},
    temporal_extent=["2021-02-01", "2021-02-20"],
    bands=["B02", "B04"],
)
udf = openeo.UDF(
"""
from openeo.udf import XarrayDataCube
from openeo.udf.debug import inspect
def apply_datacube(cube: XarrayDataCube, context: dict) -> XarrayDataCube:
    inspect(message="Hello UDF logging")
    inspect(data=[1,2,3], message="Hello UDF logging with data")
    array = cube.get_array()
    array.values = 0.0001 * array.values
    return cube
"""
)
cube = cube.apply(process=udf)
job = cube.execute_batch()