Open-EO / openeo-geotrellis-extensions

Java/Scala extensions for Geotrellis, for use with OpenEO GeoPySpark backend.
Apache License 2.0
5 stars 4 forks source link

support less than in filter_labels along time dimension #333

Open soxofaan opened 1 month ago

soxofaan commented 1 month ago

In context of https://forum.dataspace.copernicus.eu/t/sentinel-3-lst-day-time-images/1237 I was playing with filter_labels along time dimension:

import openeo
connection = openeo.connect(url="openeo.dataspace.copernicus.eu").authenticate_oidc()

raw = connection.load_collection(
    "SENTINEL3_SLSTR_L2_LST",
    spatial_extent={"west":3, "south": 51, "east": 3.1, "north": 51.1},
    temporal_extent=["2024-09-01", "2024-09-05"],
    bands=["LST"],
)

filtered = raw.filter_labels(
    (lambda label: label < "2024-09-01T15"), 
    dimension="t",
)

(synchronous) download fails with

OpenEoApiError: [400] BadRequest: java.lang.IllegalArgumentException: Process [lt] expects a y argument. These arguments were found: xfunction tree: lt (ref: r-2410155ebb1949f1ae070a685954329e)

While the process graph actually looks fine here (has a y argument in the lt process):

{
  "process_graph": {
    "loadcollection1": {
      "process_id": "load_collection",
      "arguments": {
        "bands": ["LST"],
        "id": "SENTINEL3_SLSTR_L2_LST",
        "spatial_extent": {"west": 3, "south": 51, "east": 3.1, "north": 51.1},
        "temporal_extent": ["2024-09-01", "2024-09-05"]
      }
    },
    "filterlabels1": {
      "process_id": "filter_labels",
      "arguments": {
        "condition": {
          "process_graph": {
            "lt1": {
              "process_id": "lt",
              "arguments": {
                "x": {"from_parameter": "value"},
                "y": "2024-09-01T15"
              },
              "result": true
            }
          }
        },
        "data": {"from_node": "loadcollection1"},
        "dimension": "t"
      },
      "result": true
    }
  }
}

So this looks like a bug in the filter_labels implemenation

soxofaan commented 1 month ago

dump from relevant logs:

  File "/opt/openeo/lib/python3.8/site-packages/openeogeotrellis/layercatalog.py", line 114, in load_collection
    return self._load_collection_cached(collection_id, load_params, WhiteListEvalEnv(env, WHITELIST))
  File "/opt/openeo/lib/python3.8/site-packages/openeogeotrellis/layercatalog.py", line 260, in _load_collection_cached
    datacubeParams, single_level = datacube_parameters.create(load_params, env, jvm)
  File "/opt/openeo/lib/python3.8/site-packages/openeogeotrellis/datacube_parameters.py", line 67, in create
    labels_filter = GeoPySparkBackendImplementation.accept_process_graph(
  File "/opt/openeo/lib/python3.8/site-packages/openeogeotrellis/backend.py", line 985, in accept_process_graph
    return GeotrellisTileProcessGraphVisitor.create(default_input_parameter=default_input_parameter,default_input_datatype=default_input_datatype).accept_process_graph(process_graph)
  File "/opt/openeo/lib/python3.8/site-packages/openeo/internal/process_graph_visitor.py", line 78, in accept_process_graph
    self.accept_node(graph[top_level_node])
  File "/opt/openeo/lib/python3.8/site-packages/openeo/internal/process_graph_visitor.py", line 89, in accept_node
    self._accept_process(process_id=pid, arguments=arguments, namespace=namespace)
  File "/opt/openeo/lib/python3.8/site-packages/openeo/internal/process_graph_visitor.py", line 105, in _accept_process
    self.leaveProcess(process_id=process_id, arguments=arguments, namespace=namespace)
  File "/opt/openeo/lib/python3.8/site-packages/openeogeotrellis/processgraphvisiting.py", line 35, in leaveProcess
    self.builder.expressionEnd(process_id, arguments)
  File "/usr/local/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1322, in __call__
    return_value = get_return_value(
  File "/usr/local/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/protocol.py", line 326, in get_return_value
    raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling o14466.expressionEnd.
: java.lang.IllegalArgumentException: Process [lt] expects a y argument. These arguments were found: xfunction tree: lt
    at org.openeo.geotrellis.OpenEOProcessScriptBuilder.$anonfun$getProcessArg$1(OpenEOProcessScriptBuilder.scala:611)
    at scala.collection.MapLike.getOrElse(MapLike.scala:131)
    at scala.collection.MapLike.getOrElse$(MapLike.scala:129)
    at scala.collection.AbstractMap.getOrElse(Map.scala:65)
    at org.openeo.geotrellis.OpenEOProcessScriptBuilder.getProcessArg(OpenEOProcessScriptBuilder.scala:611)
    at org.openeo.geotrellis.OpenEOProcessScriptBuilder.xyFunction(OpenEOProcessScriptBuilder.scala:885)
    at org.openeo.geotrellis.OpenEOProcessScriptBuilder.expressionEnd(OpenEOProcessScriptBuilder.scala:1123)
    at jdk.internal.reflect.GeneratedMethodAccessor581.invoke(Unknown Source)
jdries commented 2 weeks ago

Note: tested approach for filter labels is based on 'date_between'. Other approaches have not yet been validated, and seem to give this weird error.

soxofaan commented 2 weeks ago

indeed, if something is not supported ("less than" on date labels in this example), we should have that more explicit in the error message.