Batch job crashes when gathering metadata from CRS that has no equivalent EPSG code

[EmileSonneveld]

Code that gives crash:

GLASS_FAPAR = connection.load_disk_collection(
    format="GTiff",
    glob_pattern="/data/users/Public/emile.sonneveld/GLASS_FAPAR_Layer/tiff_collection/*/*/*/*.hdf.tiff",
    options=dict(date_regex=r".*tiff_collection/(\d{4})/(\d{2})/(\d{2})/.*"),
).filter_temporal("2021-01-01", "2022-12-28")
# ... launch GLASS_FAPAR as batch job

Fix:

resolution = 0.00297619047619
GLASS_FAPAR = GLASS_FAPAR.resample_spatial(resolution=resolution, projection=4326)

Error:

Traceback (most recent call last):
  File "batch_job.py", line 1278, in <module>
    main(sys.argv)
  File "batch_job.py", line 1013, in main
    run_driver()
  File "batch_job.py", line 984, in run_driver
    run_job(
  File "/opt/venv/lib64/python3.8/site-packages/openeogeotrellis/utils.py", line 54, in memory_logging_wrapper
    return function(*args, **kwargs)
  File "batch_job.py", line 1193, in run_job
    result_metadata = _assemble_result_metadata(tracer=tracer, result=result, output_file=output_file,
  File "batch_job.py", line 227, in _assemble_result_metadata
    epsg = epsg_code(max_level.layer_metadata.crs)
  File "batch_job.py", line 212, in epsg_code
    return crs.get().epsgCode().getOrElse(None) if crs.isDefined() else None
  File "/opt/spark3_4_0/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1322, in __call__
    return_value = get_return_value(
  File "/opt/spark3_4_0/python/lib/py4j-0.10.9.7-src.zip/py4j/protocol.py", line 326, in get_return_value
    raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling o1695.getOrElse.
: java.lang.NullPointerException
    at scala.Option.getOrElse(Option.scala:189)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
    at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
    at java.base/java.lang.Thread.run(Thread.java:829)

job_id: j-231031a8df86464d9bdacfe6cf0e69af

[EmileSonneveld]

A better error message would already help. For example: "Only CRS that has a representing EPSG code can be used here. Consider using 'resample_spatial(...)' with a well known projection."

[EmileSonneveld]

Maybe load_stac avoids this issue? Dependant on: https://github.com/Open-EO/openeo-geopyspark-driver/issues/551

[jdries]

The error seems to happen here:

_assemble_result_metadata

so is just about assembling metadata. Perhaps we can even skip putting epsg in metadata if it can not be found. Looks like a quickfix might be possible.

---

From: Emile @.> Sent: Monday, November 13, 2023 12:13 PM To: Open-EO/openeo-geotrellis-extensions @.> Cc: Subscribed @.***> Subject: Re: [Open-EO/openeo-geotrellis-extensions] Batch job crashes when gathering metadata from CRS that has no equivalent EPSG code (Issue #224)

Maybe load_stac avoids this issue?

— Reply to this email directly, view it on GitHubhttps://github.com/Open-EO/openeo-geotrellis-extensions/issues/224#issuecomment-1807958115, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABNJPSEVBTWKTSXAQH46SMDYEH6HLAVCNFSM6AAAAAA62UOXWSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBXHE2TQMJRGU. You are receiving this because you are subscribed to this thread.Message ID: @.***>

VITO Disclaimer: http://www.vito.be/e-maildisclaimer