Open-EO / openeo-geopyspark-driver

OpenEO driver for GeoPySpark (Geotrellis)
Apache License 2.0
25 stars 4 forks source link

Same graph works as sync but not as batch job #643

Closed clausmichele closed 4 months ago

clausmichele commented 5 months ago

Simple graph using dimension_labels works fine if run as a sync call but not as a batch job, returning the error: OpenEO batch job failed: TypeError('Object of type datetime is not JSON serializable')

{
  "process_graph": {
    "load2": {
      "process_id": "load_collection",
      "arguments": {
        "id": "SENTINEL2_L2A",
        "spatial_extent": {
          "west": 11.340596258935957,
          "east": 11.354140885309608,
          "south": 46.49069452496346,
          "north": 46.494717284338066
        },
        "temporal_extent": [
          "2022-08-01T00:00:00Z",
          "2022-08-31T00:00:00Z"
        ],
        "bands": [
          "B04"
        ]
      }
    },
    "dimension3": {
      "process_id": "dimension_labels",
      "arguments": {
        "data": {
          "from_node": "load2"
        },
        "dimension": "t"
      }
    },
    "save1": {
      "process_id": "save_result",
      "arguments": {
        "format": "JSON",
        "data": {
          "from_node": "dimension3"
        }
      },
      "result": true
    }
  },
  "parameters": []
}
soxofaan commented 5 months ago

do you have a job id or request id of that failure?

clausmichele commented 5 months ago

Here it is: j-24011939596a47a089070b753278b958 I was connected directly to the VITO back-end using the url: https://openeo.vito.be/openeo/1.2/

soxofaan commented 5 months ago

traceback from the logs:

Traceback (most recent call last):
  File "batch_job.py", line 1278, in <module>
    main(sys.argv)
  File "batch_job.py", line 1013, in main
    run_driver()
  File "batch_job.py", line 984, in run_driver
    run_job(
  File "/opt/venv/lib64/python3.8/site-packages/openeogeotrellis/utils.py", line 54, in memory_logging_wrapper
    return function(*args, **kwargs)
  File "batch_job.py", line 1121, in run_job
    the_assets_metadata = result.write_assets(str(output_file))
  File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/save_result.py", line 179, in write_assets
    json.dump(self.prepare_for_json(), f)
  File "/usr/lib64/python3.8/json/__init__.py", line 179, in dump
    for chunk in iterable:
  File "/usr/lib64/python3.8/json/encoder.py", line 429, in _iterencode
    yield from _iterencode_list(o, _current_indent_level)
  File "/usr/lib64/python3.8/json/encoder.py", line 325, in _iterencode_list
    yield from chunks
  File "/usr/lib64/python3.8/json/encoder.py", line 438, in _iterencode
    o = _default(o)
  File "/usr/lib64/python3.8/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type datetime is not JSON serializable
jdries commented 4 months ago

added simple fix + test