nasa / opera-sds-pcm

Observational Products for End-Users from Remote Sensing Analysis (OPERA)
Apache License 2.0
16 stars 12 forks source link

[Bug]: Error "Object of type datetime is not JSON serializable" when downloading SLC data #868

Closed philipjyoon closed 3 months ago

philipjyoon commented 3 months ago

Checked for duplicates

Yes - I've already checked

Describe the bug

Run the query command below and the download job errors with the following trace stack. I've retried and get the same error. This data used to work... at least a month ago.

[2024-06-06 14:13:36,579: INFO/main] Determining Ionosphere file for input file /data/work/jobs/2024/06/06/14/10/job-WF-slc_download-20240606T125310.268370Z/downloads/S1A_IW_SLC__1SDV_20230724T191223_20230724T191253_049569_05F5E3_13DB.zip-r3/S1A_IW_SLC__1SDV_20230724T191223_20230724T191253_049569_05F5E3_13DB.zip
[2024-06-06 14:13:36,579: INFO/main] Parsed start date 20230724 from filename
[2024-06-06 14:13:37,395: INFO/main] URL-only requested
[2024-06-06 14:13:37,395: INFO/main] https://cddis.nasa.gov/archive/gnss/products/ionex/2023/205/jplg2050.23i.Z
[2024-06-06 14:13:37,397: INFO/get_ionosphere_correction_file_url] Added JPLG Ionosphere correction file to dataset
[2024-06-06 14:13:37,398: INFO/supply_job_id] job_id='5228ed85-6f09-496f-a106-300ff9af652a'
[2024-06-06 14:13:37,398: INFO/update_pending_dataset_metadata_with_ionosphere_metadata] Updating dataset's met.json with ionosphere metadata
https://cddis.nasa.gov/archive/gnss/products/ionex/2023/205/jplg2050.23i.Z
Traceback (most recent call last):
  File "/home/ops/verdi/ops/opera-pcm/data_subscriber/daac_data_subscriber.py", line 304, in <module>
    main()
  File "/home/ops/verdi/ops/opera-pcm/util/exec_util.py", line 35, in wrapper
    status = func(*args, **kwargs)
  File "/home/ops/verdi/ops/opera-pcm/data_subscriber/daac_data_subscriber.py", line 57, in main
    asyncio.run(run(sys.argv))
  File "/opt/conda/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/opt/conda/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/home/ops/verdi/ops/opera-pcm/data_subscriber/daac_data_subscriber.py", line 112, in run
    results["download"] = await run_download(args, token, es_conn, netloc, username, password, job_id)
  File "/home/ops/verdi/ops/opera-pcm/data_subscriber/daac_data_subscriber.py", line 159, in run_download
    await downloader.run_download(args, token, es_conn, netloc, username, password, job_id)
  File "/home/ops/verdi/ops/opera-pcm/data_subscriber/download.py", line 79, in run_download
    product_to_product_filepaths_map = self.perform_download(
  File "/home/ops/verdi/ops/opera-pcm/data_subscriber/asf_download.py", line 99, in perform_download
    self.download_ionosphere_file(new_dataset_dir, product_filepath)
  File "/home/ops/verdi/ops/opera-pcm/data_subscriber/asf_slc_download.py", line 122, in download_ionosphere_file
    self.update_pending_dataset_metadata_with_ionosphere_metadata(dataset_dir, ionosphere_metadata)
  File "/home/ops/verdi/ops/opera-pcm/data_subscriber/asf_slc_download.py", line 136, in update_pending_dataset_metadata_with_ionosphere_metadata
    json.dump(met_json, fp)
  File "/opt/conda/lib/python3.9/json/__init__.py", line 179, in dump
    for chunk in iterable:
  File "/opt/conda/lib/python3.9/json/encoder.py", line 431, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "/opt/conda/lib/python3.9/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/opt/conda/lib/python3.9/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/opt/conda/lib/python3.9/json/encoder.py", line 438, in _iterencode
    o = _default(o)
  File "/opt/conda/lib/python3.9/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type datetime is not JSON serializable

What did you expect?

n/t

Reproducible steps

python daac_data_subscriber.py  query -c SENTINEL-1A_SLC --processing-mode=reprocessing --job-queue=opera-job_worker-slc_data_download --chunk-size=1 --native-id=S1A_IW_SLC__1SDV_20230724T191223_20230724T191253_049569_05F5E3_13DB-SLC

Environment

- Version of this software [e.g. vX.Y.Z]
- Operating System: [e.g. MacOSX with Docker Desktop vX.Y]
...