search

nasa / opera-sds-pcm

Observational Products for End-Users from Remote Sensing Analysis (OPERA)
Apache License 2.0
16 stars 12 forks source link

SLC orbit file name issue #805

Closed philipjyoon closed 7 months ago

philipjyoon commented 7 months ago

Checked for duplicates

Yes - I've already checked

Describe the bug

Happened on 2.2.1. One out of 400 download jobs so far

[2024-04-16 22:42:26,205: INFO/run] argv=['/home/ops/verdi/ops/opera-pcm/data_subscriber/daac_data_subscriber.py', 'download', '-p', 'ASF', '--endpoint=OPS', '--batch-ids', 'S1A_IW_SLC__1SSV_20160104T014224_20160104T014251_009338_00D7FE_7523.zip-r10', '--start-date=2016-01-02T16:00:00Z', '--end-date=2016-01-04T08:00:00Z', '--use-temporal', '--transfer-protocol=auto']

[2024-04-16 22:42:48,715: INFO/perform_download] /data/work/jobs/2024/04/16/22/42/job-WF-slc_download-20240416T213600.501149Z/S1A_IW_SLC__1SSV_20160104T014224_20160104T014251_009338_00D7FE_7523-r10
[2024-04-16 22:42:48,715: INFO/download_orbit_file] Downloading associated orbit file
[2024-04-16 22:42:48,716: INFO/download_orbit_file] Querying for Precise Ephemeris Orbit (POEORB) file
[2024-04-16 22:42:48,717: INFO/main] Determining Orbit file for input SAFE file /data/work/jobs/2024/04/16/22/42/job-WF-slc_download-20240416T213600.501149Z/downloads/S1A_IW_SLC__1SSV_20160104T014224_20160104T014251_009338_00D7FE_7523.zip-r10/S1A_IW_SLC__1SSV_20160104T014224_20160104T014251_009338_00D7FE_7523.zip
[2024-04-16 22:42:48,717: INFO/main] Parsed time range 20160104T014224 - 20160104T014251 from SAFE filename
[2024-04-16 22:42:48,717: INFO/main] Querying for Orbit file(s) from endpoint https://catalogue.dataspace.copernicus.eu/odata/v1/Products
[2024-04-16 22:42:49,500: ERROR/_log_giveup] Giving up query_orbit_file_service(...) after 1 tries (requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://catalogue.dataspace.copernicus.eu/odata/v1/Products?%24filter=startswith%28Name%2C%27S1A%27%29+and+contains%28Name%2C%27AUX_POEORB%27%29+and+ContentDate%2FStart+lt+%272016-01-04T00%3A02%3A39.000000Z%27+and+ContentDate%2FEnd+gt+%272016-01-04T01%3A43%3A51.000000Z%27&%24orderby=ContentDate%2FStart+asc&%24top=1)
Traceback (most recent call last):
  File "/home/ops/verdi/ops/opera-pcm/data_subscriber/daac_data_subscriber.py", line 321, in <module>
    asyncio.run(run(sys.argv))
  File "/opt/conda/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/opt/conda/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/home/ops/verdi/ops/opera-pcm/data_subscriber/daac_data_subscriber.py", line 88, in run
    results["download"] = run_download(args, token, es_conn, netloc, username, password, job_id)  # return None
  File "/home/ops/verdi/ops/opera-pcm/data_subscriber/download.py", line 53, in run_download
    download.run_download(args, token, es_conn, netloc, username, password, job_id)
  File "/home/ops/verdi/ops/opera-pcm/data_subscriber/download.py", line 121, in run_download
    self.perform_download(session, es_conn, downloads, args, token, job_id)
  File "/home/ops/verdi/ops/opera-pcm/data_subscriber/asf_download.py", line 97, in perform_download
    self.download_orbit_file(new_dataset_dir, product_filepath, additional_metadata)
  File "/home/ops/verdi/ops/opera-pcm/data_subscriber/asf_download.py", line 136, in download_orbit_file
    stage_orbit_file.main(stage_orbit_file_args)
  File "/home/ops/verdi/ops/opera-pcm/tools/stage_orbit_file.py", line 626, in main
    query_results = query_orbit_file_service(args.query_endpoint, query)
  File "/home/ops/verdi/lib/python3.9/site-packages/backoff/_sync.py", line 105, in retry
    ret = target(*args, **kwargs)
  File "/home/ops/verdi/ops/opera-pcm/tools/stage_orbit_file.py", line 323, in query_orbit_file_service
    response.raise_for_status()
  File "/opt/conda/lib/python3.9/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://catalogue.dataspace.copernicus.eu/odata/v1/Products?%24filter=startswith%28Name%2C%27S1A%27%29+and+contains%28Name%2C%27AUX_POEORB%27%29+and+ContentDate%2FStart+lt+%272016-01-04T00%3A02%3A39.000000Z%27+and+ContentDate%2FEnd+gt+%272016-01-04T01%3A43%3A51.000000Z%27&%24orderby=ContentDate%2FStart+asc&%24top=1

What did you expect?

To work

Reproducible steps

1.
2.
3.
...

Environment

OPERA PCM 2.2.1
philipjyoon commented 7 months ago

Succeeded after one retry. Transient issue and not a bug.