Transient Issue 2 - Reading from Natural Earth

Here is a sample log: https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/7ab7b6c1-4f90-4e2c-92f7-7b619049135a/7ab7b6c1-4f90-4e2c-92f7-7b619049135a.log

The error is:

Downloading SLCs:   0%|          | 0/4 [00:00<?, ?it/s]
Downloading SLCs:  25%|â–ˆâ–ˆâ–Œ       | 1/4 [02:06<06:20, 126.72s/it]
Downloading SLCs:  50%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆ     | 2/4 [02:35<02:18, 69.14s/it] 
Downloading SLCs: 100%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆ| 4/4 [03:19<00:00, 39.56s/it]
Downloading SLCs: 100%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆ| 4/4 [03:19<00:00, 49.87s/it]
2024-02-16 01:47:06,389 - root - WARNING - Error encountered fetching AUX_POEORB orbit file from ESA; looking for another
2024-02-16 01:51:06,575 - root - WARNING - Error encountered fetching AUX_POEORB orbit file from ASF; looking for another
2024-02-16 01:51:07,943 - root - WARNING - Error encountered fetching AUX_RESORB orbit file from ESA; looking for another
2024-02-16 01:55:08,131 - root - WARNING - Error encountered fetching AUX_RESORB orbit file from ASF; looking for another
Traceback (most recent call last):
  File "/opt/conda/envs/topsapp_env/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/conda/envs/topsapp_env/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/__main__.py", line 452, in <module>
    main()
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/__main__.py", line 448, in main
    sys.exit(process_entry_point.load()())
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/__main__.py", line 235, in gunw_slc
    loc_data = localize_data(
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/__main__.py", line 56, in localize_data
    out_orbits = download_orbits(
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/localize_orbits.py", line 36, in download_orbits
    orbit_file, _ = orbit_fetcher(scene, str(orbit_dir), esa_credentials=esa_credentials)
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/hyp3lib/get_orb.py", line 189, in downloadSentinelOrbitFile
    raise OrbitDownloadError(f'Unable to find a valid orbit file from providers: {providers}')
hyp3lib.exceptions.OrbitDownloadError: Unable to find a valid orbit file from providers: ('ESA', 'ASF')

The job spec is:

{'job_id': '7ab7b6c1-4f90-4e2c-92f7-7b619049135a',
  'job_type': 'INSAR_ISCE',
  'request_time': '2024-02-16T01:15:50+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'tibet-track136',
  'job_parameters': {'compute_solid_earth_tide': True,
   'esd_coherence_threshold': -1,
   'estimate_ionosphere_delay': True,
   'frame_id': 21164,
   'granules': ['S1A_IW_SLC__1SDV_20180527T004748_20180527T004815_022083_026319_E770',
    'S1A_IW_SLC__1SDV_20180527T004813_20180527T004839_022083_026319_8249'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20180503T004746_20180503T004813_021733_0257F4_6E5E',
    'S1A_IW_SLC__1SDV_20180503T004811_20180503T004838_021733_0257F4_FE07'],
   'weather_model': 'None'},
  'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/7ab7b6c1-4f90-4e2c-92f7-7b619049135a/7ab7b6c1-4f90-4e2c-92f7-7b619049135a.log'],
  'expiration_time': '2024-03-18T00:00:00+00:00',
  'processing_times': [8.269],
  'credit_cost': 1}

Again, this has not been an issue previously but appeared when submitting 20k jobs to Tibet.

ACCESS-Cloud-Based-InSAR / DockerizedTopsApp

Transient Issue 2 - Reading from Natural Earth #170