search

ACCESS-Cloud-Based-InSAR / DockerizedTopsApp

Apache License 2.0
21 stars 2 forks source link

Transient Issues 1 - Orbit Downloading Issues #169

Closed cmarshak closed 7 months ago

cmarshak commented 7 months ago

These issues appeared after submitting about 20k jobs in the Tibet account.

{'job_id': '6698b2f0-e367-4c84-8d69-72b788eb2029',
  'job_type': 'INSAR_ISCE',
  'request_time': '2024-02-16T01:15:36+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'tibet-track136',
  'job_parameters': {'compute_solid_earth_tide': True,
   'esd_coherence_threshold': -1,
   'estimate_ionosphere_delay': True,
   'frame_id': 21174,
   'granules': ['S1A_IW_SLC__1SDV_20200317T005047_20200317T005114_031708_03A802_1158',
    'S1A_IW_SLC__1SDV_20200317T005112_20200317T005139_031708_03A802_B795'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20200305T005052_20200305T005119_031533_03A1F5_4AB2',
    'S1A_IW_SLC__1SDV_20200305T005117_20200305T005144_031533_03A1F5_9CDB'],
   'weather_model': 'None'},
  'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/6698b2f0-e367-4c84-8d69-72b788eb2029/6698b2f0-e367-4c84-8d69-72b788eb2029.log'],
  'expiration_time': '2024-03-18T00:00:00+00:00',
  'processing_times': [590.843],
  'credit_cost': 1},
 {'job_id': '8e3dc277-d7eb-40ac-bd8a-883f4331624c',
  'job_type': 'INSAR_ISCE',
  'request_time': '2024-02-16T01:15:36+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'tibet-track136',
  'job_parameters': {'compute_solid_earth_tide': True,
   'esd_coherence_threshold': -1,
   'estimate_ionosphere_delay': True,
   'frame_id': 21169,
   'granules': ['S1A_IW_SLC__1SDV_20200317T004931_20200317T004958_031708_03A802_4C57',
    'S1A_IW_SLC__1SDV_20200317T004956_20200317T005024_031708_03A802_0442'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20200222T004912_20200222T004939_031358_039BE4_1A47',
    'S1A_IW_SLC__1SDV_20200222T004937_20200222T005005_031358_039BE4_5E8B'],
   'weather_model': 'None'},
  'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/8e3dc277-d7eb-40ac-bd8a-883f4331624c/8e3dc277-d7eb-40ac-bd8a-883f4331624c.log'],
  'expiration_time': '2024-03-18T00:00:00+00:00',
  'processing_times': [660.159],
  'credit_cost': 1},
 {'job_id': '071decc2-d96c-4854-9109-a5fff179e1ab',
  'job_type': 'INSAR_ISCE',
  'request_time': '2024-02-16T01:15:36+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'tibet-track136',
  'job_parameters': {'compute_solid_earth_tide': True,
   'esd_coherence_threshold': -1,
   'estimate_ionosphere_delay': True,
   'frame_id': 21167,
   'granules': ['S1A_IW_SLC__1SDV_20200516T004838_20200516T004905_032583_03C61F_D08D',
    'S1A_IW_SLC__1SDV_20200516T004903_20200516T004930_032583_03C61F_2068'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20200305T004847_20200305T004914_031533_03A1F5_52CD',
    'S1A_IW_SLC__1SDV_20200305T004912_20200305T004939_031533_03A1F5_084C'],
   'weather_model': 'None'},
  'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/071decc2-d96c-4854-9109-a5fff179e1ab/071decc2-d96c-4854-9109-a5fff179e1ab.log'],
  'expiration_time': '2024-03-18T00:00:00+00:00',
  'processing_times': [698.788],
  'credit_cost': 1},
 {'job_id': 'fa9b4d15-0ed7-4e64-8381-35f2d30bb418',
  'job_type': 'INSAR_ISCE',
  'request_time': '2024-02-16T01:15:36+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'tibet-track136',
  'job_parameters': {'compute_solid_earth_tide': True,
   'esd_coherence_threshold': -1,
   'estimate_ionosphere_delay': True,
   'frame_id': 21175,
   'granules': ['S1A_IW_SLC__1SDV_20200317T005112_20200317T005139_031708_03A802_B795',
    'S1A_IW_SLC__1SDV_20200317T005136_20200317T005203_031708_03A802_8759'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20200222T005117_20200222T005144_031358_039BE4_951D',
    'S1A_IW_SLC__1SDV_20200222T005142_20200222T005209_031358_039BE4_0936'],
   'weather_model': 'None'},
  'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/fa9b4d15-0ed7-4e64-8381-35f2d30bb418/fa9b4d15-0ed7-4e64-8381-35f2d30bb418.log'],
  'expiration_time': '2024-03-18T00:00:00+00:00',
  'processing_times': [692.04],
  'credit_cost': 1},
 {'job_id': '36a4e133-ce79-4cb1-b5ca-8a4a10cc67e8',
  'job_type': 'INSAR_ISCE',
  'request_time': '2024-02-16T01:15:36+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'tibet-track136',
  'job_parameters': {'compute_solid_earth_tide': True,
   'esd_coherence_threshold': -1,
   'estimate_ionosphere_delay': True,
   'frame_id': 21169,
   'granules': ['S1A_IW_SLC__1SDV_20200305T004912_20200305T004939_031533_03A1F5_084C',
    'S1A_IW_SLC__1SDV_20200305T004937_20200305T005005_031533_03A1F5_491C'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20200210T004912_20200210T004939_031183_0395E0_4254',
    'S1A_IW_SLC__1SDV_20200210T004937_20200210T005005_031183_0395E0_FCDC'],
   'weather_model': 'None'},
  'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/36a4e133-ce79-4cb1-b5ca-8a4a10cc67e8/36a4e133-ce79-4cb1-b5ca-8a4a10cc67e8.log'],
  'expiration_time': '2024-03-18T00:00:00+00:00',
  'processing_times': [622.821],
  'credit_cost': 1}

Here are some sample jobs.

Here are some sample errors from above:

Downloading SLCs:   0%|          | 0/4 [00:00<?, ?it/s]
Downloading SLCs:  25%|██▌       | 1/4 [01:55<05:45, 115.11s/it]
Downloading SLCs:  75%|███████▌  | 3/4 [02:46<00:48, 48.76s/it] 
Downloading SLCs: 100%|██████████| 4/4 [02:46<00:00, 41.55s/it]
2024-02-16 01:47:37,476 - root - WARNING - Error encountered fetching AUX_POEORB orbit file from ESA; looking for another
2024-02-16 01:51:37,661 - root - WARNING - Error encountered fetching AUX_POEORB orbit file from ASF; looking for another
2024-02-16 01:51:39,070 - root - INFO - Downloading None
2024-02-16 01:51:39,686 - root - WARNING - Error encountered fetching AUX_RESORB orbit file from ESA; looking for another

or

Downloading SLCs:   0%|          | 0/4 [00:00<?, ?it/s]
Downloading SLCs:  25%|██▌       | 1/4 [02:06<06:20, 126.72s/it]
Downloading SLCs:  50%|█████     | 2/4 [02:35<02:18, 69.14s/it] 
Downloading SLCs: 100%|██████████| 4/4 [03:19<00:00, 39.56s/it]
Downloading SLCs: 100%|██████████| 4/4 [03:19<00:00, 49.87s/it]
2024-02-16 01:47:06,389 - root - WARNING - Error encountered fetching AUX_POEORB orbit file from ESA; looking for another
2024-02-16 01:51:06,575 - root - WARNING - Error encountered fetching AUX_POEORB orbit file from ASF; looking for another
2024-02-16 01:51:07,943 - root - WARNING - Error encountered fetching AUX_RESORB orbit file from ESA; looking for another
2024-02-16 01:55:08,131 - root - WARNING - Error encountered fetching AUX_RESORB orbit file from ASF; looking for another
Traceback (most recent call last):
  File "/opt/conda/envs/topsapp_env/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/conda/envs/topsapp_env/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/__main__.py", line 452, in <module>
    main()
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/__main__.py", line 448, in main
    sys.exit(process_entry_point.load()())
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/__main__.py", line 235, in gunw_slc
    loc_data = localize_data(
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/__main__.py", line 56, in localize_data
    out_orbits = download_orbits(
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/localize_orbits.py", line 36, in download_orbits
    orbit_file, _ = orbit_fetcher(scene, str(orbit_dir), esa_credentials=esa_credentials)
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/hyp3lib/get_orb.py", line 189, in downloadSentinelOrbitFile
    raise OrbitDownloadError(f'Unable to find a valid orbit file from providers: {providers}')
hyp3lib.exceptions.OrbitDownloadError: Unable to find a valid orbit file from providers: ('ESA', 'ASF')

Having a hard time understanding why this is failing... orbit downloading has been a stalwart of this plugin.