`ValueError: Reference and/or secondary dates were not connected in their coverage (multipolygons)`

[cmarshak]

From Processing over Track 137 in CA, there were some jobs that were cancelled before getting into ISCE2 processing. These are extraordinarily rare, but worth tracking. (Less than 20 jobs of ~10,000 submitted).

This is the Open Source version of ISCE.
Some of the workflows depend on a separate licensed package.
To obtain the licensed package, please make a request for ISCE
through the website: https://download.jpl.nasa.gov/ops/request/index.cfm.
Alternatively, if you are a member, or can become a member of WinSAR
you may be able to obtain access to a version of the licensed sofware at
https://winsar.unavco.org/software/isce
Traceback (most recent call last):
  File "/opt/conda/envs/topsapp_env/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/conda/envs/topsapp_env/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/__main__.py", line 365, in <module>
    main()
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/__main__.py", line 361, in main
    sys.exit(process_entry_point.load()())
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/__main__.py", line 166, in gunw_slc
    loc_data = localize_data(
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/__main__.py", line 46, in localize_data
    out_slc = download_slcs(
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/localize_slc.py", line 178, in download_slcs
    ifg_geo = get_interferogram_geo(reference_obs,
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/localize_slc.py", line 74, in get_interferogram_geo
    raise ValueError('Reference and/or secondary dates were not connected'
ValueError: Reference and/or secondary dates were not connected in their coverage (multipolygons)

Here are some jobs that gave such an error:

[{'job_id': '3e81367e-53e7-4603-a88e-44170027075e',
 'job_type': 'INSAR_ISCE',
 'request_time': '2023-09-06T22:30:41+00:00',
 'status_code': 'FAILED',
 'user_id': 'access_cloud_based_insar',
 'name': 'track137-n3',
 'job_parameters': {'compute_solid_earth_tide': True,
  'estimate_ionosphere_delay': True,
  'frame_id': 21252,
  'granules': ['S1A_IW_SLC__1SSV_20150531T015902_20150531T015926_006159_008023_52F4',
   'S1A_IW_SLC__1SSV_20150531T015933_20150531T020003_006159_008023_5A0F'],
  'secondary_granules': ['S1A_IW_SLC__1SDV_20150401T015853_20150401T015920_005284_006AF0_E431',
   'S1A_IW_SLC__1SDV_20150401T015918_20150401T015945_005284_006AF0_B674'],
  'weather_model': 'HRRR'},
 'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/3e81367e-53e7-4603-a88e-44170027075e/3e81367e-53e7-4603-a88e-44170027075e.log'],
 'expiration_time': '2024-03-06T00:00:00+00:00',
 'processing_times': [5.401]},
{'job_id': 'c5fc40f1-a20f-4de0-aa32-0c3cb6bec13e',
 'job_type': 'INSAR_ISCE',
 'request_time': '2023-09-06T22:33:03+00:00',
 'status_code': 'FAILED',
 'user_id': 'access_cloud_based_insar',
 'name': 'track137-n3',
 'job_parameters': {'compute_solid_earth_tide': True,
  'estimate_ionosphere_delay': True,
  'frame_id': 21252,
  'granules': ['S1A_IW_SLC__1SSV_20160805T015909_20160805T015936_012459_013772_7228',
   'S1A_IW_SLC__1SSV_20160805T015934_20160805T020001_012459_013772_BED0'],
  'secondary_granules': ['S1A_IW_SLC__1SSV_20150531T015902_20150531T015926_006159_008023_52F4',
   'S1A_IW_SLC__1SSV_20150531T015933_20150531T020003_006159_008023_5A0F'],
  'weather_model': 'HRRR'},
 'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/c5fc40f1-a20f-4de0-aa32-0c3cb6bec13e/c5fc40f1-a20f-4de0-aa32-0c3cb6bec13e.log'],
 'expiration_time': '2024-03-06T00:00:00+00:00',
 'processing_times': [4.174]},
{'job_id': '3e81367e-53e7-4603-a88e-44170027075e',
 'job_type': 'INSAR_ISCE',
 'request_time': '2023-09-06T22:30:41+00:00',
 'status_code': 'FAILED',
 'user_id': 'access_cloud_based_insar',
 'name': 'track137-n3',
 'job_parameters': {'compute_solid_earth_tide': True,
  'estimate_ionosphere_delay': True,
  'frame_id': 21252,
  'granules': ['S1A_IW_SLC__1SSV_20150531T015902_20150531T015926_006159_008023_52F4',
   'S1A_IW_SLC__1SSV_20150531T015933_20150531T020003_006159_008023_5A0F'],
  'secondary_granules': ['S1A_IW_SLC__1SDV_20150401T015853_20150401T015920_005284_006AF0_E431',
   'S1A_IW_SLC__1SDV_20150401T015918_20150401T015945_005284_006AF0_B674'],
  'weather_model': 'HRRR'},
 'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/3e81367e-53e7-4603-a88e-44170027075e/3e81367e-53e7-4603-a88e-44170027075e.log'],
 'expiration_time': '2024-03-06T00:00:00+00:00',
 'processing_times': [5.401]}]

May also be related to frame-enumeration submission.