Quick Terminations for Hyp3 Plugin

Approximately 200 jobs failed instantly (out of batch of ~10k) with (indecipherable) gdal errors such as:

...
2023-09-06 22:37:31,823 - fiona.ogrext - DEBUG - Next index: 27438
2023-09-06 22:37:31,823 - fiona.ogrext - DEBUG - Next index: 27439
2023-09-06 22:37:31,823 - fiona.ogrext - DEBUG - Next index: 27440
2023-09-06 22:37:31,823 - fiona.ogrext - DEBUG - Next index: 27441
2023-09-06 22:37:31,823 - fiona.ogrext - DEBUG - Next index: 27442
2023-09-06 22:37:31,823 - fiona.ogrext - DEBUG - Next index: 27443
2023-09-06 22:37:31,824 - fiona.ogrext - DEBUG - Next index: 27444
2023-09-06 22:37:31,824 - fiona.ogrext - DEBUG - Next index: 27445
2023-09-06 22:37:31,824 - fiona.ogrext - DEBUG - Next index: 27446
2023-09-06 22:37:31,824 - fiona.ogrext - DEBUG - Next index: 27447
2023-09-06 22:37:31,824 - fiona.ogrext - DEBUG - Next index: 27448
2023-09-06 22:37:31,824 - fiona.ogrext - DEBUG - Next index: 27449
2023-09-06 22:37:31,824 - fiona.ogrext - DEBUG - Next index: 27450
2023-09-06 22:37:31,825 - fiona.ogrext - DEBUG - Next index: 27451
2023-09-06 22:37:31,825 - fiona.ogrext - DEBUG - Next index: 27452
2023-09-06 22:37:31,825 - fiona.ogrext - DEBUG - Next index: 27453
2023-09-06 22:37:31,825 - fiona.ogrext - DEBUG - Next index: 27454
2023-09-06 22:37:31,825 - fiona.ogrext - DEBUG - Next index: 27455
2023-09-06 22:37:31,825 - fiona.ogrext - DEBUG - Next index: 27456
2023-09-06 22:37:31,825 - fiona.ogrext - DEBUG - Next index: 27457
2023-09-06 22:37:31,825 - fiona.ogrext - DEBUG - Next index: 27458
2023-09-06 22:37:31,826 - fiona.ogrext - DEBUG - Next index: 27459
2023-09-06 22:37:31,826 - fiona.ogrext - DEBUG - Next index: 27460
2023-09-06 22:37:31,826 - fiona.ogrext - DEBUG - Next index: 27461
2023-09-06 22:37:31,826 - fiona.ogrext - DEBUG - Next index: 27462
2023-09-06 22:37:31,827 - fiona.ogrext - DEBUG - Next index: 27463
2023-09-06 22:37:31,828 - fiona.ogrext - DEBUG - Next index: 27464
2023-09-06 22:37:31,828 - fiona.ogrext - DEBUG - Next index: 27465
2023-09-06 22:37:31,828 - fiona.ogrext - DEBUG - Next index: 27466
2023-09-06 22:37:31,828 - fiona.ogrext - DEBUG - Next index: 27467
2023-09-06 22:37:31,828 - fiona.ogrext - DEBUG - Next index: 27468
2023-09-06 22:37:31,828 - fiona.ogrext - DEBUG - Next index: 27469
2023-09-06 22:37:31,828 - fiona.ogrext - DEBUG - Next index: 27470
2023-09-06 22:37:31,828 - fiona.ogrext - DEBUG - Next index: 27471
2023-09-06 22:37:31,829 - fiona.ogrext - DEBUG - Next index: 27472
2023-09-06 22:37:31,829 - fiona.ogrext - DEBUG - Next index: 27473
2023-09-06 22:37:31,829 - fiona.ogrext - DEBUG - Next index: 27474
2023-09-06 22:37:31,829 - fiona.ogrext - DEBUG - Next index: 27475
2023-09-06 22:37:31,829 - fiona.ogrext - DEBUG - Next index: 27476
2023-09-06 22:37:31,829 - fiona.ogrext - DEBUG - Next index: 27477
2023-09-06 22:37:31,829 - fiona.ogrext - DEBUG - Next index: 27478
2023-09-06 22:37:31,830 - fiona.ogrext - DEBUG - Next index: 27479
2023-09-06 22:37:31,830 - fiona.ogrext - DEBUG - Next index: 27480
2023-09-06 22:37:31,830 - fiona.ogrext - DEBUG - Next index: 27481
2023-09-06 22:37:31,830 - fiona.ogrext - DEBUG - Next index: 27482
2023-09-06 22:37:31,830 - fiona.ogrext - DEBUG - Next index: 27483
2023-09-06 22:37:31,830 - fiona.ogrext - DEBUG - Next index: 27484
2023-09-06 22:37:31,830 - fiona.ogrext - DEBUG - Next index: 27485
2023-09-06 22:37:31,830 - fiona.ogrext - DEBUG - Next index: 27486
2023-09-06 22:37:31,831 - fiona.ogrext - DEBUG - Next index: 27487
2023-09-06 22:37:31,831 - fiona.ogrext - DEBUG - Next index: 27488
2023-09-06 22:37:31,831 - fiona.ogrext - DEBUG - Next index: 27489
2023-09-06 22:37:31,831 - fiona.ogrext - DEBUG - Next index: 27490
2023-09-06 22:37:31,831 - fiona.ogrext - DEBUG - Next index: 27491
2023-09-06 22:37:31,831 - fiona.ogrext - DEBUG - Next index: 27492
2023-09-06 22:37:31,831 - fiona.ogrext - DEBUG - Next index: 27493
2023-09-06 22:37:31,831 - fiona.ogrext - DEBUG - Next index: 27494
2023-09-06 22:37:31,831 - fiona.ogrext - DEBUG - Next index: 27495
2023-09-06 22:37:31,832 - fiona.ogrext - DEBUG - Next index: 27496
2023-09-06 22:37:31,832 - fiona.ogrext - DEBUG - Next index: 27497
2023-09-06 22:37:31,832 - fiona.ogrext - DEBUG - Next index: 27498
2023-09-06 22:37:31,832 - fiona.ogrext - DEBUG - Next index: 27499
2023-09-06 22:37:31,832 - fiona.ogrext - DEBUG - Next index: 27500
2023-09-06 22:37:31,832 - fiona.ogrext - DEBUG - Next index: 27501
2023-09-06 22:37:31,833 - fiona.ogrext - DEBUG - Next index: 27502
2023-09-06 22:37:31,834 - fiona.ogrext - DEBUG - Next index: 27503
2023-09-06 22:37:31,834 - fiona.ogrext - DEBUG - Next index: 27504
2023-09-06 22:37:31,834 - fiona.ogrext - DEBUG - Next index: 27505
2023-09-06 22:37:31,834 - fiona.ogrext - DEBUG - Next index: 27506
2023-09-06 22:37:31,834 - fiona.ogrext - DEBUG - Next index: 27507
2023-09-06 22:37:31,834 - fiona.ogrext - DEBUG - Next index: 27508
2023-09-06 22:37:31,834 - fiona.ogrext - DEBUG - Next index: 27509
2023-09-06 22:37:31,834 - fiona.ogrext - DEBUG - Next index: 27510
2023-09-06 22:37:31,835 - fiona.ogrext - DEBUG - Next index: 27511
2023-09-06 22:37:31,835 - fiona.ogrext - DEBUG - Next index: 27512
2023-09-06 22:37:31,835 - fiona.ogrext - DEBUG - Next index: 27513
2023-09-06 22:37:31,835 - fiona.ogrext - DEBUG - Next index: 27514
2023-09-06 22:37:31,835 - fiona.ogrext - DEBUG - Next index: 27515
2023-09-06 22:37:31,835 - fiona.ogrext - DEBUG - Next index: 27516
2023-09-06 22:37:31,835 - fiona.ogrext - DEBUG - Next index: 27517
2023-09-06 22:37:31,835 - fiona.ogrext - DEBUG - Next index: 27518
2023-09-06 22:37:31,836 - fiona.ogrext - DEBUG - Next index: 27519
2023-09-06 22:37:31,836 - fiona.ogrext - DEBUG - Next index: 27520
2023-09-06 22:37:31,836 - fiona.ogrext - DEBUG - Next index: 27521
2023-09-06 22:37:31,836 - fiona.ogrext - DEBUG - Next index: 27522
2023-09-06 22:37:31,836 - fiona.ogrext - DEBUG - Next index: 27523
2023-09-06 22:37:31,836 - fiona.ogrext - DEBUG - Next index: 27524
2023-09-06 22:37:31,836 - fiona.ogrext - DEBUG - Next index: 27525
2023-09-06 22:37:31,837 - fiona.ogrext - DEBUG - Next index: 27526
2023-09-06 22:37:31,837 - fiona.ogrext - DEBUG - Next index: 27527
2023-09-06 22:37:31,837 - fiona.ogrext - DEBUG - Next index: 27528
2023-09-06 22:37:31,837 - fiona.ogrext - DEBUG - Next index: 27529
2023-09-06 22:37:31,837 - fiona.ogrext - DEBUG - Next index: 27530
2023-09-06 22:37:31,837 - fiona.ogrext - DEBUG - Next index: 27531
2023-09-06 22:37:31,837 - fiona.ogrext - DEBUG - Next index: 27532
2023-09-06 22:37:31,838 - fiona.ogrext - DEBUG - Next index: 27533
2023-09-06 22:37:31,839 - fiona.ogrext - DEBUG - Next index: 27534
2023-09-06 22:37:31,839 - fiona.ogrext - DEBUG - Next index: 27535
2023-09-06 22:37:31,839 - fiona.ogrext - DEBUG - Next index: 27536
2023-09-06 22:37:31,839 - fiona.ogrext - DEBUG - Next index: 27537
2023-09-06 22:37:31,839 - fiona.ogrext - DEBUG - Next index: 27538
2023-09-06 22:37:31,840 - fiona.ogrext - DEBUG - Next index: 27539
2023-09-06 22:37:31,840 - fiona.ogrext - DEBUG - Next index: 27540
2023-09-06 22:37:31,840 - fiona.ogrext - DEBUG - Next index: 27541
2023-09-06 22:37:31,840 - fiona.ogrext - DEBUG - Next index: 27542
2023-09-06 22:37:31,840 - fiona.ogrext - DEBUG - Next index: 27543
2023-09-06 22:37:31,840 - fiona.ogrext - DEBUG - Next index: 27544
2023-09-06 22:37:31,840 - fiona.ogrext - DEBUG - Next index: 27545
2023-09-06 22:37:31,841 - fiona.ogrext - DEBUG - Next index: 27546
2023-09-06 22:37:31,841 - fiona.ogrext - DEBUG - Next index: 27547
2023-09-06 22:37:31,841 - fiona.ogrext - DEBUG - Next index: 27548
2023-09-06 22:37:31,841 - fiona.ogrext - DEBUG - Next index: 27549
2023-09-06 22:37:31,841 - fiona.ogrext - DEBUG - Next index: 27550
2023-09-06 22:37:31,841 - fiona.ogrext - DEBUG - Next index: 27551
2023-09-06 22:37:31,842 - fiona.ogrext - DEBUG - Next index: 27552
2023-09-06 22:37:31,842 - fiona.ogrext - DEBUG - Next index: 27553
2023-09-06 22:37:31,843 - fiona.ogrext - DEBUG - Next index: 27554
2023-09-06 22:37:31,843 - fiona.ogrext - DEBUG - Next index: 27555
2023-09-06 22:37:31,843 - fiona.ogrext - DEBUG - Next index: 27556
2023-09-06 22:37:31,843 - fiona.ogrext - DEBUG - Next index: 27557
2023-09-06 22:37:31,843 - fiona.ogrext - DEBUG - Next index: 27558
2023-09-06 22:37:31,843 - fiona.ogrext - DEBUG - Next index: 27559
2023-09-06 22:37:31,843 - fiona.ogrext - DEBUG - Next index: 27560
2023-09-06 22:37:31,844 - fiona.ogrext - DEBUG - Next index: 27561
2023-09-06 22:37:31,844 - fiona.ogrext - DEBUG - Next index: 27562
2023-09-06 22:37:31,844 - fiona.ogrext - DEBUG - Next index: 27563
2023-09-06 22:37:31,880 - fiona.collection - DEBUG - Flushed buffer
2023-09-06 22:37:31,880 - fiona.collection - DEBUG - Stopped session
2023-09-06 22:37:31,880 - fiona.env - DEBUG - Exiting env context: <fiona.env.Env object at 0x7f233deabb50>
2023-09-06 22:37:31,881 - fiona.env - DEBUG - Cleared existing <fiona._env.GDALEnv object at 0x7f233deaf280> options
2023-09-06 22:37:31,881 - fiona._env - DEBUG - Stopping GDALEnv <fiona._env.GDALEnv object at 0x7f233deaf280>.
2023-09-06 22:37:31,881 - fiona._env - DEBUG - Error handler popped.
2023-09-06 22:37:31,881 - fiona._env - DEBUG - Stopped GDALEnv <fiona._env.GDALEnv object at 0x7f233deaf280>.
2023-09-06 22:37:31,881 - fiona.env - DEBUG - Exiting outermost env
2023-09-06 22:37:31,881 - fiona.env - DEBUG - Exited env context: <fiona.env.Env object at 0x7f233deabb50>
2023-09-06 22:37:32,053 - fiona.env - DEBUG - Entering env context: <fiona.env.Env object at 0x7f234f1e2e80>
2023-09-06 22:37:32,053 - fiona.env - DEBUG - Starting outermost env
2023-09-06 22:37:32,054 - fiona.env - DEBUG - No GDAL environment exists
2023-09-06 22:37:32,054 - fiona.env - DEBUG - New GDAL environment <fiona._env.GDALEnv object at 0x7f233ddddeb0> created
2023-09-06 22:37:32,054 - fiona._env - DEBUG - GDAL_DATA found in environment.
2023-09-06 22:37:32,054 - fiona._env - DEBUG - PROJ data files are available at built-in paths.
2023-09-06 22:37:32,054 - fiona._env - DEBUG - Started GDALEnv: self=<fiona._env.GDALEnv object at 0x7f233ddddeb0>.
2023-09-06 22:37:32,055 - fiona.env - DEBUG - Updated existing <fiona._env.GDALEnv object at 0x7f233ddddeb0> with options {}
2023-09-06 22:37:32,055 - fiona.env - DEBUG - Entered env context: <fiona.env.Env object at 0x7f234f1e2e80>
2023-09-06 22:37:32,055 - fiona._env - ERROR - `/vsimem/d28b756358994cdfb4be51956ad71c98' not recognized as a supported file format.
2023-09-06 22:37:32,055 - fiona.env - DEBUG - Exiting env context: <fiona.env.Env object at 0x7f234f1e2e80>
2023-09-06 22:37:32,056 - fiona.env - DEBUG - Cleared existing <fiona._env.GDALEnv object at 0x7f233ddddeb0> options
2023-09-06 22:37:32,056 - fiona._env - DEBUG - Stopping GDALEnv <fiona._env.GDALEnv object at 0x7f233ddddeb0>.
2023-09-06 22:37:32,056 - fiona._env - DEBUG - Error handler popped.
2023-09-06 22:37:32,056 - fiona._env - DEBUG - Stopped GDALEnv <fiona._env.GDALEnv object at 0x7f233ddddeb0>.
2023-09-06 22:37:32,056 - fiona.env - DEBUG - Exiting outermost env
2023-09-06 22:37:32,056 - fiona.env - DEBUG - Exited env context: <fiona.env.Env object at 0x7f234f1e2e80>
Traceback (most recent call last):
  File "fiona/_shim.pyx", line 83, in fiona._shim.gdal_open_vector
  File "fiona/_err.pyx", line 291, in fiona._err.exc_wrap_pointer
fiona._err.CPLE_OpenFailedError: '/vsimem/d28b756358994cdfb4be51956ad71c98' not recognized as a supported file format.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/conda/envs/topsapp_env/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/conda/envs/topsapp_env/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/__main__.py", line 365, in <module>
    main()
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/__main__.py", line 361, in main
    sys.exit(process_entry_point.load()())
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/__main__.py", line 166, in gunw_slc
    loc_data = localize_data(
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/__main__.py", line 46, in localize_data
    out_slc = download_slcs(
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/localize_slc.py", line 182, in download_slcs
    percent_water_low_res = get_percent_water_from_ne_land(ifg_geo)
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/localize_slc.py", line 137, in get_percent_water_from_ne_land
    df_world = get_world_df()
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/localize_slc.py", line 131, in get_world_df
    df_world = gpd.read_file(io.BytesIO(resp.content))
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/geopandas/io/file.py", line 281, in _read_file
    return _read_file_fiona(
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/geopandas/io/file.py", line 322, in _read_file_fiona
    with reader(path_or_bytes, **kwargs) as features:
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/fiona/collection.py", line 555, in __init__
    super(BytesCollection, self).__init__(self.virtual_file, vsi=filetype, **kwds)
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/fiona/collection.py", line 162, in __init__
    self.session.start(self, **kwargs)
  File "fiona/ogrext.pyx", line 540, in fiona.ogrext.Session.start
  File "fiona/_shim.pyx", line 90, in fiona._shim.gdal_open_vector
fiona.errors.DriverError: '/vsimem/d28b756358994cdfb4be51956ad71c98' not recognized as a supported file format.

Here are some sample input parameters

[{'expiration_time': '2024-03-05T00:00:00+00:00',
 'job_id': 'cf25cfe9-7058-4b82-8d3b-cb341a1275da',
 'job_parameters': {'compute_solid_earth_tide': True,
                    'estimate_ionosphere_delay': True,
                    'frame_id': 21249,
                    'granules': ['S1B_IW_SLC__1SDV_20190703T015756_20190703T015824_016963_01FEAF_7691',
                                 'S1B_IW_SLC__1SDV_20190703T015821_20190703T015848_016963_01FEAF_0A5E'],
                    'secondary_granules': ['S1B_IW_SLC__1SDV_20190528T015754_20190528T015821_016438_01EF17_2F06',
                                           'S1B_IW_SLC__1SDV_20190528T015819_20190528T015846_016438_01EF17_38B0'],
                    'weather_model': 'HRRR'},
 'job_type': 'INSAR_ISCE',
 'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/cf25cfe9-7058-4b82-8d3b-cb341a1275da/cf25cfe9-7058-4b82-8d3b-cb341a1275da.log'],
 'name': 'track137-n3',
 'processing_times': [20.156],
 'request_time': '2023-09-06T22:31:58+00:00',
 'status_code': 'FAILED',
 'user_id': 'access_cloud_based_insar'},
{'expiration_time': '2024-03-05T00:00:00+00:00',
 'job_id': '62c0814b-ad15-4aae-8108-de3f692aae02',
 'job_parameters': {'compute_solid_earth_tide': True,
                    'estimate_ionosphere_delay': True,
                    'frame_id': 21255,
                    'granules': ['S1A_IW_SLC__1SDV_20190522T020034_20190522T020102_027334_031536_DDC4',
                                 'S1A_IW_SLC__1SDV_20190522T020100_20190522T020117_027334_031536_A526'],
                    'secondary_granules': ['S1A_IW_SLC__1SDV_20190404T020032_20190404T020100_026634_02FCEC_248C',
                                           'S1A_IW_SLC__1SDV_20190404T020058_20190404T020115_026634_02FCEC_576A'],
                    'weather_model': 'HRRR'},
 'job_type': 'INSAR_ISCE',
 'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/62c0814b-ad15-4aae-8108-de3f692aae02/62c0814b-ad15-4aae-8108-de3f692aae02.log'],
 'name': 'track137-n3',
 'processing_times': [20.674],
 'request_time': '2023-09-06T22:31:58+00:00',
 'status_code': 'FAILED',
 'user_id': 'access_cloud_based_insar'},
{'expiration_time': '2024-03-05T00:00:00+00:00',
 'job_id': '88dc9441-e17f-4c90-9774-4370dde8b339',
 'job_parameters': {'compute_solid_earth_tide': True,
                    'estimate_ionosphere_delay': True,
                    'frame_id': 21251,
                    'granules': ['S1B_IW_SLC__1SDV_20190925T015826_20190925T015853_018188_0223F8_D54C',
                                 'S1B_IW_SLC__1SDV_20190925T015851_20190925T015918_018188_0223F8_B611'],
                    'secondary_granules': ['S1B_IW_SLC__1SDV_20190820T015824_20190820T015851_017663_0213AD_8E10',
                                           'S1B_IW_SLC__1SDV_20190820T015849_20190820T015916_017663_0213AD_49C0'],
                    'weather_model': 'HRRR'},
 'job_type': 'INSAR_ISCE',
 'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/88dc9441-e17f-4c90-9774-4370dde8b339/88dc9441-e17f-4c90-9774-4370dde8b339.log'],
 'name': 'track137-n3',
 'processing_times': [20.131],
 'request_time': '2023-09-06T22:31:51+00:00',
 'status_code': 'FAILED',
 'user_id': 'access_cloud_based_insar'}]

Maybe related to #154 and unusual geometries being passed via the geometric metadata retreived from ASF? Bad frame enumeration? Will investigate later. Could also be transient.

ACCESS-Cloud-Based-InSAR / DockerizedTopsApp

Quick Terminations for Hyp3 Plugin #155