search

ACCESS-Cloud-Based-InSAR / DockerizedTopsApp

Apache License 2.0
21 stars 2 forks source link

[Bug] SET azimuth time interpolation - `Interpolation Time Out of Bounds` - Track 106 Exclusively #152

Open cmarshak opened 1 year ago

cmarshak commented 1 year ago

Describe the bug

This accounts for 207 errors of the 94k jobs submitted. The error log looks like this:

Traceback (most recent call last):
  File "/opt/conda/envs/topsapp_env/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/conda/envs/topsapp_env/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/__main__.py", line 365, in <module>
    main()
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/__main__.py", line 361, in main
    sys.exit(process_entry_point.load()())
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/__main__.py", line 233, in gunw_slc
    nc_path = update_gunw_with_solid_earth_tide(nc_path, "secondary", loc_data['secondary_orbits'])
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/solid_earth_tides.py", line 474, in update_gunw_with_solid_earth_tide
    solid_earth_tide_ds = compute_solid_earth_tide_from_gunw(gunw_path=gunw_path,
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/solid_earth_tides.py", line 204, in compute_solid_earth_tide_from_gunw
    tide_e, tide_n, tide_u = compute_enu_solid_earth_tide(orbit_xmls=orbit_xmls,
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/solid_earth_tides.py", line 372, in compute_enu_solid_earth_tide
    azimuth_time_arr = get_azimuth_time_array(orbit_xmls,
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/solid_earth_tides.py", line 270, in get_azimuth_time_array
    datetime_isce, rng = orb.geo2rdr([lat, lon, hgt])
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce/components/isceobj/Orbit/Orbit.py", line 1055, in geo2rdr
    raise Exception('Interpolation time out of bounds')
Exception: Interpolation time out of bounds

Here are 10 input dicts for the errors

{'job_id': '9b8cfae5-4f17-47b8-96a7-4c2ed43844cf',
  'job_type': 'INSAR_ISCE',
  'request_time': '2023-08-30T20:11:14+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'geo2rdr_2023-08-30',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 16421,
   'granules': ['S1A_IW_SLC__1SDV_20200910T225942_20200910T230009_034303_03FCC5_F00F',
    'S1A_IW_SLC__1SDV_20200910T230007_20200910T230034_034303_03FCC5_0D86'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20200817T225941_20200817T230008_033953_03F07B_3FDA',
    'S1A_IW_SLC__1SDV_20200817T230006_20200817T230033_033953_03F07B_0550'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/9b8cfae5-4f17-47b8-96a7-4c2ed43844cf/9b8cfae5-4f17-47b8-96a7-4c2ed43844cf.log'],
  'expiration_time': '2023-10-31T00:00:00+00:00',
  'processing_times': [8386.931]},
 {'job_id': 'ac35c53e-1b4e-4a1d-9560-34e8b09ec3ff',
  'job_type': 'INSAR_ISCE',
  'request_time': '2023-08-30T20:11:14+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'geo2rdr_2023-08-30',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 16421,
   'granules': ['S1A_IW_SLC__1SDV_20200829T225942_20200829T230009_034128_03F6AB_2357',
    'S1A_IW_SLC__1SDV_20200829T230007_20200829T230034_034128_03F6AB_6ECC'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20200817T225941_20200817T230008_033953_03F07B_3FDA',
    'S1A_IW_SLC__1SDV_20200817T230006_20200817T230033_033953_03F07B_0550'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/ac35c53e-1b4e-4a1d-9560-34e8b09ec3ff/ac35c53e-1b4e-4a1d-9560-34e8b09ec3ff.log'],
  'expiration_time': '2023-10-31T00:00:00+00:00',
  'processing_times': [8647.712]},
 {'job_id': '79b9485e-9d86-4a1b-a634-d00afb93a9ef',
  'job_type': 'INSAR_ISCE',
  'request_time': '2023-08-30T20:11:14+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'geo2rdr_2023-08-30',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 16421,
   'granules': ['S1A_IW_SLC__1SDV_20200910T225942_20200910T230009_034303_03FCC5_F00F',
    'S1A_IW_SLC__1SDV_20200910T230007_20200910T230034_034303_03FCC5_0D86'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20200829T225942_20200829T230009_034128_03F6AB_2357',
    'S1A_IW_SLC__1SDV_20200829T230007_20200829T230034_034128_03F6AB_6ECC'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/79b9485e-9d86-4a1b-a634-d00afb93a9ef/79b9485e-9d86-4a1b-a634-d00afb93a9ef.log'],
  'expiration_time': '2023-10-31T00:00:00+00:00',
  'processing_times': [8434.798]},
 {'job_id': '172d77d1-c524-43c9-8e73-5b53ccc0f0d9',
  'job_type': 'INSAR_ISCE',
  'request_time': '2023-08-30T20:11:14+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'geo2rdr_2023-08-30',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 16421,
   'granules': ['S1A_IW_SLC__1SDV_20201004T225943_20201004T230010_034653_04092E_B85D',
    'S1A_IW_SLC__1SDV_20201004T230008_20201004T230035_034653_04092E_3208'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20200910T225942_20200910T230009_034303_03FCC5_F00F',
    'S1A_IW_SLC__1SDV_20200910T230007_20200910T230034_034303_03FCC5_0D86'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/172d77d1-c524-43c9-8e73-5b53ccc0f0d9/172d77d1-c524-43c9-8e73-5b53ccc0f0d9.log'],
  'expiration_time': '2023-10-31T00:00:00+00:00',
  'processing_times': [8500.854]},
 {'job_id': '6602b23b-d1f1-427a-bd04-bf299ae041c9',
  'job_type': 'INSAR_ISCE',
  'request_time': '2023-08-30T20:11:14+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'geo2rdr_2023-08-30',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 16421,
   'granules': ['S1A_IW_SLC__1SDV_20200922T225943_20200922T230010_034478_04030D_928D',
    'S1A_IW_SLC__1SDV_20200922T230007_20200922T230034_034478_04030D_DD2D'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20200829T225942_20200829T230009_034128_03F6AB_2357',
    'S1A_IW_SLC__1SDV_20200829T230007_20200829T230034_034128_03F6AB_6ECC'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/6602b23b-d1f1-427a-bd04-bf299ae041c9/6602b23b-d1f1-427a-bd04-bf299ae041c9.log'],
  'expiration_time': '2023-10-31T00:00:00+00:00',
  'processing_times': [8770.476]},
 {'job_id': '33692e6b-8835-4a26-990a-0ccccec5502e',
  'job_type': 'INSAR_ISCE',
  'request_time': '2023-08-30T20:11:09+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'geo2rdr_2023-08-30',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 16422,
   'granules': ['S1A_IW_SLC__1SDV_20180124T225946_20180124T230013_020303_022ABA_6976',
    'S1A_IW_SLC__1SDV_20180124T230010_20180124T230048_020303_022ABA_CBE6'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20171219T225947_20171219T230014_019778_021A33_184F',
    'S1A_IW_SLC__1SDV_20171219T230012_20171219T230049_019778_021A33_6C12'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/33692e6b-8835-4a26-990a-0ccccec5502e/33692e6b-8835-4a26-990a-0ccccec5502e.log'],
  'expiration_time': '2023-10-31T00:00:00+00:00',
  'processing_times': [8454.583]},
 {'job_id': '2645f3f9-f796-41ec-afab-45c7cb8c2f6f',
  'job_type': 'INSAR_ISCE',
  'request_time': '2023-08-30T20:11:09+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'geo2rdr_2023-08-30',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 16422,
   'granules': ['S1A_IW_SLC__1SDV_20180406T225946_20180406T230013_021353_024C06_12D5',
    'S1A_IW_SLC__1SDV_20180406T230011_20180406T230048_021353_024C06_57E4'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20180301T225945_20180301T230012_020828_023B72_4B9D',
    'S1A_IW_SLC__1SDV_20180301T230010_20180301T230047_020828_023B72_C96D'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/2645f3f9-f796-41ec-afab-45c7cb8c2f6f/2645f3f9-f796-41ec-afab-45c7cb8c2f6f.log'],
  'expiration_time': '2023-10-31T00:00:00+00:00',
  'processing_times': [8444.457]},
 {'job_id': 'a5358300-2c10-44c2-bf8c-b5750774baa1',
  'job_type': 'INSAR_ISCE',
  'request_time': '2023-08-30T20:11:09+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'geo2rdr_2023-08-30',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 16421,
   'granules': ['S1A_IW_SLC__1SDV_20200910T225942_20200910T230009_034303_03FCC5_F00F',
    'S1A_IW_SLC__1SDV_20200910T230007_20200910T230034_034303_03FCC5_0D86'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20200314T225933_20200314T230000_031678_03A6F1_1983',
    'S1A_IW_SLC__1SDV_20200314T225958_20200314T230025_031678_03A6F1_931E'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/a5358300-2c10-44c2-bf8c-b5750774baa1/a5358300-2c10-44c2-bf8c-b5750774baa1.log'],
  'expiration_time': '2023-10-31T00:00:00+00:00',
  'processing_times': [7856.89]},
 {'job_id': 'd5cacfdd-fb3e-4da8-979b-c168d891f743',
  'job_type': 'INSAR_ISCE',
  'request_time': '2023-08-30T20:11:09+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'geo2rdr_2023-08-30',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 16422,
   'granules': ['S1A_IW_SLC__1SDV_20190224T225952_20190224T230019_026078_02E88A_1102',
    'S1A_IW_SLC__1SDV_20190224T230016_20190224T230054_026078_02E88A_A34D'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20180205T225945_20180205T230012_020478_02304E_B53C',
    'S1A_IW_SLC__1SDV_20180205T230010_20180205T230047_020478_02304E_3008'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/d5cacfdd-fb3e-4da8-979b-c168d891f743/d5cacfdd-fb3e-4da8-979b-c168d891f743.log'],
  'expiration_time': '2023-10-31T00:00:00+00:00',
  'processing_times': [8640.922]},
 {'job_id': '9b91341b-8d28-4ee2-9cb6-d24097545928',
  'job_type': 'INSAR_ISCE',
  'request_time': '2023-08-30T20:11:09+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'geo2rdr_2023-08-30',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 16422,
   'granules': ['S1A_IW_SLC__1SDV_20181027T225955_20181027T230022_024328_02A9DB_AD58',
    'S1A_IW_SLC__1SDV_20181027T230020_20181027T230057_024328_02A9DB_85D6'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20180406T225946_20180406T230013_021353_024C06_12D5',
    'S1A_IW_SLC__1SDV_20180406T230011_20180406T230048_021353_024C06_57E4'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/9b91341b-8d28-4ee2-9cb6-d24097545928/9b91341b-8d28-4ee2-9cb6-d24097545928.log'],
  'expiration_time': '2023-10-31T00:00:00+00:00',
  'processing_times': [8542.152]}

These are concentrated exclusively in 106.