geo2rdr and azimuth time interpolation

[cmarshak]

Unfortunately, a bug for SET interpolation has cropped up for ~1/3 of jobs on track 106. This error did not show up previously on track 4 at all. Both had approximately 10k jobs. Here are some sample inputs and logs.

{'job_id': 'a72f7873-5011-42ae-af28-300dc44cbee5',
  'job_type': 'INSAR_ISCE_TEST',
  'request_time': '2023-08-25T03:36:55+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'track106-n3',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 16420,
   'granules': ['S1A_IW_SLC__1SDV_20230615T225929_20230615T225956_049003_05E492_C8D9',
    'S1A_IW_SLC__1SDV_20230615T225954_20230615T230021_049003_05E492_3D08'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20230522T225928_20230522T225955_048653_05DA0B_720C',
    'S1A_IW_SLC__1SDV_20230522T225953_20230522T230020_048653_05DA0B_80AB'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/a72f7873-5011-42ae-af28-300dc44cbee5/a72f7873-5011-42ae-af28-300dc44cbee5.log'],
  'expiration_time': '2023-10-25T00:00:00+00:00',
  'processing_times': [8633.635]}

and

{'job_id': '0519162c-6f33-48b4-ae64-e53b62ab7bf7',
  'job_type': 'INSAR_ISCE_TEST',
  'request_time': '2023-08-25T03:36:55+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'track106-n3',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 16420,
   'granules': ['S1A_IW_SLC__1SDV_20230603T225929_20230603T225956_048828_05DF3F_96A6',
    'S1A_IW_SLC__1SDV_20230603T225954_20230603T230020_048828_05DF3F_9974'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20230522T225928_20230522T225955_048653_05DA0B_720C',
    'S1A_IW_SLC__1SDV_20230522T225953_20230522T230020_048653_05DA0B_80AB'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/0519162c-6f33-48b4-ae64-e53b62ab7bf7/0519162c-6f33-48b4-ae64-e53b62ab7bf7.log'],
  'expiration_time': '2023-10-25T00:00:00+00:00',
  'processing_times': [8577.53]}

The relevant error shows up as

Traceback (most recent call last):
  File "/opt/conda/envs/topsapp_env/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/conda/envs/topsapp_env/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/__main__.py", line 365, in <module>
    main()
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/__main__.py", line 361, in main
    sys.exit(process_entry_point.load()())
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/__main__.py", line 232, in gunw_slc
    nc_path = update_gunw_with_solid_earth_tide(nc_path, "reference", loc_data['reference_orbits'])
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/solid_earth_tides.py", line 457, in update_gunw_with_solid_earth_tide
    solid_earth_tide_ds = compute_solid_earth_tide_from_gunw(gunw_path=gunw_path,
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/solid_earth_tides.py", line 187, in compute_solid_earth_tide_from_gunw
    tide_e, tide_n, tide_u = compute_enu_solid_earth_tide(orbit_xmls=orbit_xmls,
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/solid_earth_tides.py", line 355, in compute_enu_solid_earth_tide
    azimuth_time_arr = get_azimuth_time_array(orbit_xmls,
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/solid_earth_tides.py", line 253, in get_azimuth_time_array
    datetime_isce, rng = orb.geo2rdr([lat, lon, hgt])
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce/components/isceobj/Orbit/Orbit.py", line 1048, in geo2rdr
    tguess = tguess - datetime.timedelta(seconds = fn/fnprime)
ValueError: cannot convert float NaN to integer

This error previously came up when we had #131, in which duplicate orbit xmls was causing the job to fail. However, as I will demonstrate below, this is coming likely due to something more nuanced.

[cmarshak]

Please view the following looking at the first job. We don't need to generate the IFG to understand the error. I localize the important metadata to recreate the reported error.

https://gist.github.com/cmarshak/6c4f8acbb6466ed4ce8fe13f14de78e2

As you can see for the input granules, there are two orbit files obtained.

ref_orb_paths  = list(set(list(map(lambda g: get_orb.downloadSentinelOrbitFile(g)[0], ref_granules))))
ref_orb_paths

['S1A_OPER_AUX_POEORB_OPOD_20230705T080713_V20230614T225942_20230616T005942.EOF',
'S1A_OPER_AUX_POEORB_OPOD_20230706T080750_V20230615T225942_20230617T005942.EOF']

When I pass both orbits into geo2rdr, I get the error reported. When I pass just the first, same error. When I pass just the second, I get an error.

I do not understand the precise issue here. It would be good to understand what is happening from @dbekaert and @sssangha.

Let's resume this on Monday.