Issue with Water Mask Download

[cmarshak]

Describe the bug

Marin, can you please check what is happening with these jobs with respect to the water mask as in maybe there is some kind of server issue or other issue that is being obfuscated by truncated logs? Maybe the new control flow? (see below - it's a timeout, but want to understand why I am seeing these issues in 2/15 job submissions):

{'job_id': '2def9d71-30a3-4dba-9f28-ae8eb78797b8',
  'job_type': 'INSAR_ISCE_TEST',
  'request_time': '2023-06-21T23:49:11+00:00',
  'status_code': 'FAILED',
  'user_id': 'cmarshak',
  'name': 'Maine_164_HRES_0621',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 25501,
   'granules': ['S1A_IW_SLC__1SDV_20220212T222803_20220212T222830_041886_04FCA3_2B3E',
    'S1A_IW_SLC__1SDV_20220212T222828_20220212T222855_041886_04FCA3_A3E2'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20220131T222803_20220131T222830_041711_04F690_8F5F',
    'S1A_IW_SLC__1SDV_20220131T222828_20220131T222855_041711_04F690_28D7'],
   'weather_model': 'HRES'},
  'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/2def9d71-30a3-4dba-9f28-ae8eb78797b8/2def9d71-30a3-4dba-9f28-ae8eb78797b8.log'],
  'expiration_time': '2023-12-20T00:00:00+00:00',
  'processing_times': [21633.269]},
 {'job_id': '134ac178-4b75-44a8-8bc0-6242a0a5de6b',
  'job_type': 'INSAR_ISCE_TEST',
  'request_time': '2023-06-21T23:49:11+00:00',
  'status_code': 'FAILED',
  'user_id': 'cmarshak',
  'name': 'Maine_164_HRES_0621',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 25502,
   'granules': ['S1A_IW_SLC__1SDV_20220212T222828_20220212T222855_041886_04FCA3_A3E2',
    'S1A_IW_SLC__1SDV_20220212T222853_20220212T222920_041886_04FCA3_FACC'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20220119T222828_20220119T222855_041536_04F09A_179F',
    'S1A_IW_SLC__1SDV_20220119T222853_20220119T222920_041536_04F09A_90FA'],
   'weather_model': 'HRES'},
  'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/134ac178-4b75-44a8-8bc0-6242a0a5de6b/134ac178-4b75-44a8-8bc0-6242a0a5de6b.log'],
  'expiration_time': '2023-12-20T00:00:00+00:00',
  'processing_times': [21631.273]}

Here is the end of one log:

...
100 17167  100 17167    0     0  39780      0 --:--:-- --:--:-- --:--:-- 39738
API open (R): swbdLat_N47_N50_Lon_W072_W067.wbd
API close:  swbdLat_N47_N50_Lon_W072_W067.wbd
Writing geotrans to VRT for swbdLat_N47_N50_Lon_W072_W067.wbd
post-process water body file
get SRTM tiles
get tiles with SRTM and without SWBD
get SWBD tiles
checking tile: N49W072.SRTMSWBD.raw.zip
checking tile: N49W071.SRTMSWBD.raw.zip
checking tile: N49W070.SRTMSWBD.raw.zip
checking tile: N49W069.SRTMSWBD.raw.zip
checking tile: N49W068.SRTMSWBD.raw.zip
checking tile: N48W072.SRTMSWBD.raw.zip
checking tile: N48W071.SRTMSWBD.raw.zip
checking tile: N48W070.SRTMSWBD.raw.zip
checking tile: N48W069.SRTMSWBD.raw.zip
checking tile: N48W068.SRTMSWBD.raw.zip
checking tile: N47W072.SRTMSWBD.raw.zip
checking tile: N47W071.SRTMSWBD.raw.zip
checking tile: N47W070.SRTMSWBD.raw.zip
checking tile: N47W069.SRTMSWBD.raw.zip
checking tile: N47W068.SRTMSWBD.raw.zip
check water body file
set areas outside of lat[-56, 60] to -2 (no data)

[cmarshak]

It looks like there is a timeout in this case - I am just confused as to why this timeout so early relative to the workflow (the downloads should take 5 - 10 minutes; timeout currently is 6 hours)? Is there issue with the standard i/o here in which there is another culprit that I am missing @asjohnston-asf?

https://github.com/ASFHyP3/hyp3/blob/develop/job_spec/INSAR_ISCE_TEST.yml#L88

[mgovorcin]

I run it locally these frame ids, and download water mask worked without any issues

[cmarshak]

Glad we can document this as a transient error.