New Ionosphere Updates do not always complete in Anchorage (Track 131)

cmarshak commented 1 year ago

Describe the bug I submitted 15 frames over Anchorage AK. 5/15 failed during the iono step. Here is a sample log:

Number of threads:            4
 Complex data interpolation
 Initializing Sinc interpolator
 Azimuth Carrier Poly
Polynomial Order: 5 - by - 3 
11967.9 -575.794    31.7785 0.954085    
-47241.6    2301.56 -111.96 -3.57909    
46620   -2299.58    97.1936 3.37479 
-9.04525e-09    1.14979e-08 -4.21725e-09    0   
6.02973e-09 -3.23611e-09    0   0   
-1.51145e-09    0   0   0   
 Range Carrier Poly
Polynomial Order: 0 - by - 0 
0   
 Range offsets poly
Polynomial Order: 0 - by - 0 
0   
 Azimuth offsets poly
Polynomial Order: 0 - by - 0 
0   
 Doppler poly
Polynomial Order: 0 - by - 2 
0.0333558   -50.0337    82464.7 
 Reading in the image
 At line         1000
 Interpolating image
 At line         1000
 Elapsed time:    15.9726562    
GDAL close: fine_offsets/IW3/range_09.off.vrt
GDAL close: fine_offsets/IW3/azimuth_09.off.vrt
GDAL close: ion/upper/fine_interferogram/IW3/secondary_burst_09.slc.vrt
API close:  ion/upper/fine_interferogram/IW3/secondary_resamp_burst_09.slc
API open (R): ion/upper/fine_interferogram/IW3/secondary_resamp_burst_09.slc
API close:  ion/upper/fine_interferogram/IW3/secondary_resamp_burst_09.slc
Adjust valid samples
Before:  274 23521
Offsets :  11.545783996582031 12.257086753845215
Adjust valid lines
Before:  27 1468
Offsets :  -1.5180559158325195 -1.4518603086471558

  0%|          | 0/28 [00:00<?, ?it/s]
Geo2radar mask IW2 /10:   0%|          | 0/28 [00:00<?, ?it/s]
Geo2radar mask IW2 /10:   4%|â–Ž         | 1/28 [00:02<00:55,  2.04s/it]
Geo2radar mask IW2 /05:   4%|â–Ž         | 1/28 [00:02<00:55,  2.04s/it]
Geo2radar mask IW2 /05:   7%|â–‹         | 2/28 [00:04<01:03,  2.44s/it]
Geo2radar mask IW2 /01:   7%|â–‹         | 2/28 [00:04<01:03,  2.44s/it]
Geo2radar mask IW2 /01:  11%|â–ˆ         | 3/28 [00:07<00:59,  2.40s/it]
Geo2radar mask IW2 /03:  11%|â–ˆ         | 3/28 [00:07<00:59,  2.40s/it]
Geo2radar mask IW2 /03:  14%|â–ˆâ–        | 4/28 [00:09<00:57,  2.41s/it]
Geo2radar mask IW2 /08:  14%|â–ˆâ–        | 4/28 [00:09<00:57,  2.41s/it]
Geo2radar mask IW2 /08:  18%|â–ˆâ–Š        | 5/28 [00:11<00:54,  2.38s/it]
Geo2radar mask IW2 /09:  18%|â–ˆâ–Š        | 5/28 [00:11<00:54,  2.38s/it]
Geo2radar mask IW2 /09:  21%|â–ˆâ–ˆâ–       | 6/28 [00:14<00:52,  2.41s/it]
Geo2radar mask IW2 /06:  21%|â–ˆâ–ˆâ–       | 6/28 [00:14<00:52,  2.41s/it]
Geo2radar mask IW2 /06:  25%|â–ˆâ–ˆâ–Œ       | 7/28 [00:16<00:49,  2.37s/it]
Geo2radar mask IW2 /07:  25%|â–ˆâ–ˆâ–Œ       | 7/28 [00:16<00:49,  2.37s/it]
Geo2radar mask IW2 /07:  29%|â–ˆâ–ˆâ–Š       | 8/28 [00:19<00:47,  2.39s/it]
Geo2radar mask IW2 /04:  29%|â–ˆâ–ˆâ–Š       | 8/28 [00:19<00:47,  2.39s/it]
Geo2radar mask IW2 /04:  32%|â–ˆâ–ˆâ–ˆâ–      | 9/28 [00:21<00:45,  2.37s/it]
Geo2radar mask IW2 /02:  32%|â–ˆâ–ˆâ–ˆâ–      | 9/28 [00:21<00:45,  2.37s/it]
Geo2radar mask IW2 /02:  36%|â–ˆâ–ˆâ–ˆâ–Œ      | 10/28 [00:23<00:42,  2.38s/it]
Geo2radar mask IW1 /05:  36%|â–ˆâ–ˆâ–ˆâ–Œ      | 10/28 [00:23<00:42,  2.38s/it]
Geo2radar mask IW1 /05:  39%|â–ˆâ–ˆâ–ˆâ–‰      | 11/28 [00:25<00:37,  2.18s/it]
Geo2radar mask IW1 /01:  39%|â–ˆâ–ˆâ–ˆâ–‰      | 11/28 [00:25<00:37,  2.18s/it]
Geo2radar mask IW1 /01:  43%|â–ˆâ–ˆâ–ˆâ–ˆâ–Ž     | 12/28 [00:27<00:32,  2.04s/it]
Geo2radar mask IW1 /03:  43%|â–ˆâ–ˆâ–ˆâ–ˆâ–Ž     | 12/28 [00:27<00:32,  2.04s/it]
Geo2radar mask IW1 /03:  46%|â–ˆâ–ˆâ–ˆâ–ˆâ–‹     | 13/28 [00:28<00:29,  1.95s/it]
Geo2radar mask IW1 /08:  46%|â–ˆâ–ˆâ–ˆâ–ˆâ–‹     | 13/28 [00:28<00:29,  1.95s/it]
Geo2radar mask IW1 /08:  50%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆ     | 14/28 [00:30<00:26,  1.90s/it]
Geo2radar mask IW1 /09:  50%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆ     | 14/28 [00:30<00:26,  1.90s/it]
Geo2radar mask IW1 /09:  54%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–Ž    | 15/28 [00:32<00:24,  1.87s/it]
Geo2radar mask IW1 /06:  54%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–Ž    | 15/28 [00:32<00:24,  1.87s/it]
Geo2radar mask IW1 /06:  57%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–‹    | 16/28 [00:34<00:22,  1.86s/it]
Geo2radar mask IW1 /07:  57%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–‹    | 16/28 [00:34<00:22,  1.86s/it]
Geo2radar mask IW1 /07:  61%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆ    | 17/28 [00:36<00:20,  1.84s/it]
Geo2radar mask IW1 /04:  61%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆ    | 17/28 [00:36<00:20,  1.84s/it]
Geo2radar mask IW1 /04:  64%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–   | 18/28 [00:37<00:18,  1.81s/it]
Geo2radar mask IW1 /02:  64%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–   | 18/28 [00:37<00:18,  1.81s/it]
Geo2radar mask IW1 /02:  68%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–Š   | 19/28 [00:39<00:16,  1.80s/it]
Geo2radar mask IW3 /05:  68%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–Š   | 19/28 [00:39<00:16,  1.80s/it]
Geo2radar mask IW3 /05:  71%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–  | 20/28 [00:42<00:15,  1.98s/it]
Geo2radar mask IW3 /01:  71%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–  | 20/28 [00:42<00:15,  1.98s/it]
Geo2radar mask IW3 /01:  75%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–Œ  | 21/28 [00:44<00:14,  2.05s/it]
Geo2radar mask IW3 /03:  75%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–Œ  | 21/28 [00:44<00:14,  2.05s/it]
Geo2radar mask IW3 /03:  79%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–Š  | 22/28 [00:46<00:12,  2.13s/it]
Geo2radar mask IW3 /08:  79%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–Š  | 22/28 [00:46<00:12,  2.13s/it]
Geo2radar mask IW3 /08:  82%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ– | 23/28 [00:48<00:10,  2.16s/it]
Geo2radar mask IW3 /09:  82%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ– | 23/28 [00:48<00:10,  2.16s/it]
Geo2radar mask IW3 /09:  86%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–Œ | 24/28 [00:51<00:08,  2.20s/it]
Geo2radar mask IW3 /06:  86%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–Œ | 24/28 [00:51<00:08,  2.20s/it]
Geo2radar mask IW3 /06:  89%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–‰ | 25/28 [00:53<00:06,  2.21s/it]
Geo2radar mask IW3 /07:  89%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–‰ | 25/28 [00:53<00:06,  2.21s/it]
Geo2radar mask IW3 /07:  93%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–Ž| 26/28 [00:55<00:04,  2.24s/it]
Geo2radar mask IW3 /04:  93%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–Ž| 26/28 [00:55<00:04,  2.24s/it]
Geo2radar mask IW3 /04:  96%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–‹| 27/28 [00:57<00:02,  2.24s/it]
Geo2radar mask IW3 /02:  96%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–‹| 27/28 [00:57<00:02,  2.24s/it]
Geo2radar mask IW3 /02: 100%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆ| 28/28 [01:00<00:00,  2.26s/it]
Geo2radar mask IW3 /02: 100%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆ| 28/28 [01:00<00:00,  2.15s/it]

  0%|          | 0/56 [00:00<?, ?it/s]
Masking lower-iono interferograms IW2/10:   0%|          | 0/56 [00:00<?, ?it/s]
Masking lower-iono interferograms IW2/10:   2%|â–         | 1/56 [00:00<00:30,  1.79it/s]
Masking lower-iono interferograms IW2/03:   2%|â–         | 1/56 [00:00<00:30,  1.79it/s]
Masking lower-iono interferograms IW2/03:   4%|â–Ž         | 2/56 [00:01<00:42,  1.27it/s]
Masking lower-iono interferograms IW2/08:   4%|â–Ž         | 2/56 [00:01<00:42,  1.27it/s]
Masking lower-iono interferograms IW2/08:   5%|â–Œ         | 3/56 [00:02<00:49,  1.06it/s]
Masking lower-iono interferograms IW2/02:   5%|â–Œ         | 3/56 [00:02<00:49,  1.06it/s]
Masking lower-iono interferograms IW2/02:   7%|â–‹         | 4/56 [00:03<00:55,  1.07s/it]
Masking lower-iono interferograms IW2/04:   7%|â–‹         | 4/56 [00:03<00:55,  1.07s/it]
Masking lower-iono interferograms IW2/04:   9%|â–‰         | 5/56 [00:04<00:52,  1.03s/it]
Masking lower-iono interferograms IW2/05:   9%|â–‰         | 5/56 [00:04<00:52,  1.03s/it]
Masking lower-iono interferograms IW2/05:  11%|â–ˆ         | 6/56 [00:06<00:56,  1.13s/it]
Masking lower-iono interferograms IW2/06:  11%|â–ˆ         | 6/56 [00:06<00:56,  1.13s/it]
Masking lower-iono interferograms IW2/06:  12%|â–ˆâ–Ž        | 7/56 [00:07<01:05,  1.34s/it]
Masking lower-iono interferograms IW2/01:  12%|â–ˆâ–Ž        | 7/56 [00:07<01:05,  1.34s/it]
Masking lower-iono interferograms IW2/01:  14%|â–ˆâ–        | 8/56 [00:09<01:13,  1.53s/it]
Masking lower-iono interferograms IW2/07:  14%|â–ˆâ–        | 8/56 [00:09<01:13,  1.53s/it]
Masking lower-iono interferograms IW2/07:  16%|â–ˆâ–Œ        | 9/56 [00:11<01:18,  1.67s/it]
Masking lower-iono interferograms IW2/09:  16%|â–ˆâ–Œ        | 9/56 [00:11<01:18,  1.67s/it]
Masking lower-iono interferograms IW2/09:  18%|â–ˆâ–Š        | 10/56 [00:13<01:19,  1.73s/it]
Masking lower-iono interferograms IW1/03:  18%|â–ˆâ–Š        | 10/56 [00:13<01:19,  1.73s/it]
Masking lower-iono interferograms IW1/03:  20%|â–ˆâ–‰        | 11/56 [00:14<01:08,  1.53s/it]
Masking lower-iono interferograms IW1/08:  20%|â–ˆâ–‰        | 11/56 [00:14<01:08,  1.53s/it]
Masking lower-iono interferograms IW1/08:  21%|â–ˆâ–ˆâ–       | 12/56 [00:16<01:13,  1.67s/it]
Masking lower-iono interferograms IW1/02:  21%|â–ˆâ–ˆâ–       | 12/56 [00:16<01:13,  1.67s/it]
Masking lower-iono interferograms IW1/02:  23%|â–ˆâ–ˆâ–Ž       | 13/56 [00:18<01:07,  1.57s/it]
Masking lower-iono interferograms IW1/04:  23%|â–ˆâ–ˆâ–Ž       | 13/56 [00:18<01:07,  1.57s/it]
Masking lower-iono interferograms IW1/04:  25%|â–ˆâ–ˆâ–Œ       | 14/56 [00:19<01:03,  1.50s/it]
Masking lower-iono interferograms IW1/05:  25%|â–ˆâ–ˆâ–Œ       | 14/56 [00:19<01:03,  1.50s/it]
Masking lower-iono interferograms IW1/05:  27%|â–ˆâ–ˆâ–‹       | 15/56 [00:21<01:02,  1.53s/it]
Masking lower-iono interferograms IW1/06:  27%|â–ˆâ–ˆâ–‹       | 15/56 [00:21<01:02,  1.53s/it]
Masking lower-iono interferograms IW1/06:  29%|â–ˆâ–ˆâ–Š       | 16/56 [00:22<01:04,  1.61s/it]
Masking lower-iono interferograms IW1/01:  29%|â–ˆâ–ˆâ–Š       | 16/56 [00:22<01:04,  1.61s/it]
Masking lower-iono interferograms IW1/01:  30%|â–ˆâ–ˆâ–ˆ       | 17/56 [00:24<01:04,  1.67s/it]
Masking lower-iono interferograms IW1/07:  30%|â–ˆâ–ˆâ–ˆ       | 17/56 [00:24<01:04,  1.67s/it]
Masking lower-iono interferograms IW1/07:  32%|â–ˆâ–ˆâ–ˆâ–      | 18/56 [00:26<01:04,  1.70s/it]
Masking lower-iono interferograms IW1/09:  32%|â–ˆâ–ˆâ–ˆâ–      | 18/56 [00:26<01:04,  1.70s/it]
Masking lower-iono interferograms IW1/09:  34%|â–ˆâ–ˆâ–ˆâ–      | 19/56 [00:28<01:03,  1.72s/it]
Masking lower-iono interferograms IW3/03:  34%|â–ˆâ–ˆâ–ˆâ–      | 19/56 [00:28<01:03,  1.72s/it]
Masking lower-iono interferograms IW3/03:  36%|â–ˆâ–ˆâ–ˆâ–Œ      | 20/56 [00:30<01:04,  1.80s/it]
Masking lower-iono interferograms IW3/08:  36%|â–ˆâ–ˆâ–ˆâ–Œ      | 20/56 [00:30<01:04,  1.80s/it]
Masking lower-iono interferograms IW3/08:  38%|â–ˆâ–ˆâ–ˆâ–Š      | 21/56 [00:32<01:06,  1.89s/it]
Masking lower-iono interferograms IW3/02:  38%|â–ˆâ–ˆâ–ˆâ–Š      | 21/56 [00:32<01:06,  1.89s/it]
Masking lower-iono interferograms IW3/02:  39%|â–ˆâ–ˆâ–ˆâ–‰      | 22/56 [00:34<01:11,  2.09s/it]
Masking lower-iono interferograms IW3/04:  39%|â–ˆâ–ˆâ–ˆâ–‰      | 22/56 [00:34<01:11,  2.09s/it]
Masking lower-iono interferograms IW3/04:  41%|â–ˆâ–ˆâ–ˆâ–ˆ      | 23/56 [00:36<01:08,  2.09s/it]
Masking lower-iono interferograms IW3/05:  41%|â–ˆâ–ˆâ–ˆâ–ˆ      | 23/56 [00:36<01:08,  2.09s/it]
Masking lower-iono interferograms IW3/05:  43%|â–ˆâ–ˆâ–ˆâ–ˆâ–Ž     | 24/56 [00:38<01:05,  2.05s/it]
Masking lower-iono interferograms IW3/06:  43%|â–ˆâ–ˆâ–ˆâ–ˆâ–Ž     | 24/56 [00:38<01:05,  2.05s/it]
Masking lower-iono interferograms IW3/06:  45%|â–ˆâ–ˆâ–ˆâ–ˆâ–     | 25/56 [00:40<01:02,  2.03s/it]
Masking lower-iono interferograms IW3/01:  45%|â–ˆâ–ˆâ–ˆâ–ˆâ–     | 25/56 [00:40<01:02,  2.03s/it]
Masking lower-iono interferograms IW3/01:  46%|â–ˆâ–ˆâ–ˆâ–ˆâ–‹     | 26/56 [00:42<01:00,  2.01s/it]
Masking lower-iono interferograms IW3/07:  46%|â–ˆâ–ˆâ–ˆâ–ˆâ–‹     | 26/56 [00:42<01:00,  2.01s/it]
Masking lower-iono interferograms IW3/07:  48%|â–ˆâ–ˆâ–ˆâ–ˆâ–Š     | 27/56 [00:44<00:57,  1.99s/it]
Masking lower-iono interferograms IW3/09:  48%|â–ˆâ–ˆâ–ˆâ–ˆâ–Š     | 27/56 [00:44<00:57,  1.99s/it]
Masking lower-iono interferograms IW3/09:  50%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆ     | 28/56 [00:46<00:55,  1.99s/it]
Masking upper-iono interferograms IW2/10:  50%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆ     | 28/56 [00:46<00:55,  1.99s/it]
Masking upper-iono interferograms IW2/10:  52%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–    | 29/56 [00:48<00:54,  2.00s/it]
Masking upper-iono interferograms IW2/03:  52%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–    | 29/56 [00:48<00:54,  2.00s/it]
Masking upper-iono interferograms IW2/03:  54%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–Ž    | 30/56 [00:50<00:52,  2.00s/it]
Masking upper-iono interferograms IW2/08:  54%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–Ž    | 30/56 [00:50<00:52,  2.00s/it]
Masking upper-iono interferograms IW2/08:  55%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–Œ    | 31/56 [00:52<00:50,  2.01s/it]
Masking upper-iono interferograms IW2/02:  55%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–Œ    | 31/56 [00:52<00:50,  2.01s/it]
Masking upper-iono interferograms IW2/02:  57%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–‹    | 32/56 [00:54<00:48,  2.02s/it]
Masking upper-iono interferograms IW2/04:  57%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–‹    | 32/56 [00:54<00:48,  2.02s/it]
Masking upper-iono interferograms IW2/04:  59%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–‰    | 33/56 [00:56<00:46,  2.02s/it]
Masking upper-iono interferograms IW2/05:  59%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–‰    | 33/56 [00:56<00:46,  2.02s/it]
Masking upper-iono interferograms IW2/05:  61%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆ    | 34/56 [00:58<00:44,  2.02s/it]
Masking upper-iono interferograms IW2/06:  61%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆ    | 34/56 [00:58<00:44,  2.02s/it]
Masking upper-iono interferograms IW2/06:  62%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–Ž   | 35/56 [01:00<00:42,  2.02s/it]
Masking upper-iono interferograms IW2/01:  62%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–Ž   | 35/56 [01:00<00:42,  2.02s/it]
Masking upper-iono interferograms IW2/01:  64%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–   | 36/56 [01:02<00:40,  2.02s/it]
Masking upper-iono interferograms IW2/07:  64%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–   | 36/56 [01:02<00:40,  2.02s/it]
Masking upper-iono interferograms IW2/07:  66%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–Œ   | 37/56 [01:04<00:38,  2.02s/it]
Masking upper-iono interferograms IW2/09:  66%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–Œ   | 37/56 [01:04<00:38,  2.02s/it]
Masking upper-iono interferograms IW2/09:  68%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–Š   | 38/56 [01:07<00:36,  2.02s/it]
Masking upper-iono interferograms IW1/03:  68%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–Š   | 38/56 [01:07<00:36,  2.02s/it]
Masking upper-iono interferograms IW1/03:  70%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–‰   | 39/56 [01:08<00:32,  1.91s/it]
Masking upper-iono interferograms IW1/08:  70%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–‰   | 39/56 [01:08<00:32,  1.91s/it]
Masking upper-iono interferograms IW1/08:  71%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–  | 40/56 [01:10<00:29,  1.84s/it]
Masking upper-iono interferograms IW1/02:  71%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–  | 40/56 [01:10<00:29,  1.84s/it]
Masking upper-iono interferograms IW1/02:  73%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–Ž  | 41/56 [01:12<00:27,  1.81s/it]
Masking upper-iono interferograms IW1/04:  73%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–Ž  | 41/56 [01:12<00:27,  1.81s/it]
Masking upper-iono interferograms IW1/04:  75%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–Œ  | 42/56 [01:13<00:24,  1.77s/it]
Masking upper-iono interferograms IW1/05:  75%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–Œ  | 42/56 [01:13<00:24,  1.77s/it]
Masking upper-iono interferograms IW1/05:  77%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–‹  | 43/56 [01:15<00:22,  1.75s/it]
Masking upper-iono interferograms IW1/06:  77%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–‹  | 43/56 [01:15<00:22,  1.75s/it]
Masking upper-iono interferograms IW1/06:  79%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–Š  | 44/56 [01:17<00:20,  1.74s/it]
Masking upper-iono interferograms IW1/01:  79%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–Š  | 44/56 [01:17<00:20,  1.74s/it]
Masking upper-iono interferograms IW1/01:  80%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆ  | 45/56 [01:18<00:18,  1.73s/it]
Masking upper-iono interferograms IW1/07:  80%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆ  | 45/56 [01:18<00:18,  1.73s/it]
Masking upper-iono interferograms IW1/07:  82%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ– | 46/56 [01:20<00:17,  1.72s/it]
Masking upper-iono interferograms IW1/09:  82%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ– | 46/56 [01:20<00:17,  1.72s/it]
Masking upper-iono interferograms IW1/09:  84%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ– | 47/56 [01:22<00:15,  1.71s/it]
Masking upper-iono interferograms IW3/03:  84%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ– | 47/56 [01:22<00:15,  1.71s/it]
Masking upper-iono interferograms IW3/03:  86%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–Œ | 48/56 [01:24<00:14,  1.79s/it]
Masking upper-iono interferograms IW3/08:  86%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–Œ | 48/56 [01:24<00:14,  1.79s/it]
Masking upper-iono interferograms IW3/08:  88%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–Š | 49/56 [01:26<00:12,  1.85s/it]
Masking upper-iono interferograms IW3/02:  88%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–Š | 49/56 [01:26<00:12,  1.85s/it]
Masking upper-iono interferograms IW3/02:  89%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–‰ | 50/56 [01:28<00:11,  1.88s/it]
Masking upper-iono interferograms IW3/04:  89%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–‰ | 50/56 [01:28<00:11,  1.88s/it]
Masking upper-iono interferograms IW3/04:  91%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆ | 51/56 [01:30<00:09,  1.90s/it]
Masking upper-iono interferograms IW3/05:  91%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆ | 51/56 [01:30<00:09,  1.90s/it]
Masking upper-iono interferograms IW3/05:  93%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–Ž| 52/56 [01:32<00:07,  1.93s/it]
Masking upper-iono interferograms IW3/06:  93%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–Ž| 52/56 [01:32<00:07,  1.93s/it]
Masking upper-iono interferograms IW3/06:  95%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–| 53/56 [01:34<00:05,  1.94s/it]
Masking upper-iono interferograms IW3/01:  95%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–| 53/56 [01:34<00:05,  1.94s/it]
Masking upper-iono interferograms IW3/01:  96%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–‹| 54/56 [01:36<00:03,  1.94s/it]
Masking upper-iono interferograms IW3/07:  96%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–‹| 54/56 [01:36<00:03,  1.94s/it]
Masking upper-iono interferograms IW3/07:  98%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–Š| 55/56 [01:37<00:01,  1.95s/it]
Masking upper-iono interferograms IW3/09:  98%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–Š| 55/56 [01:37<00:01,  1.95s/it]
Masking upper-iono interferograms IW3/09: 100%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆ| 56/56 [01:39<00:00,  1.96s/it]
Masking upper-iono interferograms IW3/09: 100%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆ| 56/56 [01:39<00:00,  1.78s/it]
after adjustment (index in merged image, index starts from 1): 
 burst     fl      ll      fc      lc         fl      ll      fc      lc         fl      ll      fc      lc       
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
   1         51    1450     601   20600        501    1900   20601   44400       2301    3750   44401   67200    
   2       1451    2800     601   20600       1901    3250   20601   44400       3751    5100   44401   67200    
   3       2801    4150     601   20600       3251    4600   20601   44400       5101    6400   44401   67200    
   4       4151    5500     601   20600       4601    5950   20601   44400       6401    7750   44401   67200    
   5       5501    6850     601   20600       5951    7300   20601   44400       7751    9100   44401   67200    
   6       6851    8150     601   20600       7301    8650   20601   44400       9101   10450   44401   67200    
   7       8151    9500     401   20600       8651   10000   20601   44400      10451   11800   44401   67200    
   8       9501   10850     401   20600      10001   11300   20601   44400      11801   13150   44401   67200    
   9      10851   12200     401   20600      11301   12650   20601   44400      13151   14450   44401   67200    
  10      12651   14000   20201   44400    
Multilooking ion/lower/merged/topophase.flat.full ...
GDAL open (R): ion/lower/merged/topophase.flat.full.vrt
API open (WR): ion/lower/merged/topophase.flat
GDAL close: ion/lower/merged/topophase.flat.full.vrt
API close:  ion/lower/merged/topophase.flat
API open (R): ion/lower/merged/topophase.flat
API close:  ion/lower/merged/topophase.flat
Writing geotrans to VRT for ion/lower/merged/topophase.flat
after adjustment (index in merged image, index starts from 1): 
 burst     fl      ll      fc      lc         fl      ll      fc      lc         fl      ll      fc      lc       
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
   1         51    1450     601   20600        501    1900   20601   44400       2301    3750   44401   67200    
   2       1451    2800     601   20600       1901    3250   20601   44400       3751    5100   44401   67200    
   3       2801    4150     601   20600       3251    4600   20601   44400       5101    6400   44401   67200    
   4       4151    5500     601   20600       4601    5950   20601   44400       6401    7750   44401   67200    
   5       5501    6850     601   20600       5951    7300   20601   44400       7751    9100   44401   67200    
   6       6851    8150     601   20600       7301    8650   20601   44400       9101   10450   44401   67200    
   7       8151    9500     401   20600       8651   10000   20601   44400      10451   11800   44401   67200    
   8       9501   10850     401   20600      10001   11300   20601   44400      11801   13150   44401   67200    
   9      10851   12200     401   20600      11301   12650   20601   44400      13151   14450   44401   67200    
  10      12651   14000   20201   44400    
Multilooking ion/upper/merged/topophase.flat.full ...
GDAL open (R): ion/upper/merged/topophase.flat.full.vrt
API open (WR): ion/upper/merged/topophase.flat
GDAL close: ion/upper/merged/topophase.flat.full.vrt
API close:  ion/upper/merged/topophase.flat
API open (R): ion/upper/merged/topophase.flat
API close:  ion/upper/merged/topophase.flat
Writing geotrans to VRT for ion/upper/merged/topophase.flat
unwrapping lower and upper band interferograms
snaphu v1.4.2
Reading wrapped phase from file ion/lower/merged/topophase.flat
No weight file specified.  Assuming uniform weights
Reading correlation data from file ion/lower/merged/topophase.cor
Calculating smooth-solution cost parameters
Initializing flows with MCF algorithm
Integrating phase
Writing output to file ion/lower/merged/filt_topophase.unw
Growing connected components on second pass
snaphu v1.4.2
Reading unwrapped phase from file ion/lower/merged/filt_topophase.unw
No weight file specified.  Assuming uniform weights
Reading correlation data from file ion/lower/merged/topophase.cor
Calculating smooth-solution cost parameters
Growing connected component mask
Writing connected components to file ion/lower/merged/filt_topophase.unw.conncomp
API open (R): ion/lower/merged/filt_topophase.unw
API close:  ion/lower/merged/filt_topophase.unw
GDAL open (R): ion/lower/merged/filt_topophase.unw.vrt
GDAL close: ion/lower/merged/filt_topophase.unw.vrt
API open (R): ion/lower/merged/filt_topophase.unw.conncomp
API close:  ion/lower/merged/filt_topophase.unw.conncomp
GDAL open (R): ion/lower/merged/filt_topophase.unw.conncomp.vrt
GDAL close: ion/lower/merged/filt_topophase.unw.conncomp.vrt
imageMath.py -e='a_0*(abs(b)!=0);a_1*(abs(b)!=0)' --a=tmp.unw --b=ion/lower/merged/topophase.flat -s BIL -o=ion/lower/merged/filt_topophase.unw
GDAL open (R): ion/lower/merged/filt_topophase.unw.vrt
GDAL close: ion/lower/merged/filt_topophase.unw.vrt
This is the Open Source version of ISCE.
Some of the workflows depend on a separate licensed package.
To obtain the licensed package, please make a request for ISCE
through the website: https://download.jpl.nasa.gov/ops/request/index.cfm.
Alternatively, if you are a member, or can become a member of WinSAR
you may be able to obtain access to a version of the licensed sofware at
https://winsar.unavco.org/software/isce
args:  Namespace(hh=None, equation='a_0*(abs(b)!=0);a_1*(abs(b)!=0)', out='ion/lower/merged/filt_topophase.unw', scheme='BIL', dtype='float', debug=False, noxml=False)
files:  ['--a=tmp.unw', '--b=ion/lower/merged/topophase.flat']
Bounding boxes dont match. Not adding bbox info.
snaphu v1.4.2
Reading wrapped phase from file ion/upper/merged/topophase.flat
No weight file specified.  Assuming uniform weights
Reading correlation data from file ion/lower/merged/topophase.cor
Calculating smooth-solution cost parameters
Initializing flows with MCF algorithm
Integrating phase
Writing output to file ion/upper/merged/filt_topophase.unw
Growing connected components on second pass
snaphu v1.4.2
Reading unwrapped phase from file ion/upper/merged/filt_topophase.unw
No weight file specified.  Assuming uniform weights
Reading correlation data from file ion/lower/merged/topophase.cor
Calculating smooth-solution cost parameters
Growing connected component mask
Writing connected components to file ion/upper/merged/filt_topophase.unw.conncomp
API open (R): ion/upper/merged/filt_topophase.unw
API close:  ion/upper/merged/filt_topophase.unw
GDAL open (R): ion/upper/merged/filt_topophase.unw.vrt
GDAL close: ion/upper/merged/filt_topophase.unw.vrt
API open (R): ion/upper/merged/filt_topophase.unw.conncomp
API close:  ion/upper/merged/filt_topophase.unw.conncomp
GDAL open (R): ion/upper/merged/filt_topophase.unw.conncomp.vrt
GDAL close: ion/upper/merged/filt_topophase.unw.conncomp.vrt
imageMath.py -e='a_0*(abs(b)!=0);a_1*(abs(b)!=0)' --a=tmp.unw --b=ion/upper/merged/topophase.flat -s BIL -o=ion/upper/merged/filt_topophase.unw
GDAL open (R): ion/upper/merged/filt_topophase.unw.vrt
GDAL close: ion/upper/merged/filt_topophase.unw.vrt
This is the Open Source version of ISCE.
Some of the workflows depend on a separate licensed package.
To obtain the licensed package, please make a request for ISCE
through the website: https://download.jpl.nasa.gov/ops/request/index.cfm.
Alternatively, if you are a member, or can become a member of WinSAR
you may be able to obtain access to a version of the licensed sofware at
https://winsar.unavco.org/software/isce
args:  Namespace(hh=None, equation='a_0*(abs(b)!=0);a_1*(abs(b)!=0)', out='ion/upper/merged/filt_topophase.unw', scheme='BIL', dtype='float', debug=False, noxml=False)
files:  ['--a=tmp.unw', '--b=ion/upper/merged/topophase.flat']
Bounding boxes dont match. Not adding bbox info.
Do bridging ion/lower/merged/filt_topophase.unw
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
imageMath.py -e='a_0*(abs(b)!=0);a_1*(abs(b)!=0)' --a=tmp.unw --b=ion/lower/merged/filt_topophase.unw.conncomp -s BIL -o=ion/lower/merged/filt_topophase.unw
GDAL open (R): ion/lower/merged/filt_topophase.unw.vrt
GDAL close: ion/lower/merged/filt_topophase.unw.vrt
This is the Open Source version of ISCE.
Some of the workflows depend on a separate licensed package.
To obtain the licensed package, please make a request for ISCE
through the website: https://download.jpl.nasa.gov/ops/request/index.cfm.
Alternatively, if you are a member, or can become a member of WinSAR
you may be able to obtain access to a version of the licensed sofware at
https://winsar.unavco.org/software/isce
args:  Namespace(hh=None, equation='a_0*(abs(b)!=0);a_1*(abs(b)!=0)', out='ion/lower/merged/filt_topophase.unw', scheme='BIL', dtype='float', debug=False, noxml=False)
files:  ['--a=tmp.unw', '--b=ion/lower/merged/filt_topophase.unw.conncomp']
Do bridging ion/upper/merged/filt_topophase.unw
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
imageMath.py -e='a_0*(abs(b)!=0);a_1*(abs(b)!=0)' --a=tmp.unw --b=ion/upper/merged/filt_topophase.unw.conncomp -s BIL -o=ion/upper/merged/filt_topophase.unw
GDAL open (R): ion/upper/merged/filt_topophase.unw.vrt
GDAL close: ion/upper/merged/filt_topophase.unw.vrt
This is the Open Source version of ISCE.
Some of the workflows depend on a separate licensed package.
To obtain the licensed package, please make a request for ISCE
through the website: https://download.jpl.nasa.gov/ops/request/index.cfm.
Alternatively, if you are a member, or can become a member of WinSAR
you may be able to obtain access to a version of the licensed sofware at
https://winsar.unavco.org/software/isce
args:  Namespace(hh=None, equation='a_0*(abs(b)!=0);a_1*(abs(b)!=0)', out='ion/upper/merged/filt_topophase.unw', scheme='BIL', dtype='float', debug=False, noxml=False)
files:  ['--a=tmp.unw', '--b=ion/upper/merged/filt_topophase.unw.conncomp']
Writing geotrans to VRT for ion/lower/merged/40rlks_10alks/topophase.flat
Writing geotrans to VRT for ion/upper/merged/40rlks_10alks/topophase.flat
Multilooking ion/lower/merged/40rlks_10alks/topophase.flat ...
GDAL open (R): ion/lower/merged/40rlks_10alks/topophase.flat.vrt
API open (WR): ion/lower/merged/topophase.flat
GDAL close: ion/lower/merged/40rlks_10alks/topophase.flat.vrt
API close:  ion/lower/merged/topophase.flat
API open (R): ion/lower/merged/topophase.flat
API close:  ion/lower/merged/topophase.flat
Writing geotrans to VRT for ion/lower/merged/topophase.flat
Multilooking ion/upper/merged/40rlks_10alks/topophase.flat ...
GDAL open (R): ion/upper/merged/40rlks_10alks/topophase.flat.vrt
API open (WR): ion/upper/merged/topophase.flat
GDAL close: ion/upper/merged/40rlks_10alks/topophase.flat.vrt
API close:  ion/upper/merged/topophase.flat
API open (R): ion/upper/merged/topophase.flat
API close:  ion/upper/merged/topophase.flat
Writing geotrans to VRT for ion/upper/merged/topophase.flat
computing ionosphere
after adjustment:
Traceback (most recent call last):
  File "/opt/conda/envs/topsapp_env/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/conda/envs/topsapp_env/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/__main__.py", line 352, in <module>
    main()
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/__main__.py", line 348, in main
    sys.exit(process_entry_point.load()())
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/__main__.py", line 207, in gunw_slc
    iono_processing(
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/iono_proc.py", line 118, in iono_processing
    runIon.ionosphere(topsapp, ionParam)
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce/components/isceobj/TopsProc/runIon.py", line 1308, in ionosphere
    ionos = computeIonosphere(lowerUnw, upperUnw, cor, fl, fu, adjFlag, corThresholdAdj, 0)
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce/components/isceobj/TopsProc/runIon.py", line 1250, in computeIonosphere
    print('max phase difference: {}'.format(np.amax(unw_diff)))
  File "<__array_function__ internals>", line 180, in amax
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/numpy/core/fromnumeric.py", line 2793, in amax
    return _wrapreduction(a, np.maximum, 'max', axis, None, out,
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/numpy/core/fromnumeric.py", line 86, in _wrapreduction
    return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
ValueError: zero-size array to reduction operation maximum which has no identity

All the failed jobs are here:

{'expiration_time': '2023-11-22T00:00:00+00:00',
 'job_id': 'b9a37165-8c1c-4104-a52a-99e6bcd28a2a',
 'job_parameters': {'compute_solid_earth_tide': True,
                    'estimate_ionosphere_delay': True,
                    'frame_id': 20342,
                    'granules': ['S1A_IW_SLC__1SDV_20220210T162911_20220210T162938_041853_04FB8A_974D',
                                 'S1A_IW_SLC__1SDV_20220210T162936_20220210T163004_041853_04FB8A_4E1F'],
                    'secondary_granules': ['S1A_IW_SLC__1SDV_20220117T162912_20220117T162939_041503_04EF80_77A2',
                                           'S1A_IW_SLC__1SDV_20220117T162937_20220117T163004_041503_04EF80_0BFA'],
                    'weather_model': 'HRRR'},
 'job_type': 'INSAR_ISCE_TEST',
 'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/b9a37165-8c1c-4104-a52a-99e6bcd28a2a/b9a37165-8c1c-4104-a52a-99e6bcd28a2a.log'],
 'name': 'Anchorage_131_HRRR_0524',
 'processing_times': [9269.375],
 'request_time': '2023-05-24T23:49:30+00:00',
 'status_code': 'FAILED',
 'user_id': 'cmarshak'}
{'expiration_time': '2023-11-22T00:00:00+00:00',
 'job_id': 'de0e0133-39b4-4bc1-9c44-352bbdb400a8',
 'job_parameters': {'compute_solid_earth_tide': True,
                    'estimate_ionosphere_delay': True,
                    'frame_id': 20342,
                    'granules': ['S1A_IW_SLC__1SDV_20220129T162911_20220129T162938_041678_04F56E_51C5',
                                 'S1A_IW_SLC__1SDV_20220129T162936_20220129T163004_041678_04F56E_60B3'],
                    'secondary_granules': ['S1A_IW_SLC__1SDV_20220117T162912_20220117T162939_041503_04EF80_77A2',
                                           'S1A_IW_SLC__1SDV_20220117T162937_20220117T163004_041503_04EF80_0BFA'],
                    'weather_model': 'HRRR'},
 'job_type': 'INSAR_ISCE_TEST',
 'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/de0e0133-39b4-4bc1-9c44-352bbdb400a8/de0e0133-39b4-4bc1-9c44-352bbdb400a8.log'],
 'name': 'Anchorage_131_HRRR_0524',
 'processing_times': [9132.298],
 'request_time': '2023-05-24T23:49:30+00:00',
 'status_code': 'FAILED',
 'user_id': 'cmarshak'}
{'expiration_time': '2023-11-22T00:00:00+00:00',
 'job_id': '212c21e5-61df-4f58-aabc-84f437e3ab76',
 'job_parameters': {'compute_solid_earth_tide': True,
                    'estimate_ionosphere_delay': True,
                    'frame_id': 20342,
                    'granules': ['S1A_IW_SLC__1SDV_20220210T162911_20220210T162938_041853_04FB8A_974D',
                                 'S1A_IW_SLC__1SDV_20220210T162936_20220210T163004_041853_04FB8A_4E1F'],
                    'secondary_granules': ['S1A_IW_SLC__1SDV_20220129T162911_20220129T162938_041678_04F56E_51C5',
                                           'S1A_IW_SLC__1SDV_20220129T162936_20220129T163004_041678_04F56E_60B3'],
                    'weather_model': 'HRRR'},
 'job_type': 'INSAR_ISCE_TEST',
 'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/212c21e5-61df-4f58-aabc-84f437e3ab76/212c21e5-61df-4f58-aabc-84f437e3ab76.log'],
 'name': 'Anchorage_131_HRRR_0524',
 'processing_times': [9044.308],
 'request_time': '2023-05-24T23:49:30+00:00',
 'status_code': 'FAILED',
 'user_id': 'cmarshak'}
{'expiration_time': '2023-11-22T00:00:00+00:00',
 'job_id': '781c2c86-a6df-4617-bcb1-73bfa73a49c8',
 'job_parameters': {'compute_solid_earth_tide': True,
                    'estimate_ionosphere_delay': True,
                    'frame_id': 20342,
                    'granules': ['S1A_IW_SLC__1SDV_20220222T162911_20220222T162938_042028_0501A2_0E39',
                                 'S1A_IW_SLC__1SDV_20220222T162936_20220222T163004_042028_0501A2_8FED'],
                    'secondary_granules': ['S1A_IW_SLC__1SDV_20220129T162911_20220129T162938_041678_04F56E_51C5',
                                           'S1A_IW_SLC__1SDV_20220129T162936_20220129T163004_041678_04F56E_60B3'],
                    'weather_model': 'HRRR'},
 'job_type': 'INSAR_ISCE_TEST',
 'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/781c2c86-a6df-4617-bcb1-73bfa73a49c8/781c2c86-a6df-4617-bcb1-73bfa73a49c8.log'],
 'name': 'Anchorage_131_HRRR_0524',
 'processing_times': [9114.568],
 'request_time': '2023-05-24T23:49:30+00:00',
 'status_code': 'FAILED',
 'user_id': 'cmarshak'}
{'expiration_time': '2023-11-22T00:00:00+00:00',
 'job_id': 'b81616b8-26b0-4b7a-9cd0-ee4484371768',
 'job_parameters': {'compute_solid_earth_tide': True,
                    'estimate_ionosphere_delay': True,
                    'frame_id': 20342,
                    'granules': ['S1A_IW_SLC__1SDV_20220222T162911_20220222T162938_042028_0501A2_0E39',
                                 'S1A_IW_SLC__1SDV_20220222T162936_20220222T163004_042028_0501A2_8FED'],
                    'secondary_granules': ['S1A_IW_SLC__1SDV_20220210T162911_20220210T162938_041853_04FB8A_974D',
                                           'S1A_IW_SLC__1SDV_20220210T162936_20220210T163004_041853_04FB8A_4E1F'],
                    'weather_model': 'HRRR'},
 'job_type': 'INSAR_ISCE_TEST',
 'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/b81616b8-26b0-4b7a-9cd0-ee4484371768/b81616b8-26b0-4b7a-9cd0-ee4484371768.log'],
 'name': 'Anchorage_131_HRRR_0524',
 'processing_times': [9218.873],
 'request_time': '2023-05-24T23:49:30+00:00',
 'status_code': 'FAILED',
 'user_id': 'cmarshak'}

cmarshak commented 1 year ago

The errors could be related to numpy, ISCE2 and the environment (maybe using the latest isce2 w/ python 3.10 would resolve this).

dbekaert commented 1 year ago

swapping to latest isce should only be done after verification there is no new issues introduced in the GUNW product.

mgovorcin commented 1 year ago

@cmarshak @dbekaert I run the failed job locally. It looks like the water mask is the issue, the downloaded file has nodata over the whole scene. SRTM Water Body Data does not have the coverage over that area.

Coverage: https://www2.jpl.nasa.gov/srtm/images/SRTM_2-24-2016.gif

cmarshak commented 1 year ago

I see; so, Anchorage is too far North at ~61˚ latitude to generate a valid SRTM water mask for the iono correction. I was concerned why the dem staging didn't fail; forgot that SRTM has a slightly smaller area of coverage than the Copernicus DEM we are using. @dbekaert could you provide guidance as to what you would like for such a case for operational use?

Should we:

Use a backup water mask areas outside SRTM (or just a different water mask altogether) such as ESA 10 m landcover
Prevent users from submitting jobs over areas beyond SRTM coverage and include descriptive errors when they do
Something else?

dbekaert commented 1 year ago

We should really know what the coverage is of all of our ancillaries and have this documented on our website. We should capture what ancillaries are required and for which the code should error out in case we go beyond.

For example:

DEM is required for corrections on the InSAR etc. We should error.
The water mask, if the ionosphere code requires it, we should error out; if it does not require it then the code should be able to continue meaning full or have an alternative fall-back.

We have established that GSSHHS is incorrect at higher latitudes and should be avoided. What are alternatives, and if we pick an alternative we should inspect and assess the quality of the watermask globally prior to its use. What is ASF team using as water mask @jhkennedy ?

dbekaert commented 1 year ago

Some guidance to make progress @cmarshak @mgovorcin :

For the time being you could have the enumerator not allow jobs to submit in case of missing ancillary. I do think we need to implement a better watermask so we can process at higher latitudes.
We should test the stitching and then decide if we split the iono into the short and long wavelength components within the product.

jhkennedy commented 1 year ago

Here's the blurb in our INSAR_GAMMA product guide describing the water masking approach for jobs:

If the water mask option is selected, the water mask is applied prior to phase unwrapping to exclude water pixels from the process. The water mask is generated using the GSHHG dataset. To compile the reference shapefile, the full-resolution L1 dataset (boundary between land and ocean) and L5 dataset (boundary between Antarctic ice and ocean) were combined. The L3 dataset (boundary between islands and the lakes they are within) was removed from the L2 dataset (boundary between lakes and land), and this combined dataset was removed from the combined L1/L5 dataset. The portion of the shapefile covering the scene is converted to a raster for inclusion in the phase unwrapping mask during InSAR processing. The GSHHG dataset was last updated in 2017, so there may be discrepancies where shorelines have changed. Visit our InSAR Water Masking Tutorial for more information about water masking.

cmarshak commented 1 year ago

Additional issues over Chile with the same error:

[{'job_id': '6c48161f-0256-46ed-9236-8fa4d891f052',
  'job_type': 'INSAR_ISCE_TEST',
  'request_time': '2023-05-31T16:47:08+00:00',
  'status_code': 'FAILED',
  'user_id': 'cmarshak',
  'name': 'Chile_54_GMAO_0531',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 8305,
   'granules': ['S1A_IW_SLC__1SDV_20220301T100856_20220301T100923_042126_0504E4_A6FD',
    'S1A_IW_SLC__1SDV_20220301T100921_20220301T100948_042126_0504E4_FD75'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20220205T100856_20220205T100923_041776_04F8C9_1FD2',
    'S1A_IW_SLC__1SDV_20220205T100921_20220205T100948_041776_04F8C9_510B'],
   'weather_model': 'GMAO'},
  'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/6c48161f-0256-46ed-9236-8fa4d891f052/6c48161f-0256-46ed-9236-8fa4d891f052.log'],
  'expiration_time': '2023-11-29T00:00:00+00:00',
  'processing_times': [8639.944]},
 {'job_id': '74a52951-61c3-4c59-96e2-515cd9b0946f',
  'job_type': 'INSAR_ISCE_TEST',
  'request_time': '2023-05-31T16:47:08+00:00',
  'status_code': 'FAILED',
  'user_id': 'cmarshak',
  'name': 'Chile_54_GMAO_0531',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 8305,
   'granules': ['S1A_IW_SLC__1SDV_20220217T100856_20220217T100923_041951_04FEE4_394A',
    'S1A_IW_SLC__1SDV_20220217T100921_20220217T100948_041951_04FEE4_B27F'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20220205T100856_20220205T100923_041776_04F8C9_1FD2',
    'S1A_IW_SLC__1SDV_20220205T100921_20220205T100948_041776_04F8C9_510B'],
   'weather_model': 'GMAO'},
  'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/74a52951-61c3-4c59-96e2-515cd9b0946f/74a52951-61c3-4c59-96e2-515cd9b0946f.log'],
  'expiration_time': '2023-11-29T00:00:00+00:00',
  'processing_times': [8872.909]},
 {'job_id': '961e107d-9f75-4cad-a3fb-1e01de52172a',
  'job_type': 'INSAR_ISCE_TEST',
  'request_time': '2023-05-31T16:47:08+00:00',
  'status_code': 'FAILED',
  'user_id': 'cmarshak',
  'name': 'Chile_54_GMAO_0531',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 8305,
   'granules': ['S1A_IW_SLC__1SDV_20220217T100856_20220217T100923_041951_04FEE4_394A',
    'S1A_IW_SLC__1SDV_20220217T100921_20220217T100948_041951_04FEE4_B27F'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20220124T100857_20220124T100924_041601_04F2CB_D770',
    'S1A_IW_SLC__1SDV_20220124T100922_20220124T100949_041601_04F2CB_F8FF'],
   'weather_model': 'GMAO'},
  'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/961e107d-9f75-4cad-a3fb-1e01de52172a/961e107d-9f75-4cad-a3fb-1e01de52172a.log'],
  'expiration_time': '2023-11-29T00:00:00+00:00',
  'processing_times': [8859.043]}]

This is no longer a water mask issue based on coverage. Below is where the frames are:

cmarshak commented 1 year ago

For comparison, here are the successful jobs (12/15 succeeded):

[{'job_id': '556d124f-b22a-4bbc-9d1e-26fb00a8dded',
  'job_type': 'INSAR_ISCE_TEST',
  'request_time': '2023-05-31T16:47:08+00:00',
  'status_code': 'SUCCEEDED',
  'user_id': 'cmarshak',
  'name': 'Chile_54_GMAO_0531',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 8303,
   'granules': ['S1A_IW_SLC__1SDV_20220217T100807_20220217T100833_041951_04FEE4_A907',
    'S1A_IW_SLC__1SDV_20220217T100831_20220217T100858_041951_04FEE4_7453'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20220205T100803_20220205T100833_041776_04F8C9_4B36',
    'S1A_IW_SLC__1SDV_20220205T100831_20220205T100858_041776_04F8C9_36B3'],
   'weather_model': 'GMAO'},
  'files': [{'filename': 'S1-GUNW-D-R-054-tops-20220217_20220205-100832-00071W_00019S-PP-8231-v3_0_0.nc',
    's3': {'bucket': 'hyp3-a19-jpl-contentbucket-1wfnatpznlg8b',
     'key': '556d124f-b22a-4bbc-9d1e-26fb00a8dded/S1-GUNW-D-R-054-tops-20220217_20220205-100832-00071W_00019S-PP-8231-v3_0_0.nc'},
    'size': 47715811,
    'url': 'https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/556d124f-b22a-4bbc-9d1e-26fb00a8dded/S1-GUNW-D-R-054-tops-20220217_20220205-100832-00071W_00019S-PP-8231-v3_0_0.nc'}],
  'logs': [],
  'browse_images': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/556d124f-b22a-4bbc-9d1e-26fb00a8dded/S1-GUNW-D-R-054-tops-20220217_20220205-100832-00071W_00019S-PP-8231-v3_0_0.png'],
  'thumbnail_images': [],
  'expiration_time': '2023-11-28T00:00:00+00:00',
  'processing_times': [9796.528, 96.06]},
 {'job_id': '03757038-d943-46e4-a993-ce5fe1a472e9',
  'job_type': 'INSAR_ISCE_TEST',
  'request_time': '2023-05-31T16:47:08+00:00',
  'status_code': 'SUCCEEDED',
  'user_id': 'cmarshak',
  'name': 'Chile_54_GMAO_0531',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 8304,
   'granules': ['S1A_IW_SLC__1SDV_20220301T100831_20220301T100858_042126_0504E4_4CF8',
    'S1A_IW_SLC__1SDV_20220301T100856_20220301T100923_042126_0504E4_A6FD'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20220205T100831_20220205T100858_041776_04F8C9_36B3',
    'S1A_IW_SLC__1SDV_20220205T100856_20220205T100923_041776_04F8C9_1FD2'],
   'weather_model': 'GMAO'},
  'files': [{'filename': 'S1-GUNW-D-R-054-tops-20220301_20220205-100857-00071W_00020S-PP-0337-v3_0_0.nc',
    's3': {'bucket': 'hyp3-a19-jpl-contentbucket-1wfnatpznlg8b',
     'key': '03757038-d943-46e4-a993-ce5fe1a472e9/S1-GUNW-D-R-054-tops-20220301_20220205-100857-00071W_00020S-PP-0337-v3_0_0.nc'},
    'size': 49569390,
    'url': 'https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/03757038-d943-46e4-a993-ce5fe1a472e9/S1-GUNW-D-R-054-tops-20220301_20220205-100857-00071W_00020S-PP-0337-v3_0_0.nc'}],
  'logs': [],
  'browse_images': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/03757038-d943-46e4-a993-ce5fe1a472e9/S1-GUNW-D-R-054-tops-20220301_20220205-100857-00071W_00020S-PP-0337-v3_0_0.png'],
  'thumbnail_images': [],
  'expiration_time': '2023-11-28T00:00:00+00:00',
  'processing_times': [9712.794, 92.987]},
 {'job_id': 'fc61bc39-df9a-4e52-854d-1273d97c8a85',
  'job_type': 'INSAR_ISCE_TEST',
  'request_time': '2023-05-31T16:47:08+00:00',
  'status_code': 'SUCCEEDED',
  'user_id': 'cmarshak',
  'name': 'Chile_54_GMAO_0531',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 8303,
   'granules': ['S1A_IW_SLC__1SDV_20220301T100806_20220301T100833_042126_0504E4_B4F7',
    'S1A_IW_SLC__1SDV_20220301T100831_20220301T100858_042126_0504E4_4CF8'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20220217T100807_20220217T100833_041951_04FEE4_A907',
    'S1A_IW_SLC__1SDV_20220217T100831_20220217T100858_041951_04FEE4_7453'],
   'weather_model': 'GMAO'},
  'files': [{'filename': 'S1-GUNW-D-R-054-tops-20220301_20220217-100832-00071W_00019S-PP-2348-v3_0_0.nc',
    's3': {'bucket': 'hyp3-a19-jpl-contentbucket-1wfnatpznlg8b',
     'key': 'fc61bc39-df9a-4e52-854d-1273d97c8a85/S1-GUNW-D-R-054-tops-20220301_20220217-100832-00071W_00019S-PP-2348-v3_0_0.nc'},
    'size': 48579161,
    'url': 'https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/fc61bc39-df9a-4e52-854d-1273d97c8a85/S1-GUNW-D-R-054-tops-20220301_20220217-100832-00071W_00019S-PP-2348-v3_0_0.nc'}],
  'logs': [],
  'browse_images': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/fc61bc39-df9a-4e52-854d-1273d97c8a85/S1-GUNW-D-R-054-tops-20220301_20220217-100832-00071W_00019S-PP-2348-v3_0_0.png'],
  'thumbnail_images': [],
  'expiration_time': '2023-11-28T00:00:00+00:00',
  'processing_times': [8759.172, 96.099]},
 {'job_id': '71ad9bd4-260b-41e0-b31d-cc46536e1abe',
  'job_type': 'INSAR_ISCE_TEST',
  'request_time': '2023-05-31T16:47:08+00:00',
  'status_code': 'SUCCEEDED',
  'user_id': 'cmarshak',
  'name': 'Chile_54_GMAO_0531',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 8303,
   'granules': ['S1A_IW_SLC__1SDV_20220217T100807_20220217T100833_041951_04FEE4_A907',
    'S1A_IW_SLC__1SDV_20220217T100831_20220217T100858_041951_04FEE4_7453'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20220124T100804_20220124T100833_041601_04F2CB_1727',
    'S1A_IW_SLC__1SDV_20220124T100831_20220124T100859_041601_04F2CB_2404'],
   'weather_model': 'GMAO'},
  'files': [{'filename': 'S1-GUNW-D-R-054-tops-20220217_20220124-100832-00071W_00019S-PP-fe6b-v3_0_0.nc',
    's3': {'bucket': 'hyp3-a19-jpl-contentbucket-1wfnatpznlg8b',
     'key': '71ad9bd4-260b-41e0-b31d-cc46536e1abe/S1-GUNW-D-R-054-tops-20220217_20220124-100832-00071W_00019S-PP-fe6b-v3_0_0.nc'},
    'size': 48988658,
    'url': 'https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/71ad9bd4-260b-41e0-b31d-cc46536e1abe/S1-GUNW-D-R-054-tops-20220217_20220124-100832-00071W_00019S-PP-fe6b-v3_0_0.nc'}],
  'logs': [],
  'browse_images': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/71ad9bd4-260b-41e0-b31d-cc46536e1abe/S1-GUNW-D-R-054-tops-20220217_20220124-100832-00071W_00019S-PP-fe6b-v3_0_0.png'],
  'thumbnail_images': [],
  'expiration_time': '2023-11-28T00:00:00+00:00',
  'processing_times': [9830.82, 84.456]},
 {'job_id': 'a61811c4-5293-48ac-aa4e-f6c2c8c5dd28',
  'job_type': 'INSAR_ISCE_TEST',
  'request_time': '2023-05-31T16:47:08+00:00',
  'status_code': 'SUCCEEDED',
  'user_id': 'cmarshak',
  'name': 'Chile_54_GMAO_0531',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 8305,
   'granules': ['S1A_IW_SLC__1SDV_20220205T100856_20220205T100923_041776_04F8C9_1FD2',
    'S1A_IW_SLC__1SDV_20220205T100921_20220205T100948_041776_04F8C9_510B'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20220124T100857_20220124T100924_041601_04F2CB_D770',
    'S1A_IW_SLC__1SDV_20220124T100922_20220124T100949_041601_04F2CB_F8FF'],
   'weather_model': 'GMAO'},
  'files': [{'filename': 'S1-GUNW-D-R-054-tops-20220205_20220124-100922-00071W_00022S-PP-fb68-v3_0_0.nc',
    's3': {'bucket': 'hyp3-a19-jpl-contentbucket-1wfnatpznlg8b',
     'key': 'a61811c4-5293-48ac-aa4e-f6c2c8c5dd28/S1-GUNW-D-R-054-tops-20220205_20220124-100922-00071W_00022S-PP-fb68-v3_0_0.nc'},
    'size': 50281286,
    'url': 'https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/a61811c4-5293-48ac-aa4e-f6c2c8c5dd28/S1-GUNW-D-R-054-tops-20220205_20220124-100922-00071W_00022S-PP-fb68-v3_0_0.nc'}],
  'logs': [],
  'browse_images': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/a61811c4-5293-48ac-aa4e-f6c2c8c5dd28/S1-GUNW-D-R-054-tops-20220205_20220124-100922-00071W_00022S-PP-fb68-v3_0_0.png'],
  'thumbnail_images': [],
  'expiration_time': '2023-11-28T00:00:00+00:00',
  'processing_times': [8365.355, 111.25]},
 {'job_id': 'f1ec2f79-09f8-4497-a9ba-94998f851cd4',
  'job_type': 'INSAR_ISCE_TEST',
  'request_time': '2023-05-31T16:47:08+00:00',
  'status_code': 'SUCCEEDED',
  'user_id': 'cmarshak',
  'name': 'Chile_54_GMAO_0531',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 8303,
   'granules': ['S1A_IW_SLC__1SDV_20220301T100806_20220301T100833_042126_0504E4_B4F7',
    'S1A_IW_SLC__1SDV_20220301T100831_20220301T100858_042126_0504E4_4CF8'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20220205T100803_20220205T100833_041776_04F8C9_4B36',
    'S1A_IW_SLC__1SDV_20220205T100831_20220205T100858_041776_04F8C9_36B3'],
   'weather_model': 'GMAO'},
  'files': [{'filename': 'S1-GUNW-D-R-054-tops-20220301_20220205-100832-00071W_00019S-PP-88d4-v3_0_0.nc',
    's3': {'bucket': 'hyp3-a19-jpl-contentbucket-1wfnatpznlg8b',
     'key': 'f1ec2f79-09f8-4497-a9ba-94998f851cd4/S1-GUNW-D-R-054-tops-20220301_20220205-100832-00071W_00019S-PP-88d4-v3_0_0.nc'},
    'size': 48286588,
    'url': 'https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/f1ec2f79-09f8-4497-a9ba-94998f851cd4/S1-GUNW-D-R-054-tops-20220301_20220205-100832-00071W_00019S-PP-88d4-v3_0_0.nc'}],
  'logs': [],
  'browse_images': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/f1ec2f79-09f8-4497-a9ba-94998f851cd4/S1-GUNW-D-R-054-tops-20220301_20220205-100832-00071W_00019S-PP-88d4-v3_0_0.png'],
  'thumbnail_images': [],
  'expiration_time': '2023-11-28T00:00:00+00:00',
  'processing_times': [9898.379, 94.651]},
 {'job_id': 'd7b7b257-78ce-42f5-90a9-ebdfd0116987',
  'job_type': 'INSAR_ISCE_TEST',
  'request_time': '2023-05-31T16:47:08+00:00',
  'status_code': 'SUCCEEDED',
  'user_id': 'cmarshak',
  'name': 'Chile_54_GMAO_0531',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 8304,
   'granules': ['S1A_IW_SLC__1SDV_20220301T100831_20220301T100858_042126_0504E4_4CF8',
    'S1A_IW_SLC__1SDV_20220301T100856_20220301T100923_042126_0504E4_A6FD'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20220217T100831_20220217T100858_041951_04FEE4_7453',
    'S1A_IW_SLC__1SDV_20220217T100856_20220217T100923_041951_04FEE4_394A'],
   'weather_model': 'GMAO'},
  'files': [{'filename': 'S1-GUNW-D-R-054-tops-20220301_20220217-100857-00071W_00020S-PP-70d0-v3_0_0.nc',
    's3': {'bucket': 'hyp3-a19-jpl-contentbucket-1wfnatpznlg8b',
     'key': 'd7b7b257-78ce-42f5-90a9-ebdfd0116987/S1-GUNW-D-R-054-tops-20220301_20220217-100857-00071W_00020S-PP-70d0-v3_0_0.nc'},
    'size': 49272700,
    'url': 'https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/d7b7b257-78ce-42f5-90a9-ebdfd0116987/S1-GUNW-D-R-054-tops-20220301_20220217-100857-00071W_00020S-PP-70d0-v3_0_0.nc'}],
  'logs': [],
  'browse_images': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/d7b7b257-78ce-42f5-90a9-ebdfd0116987/S1-GUNW-D-R-054-tops-20220301_20220217-100857-00071W_00020S-PP-70d0-v3_0_0.png'],
  'thumbnail_images': [],
  'expiration_time': '2023-11-28T00:00:00+00:00',
  'processing_times': [8571.228, 94.219]},
 {'job_id': '1e35d0bd-4416-4db0-b638-320afa42af26',
  'job_type': 'INSAR_ISCE_TEST',
  'request_time': '2023-05-31T16:47:08+00:00',
  'status_code': 'SUCCEEDED',
  'user_id': 'cmarshak',
  'name': 'Chile_54_GMAO_0531',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 8305,
   'granules': ['S1A_IW_SLC__1SDV_20220301T100856_20220301T100923_042126_0504E4_A6FD',
    'S1A_IW_SLC__1SDV_20220301T100921_20220301T100948_042126_0504E4_FD75'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20220217T100856_20220217T100923_041951_04FEE4_394A',
    'S1A_IW_SLC__1SDV_20220217T100921_20220217T100948_041951_04FEE4_B27F'],
   'weather_model': 'GMAO'},
  'files': [{'filename': 'S1-GUNW-D-R-054-tops-20220301_20220217-100922-00071W_00022S-PP-9df1-v3_0_0.nc',
    's3': {'bucket': 'hyp3-a19-jpl-contentbucket-1wfnatpznlg8b',
     'key': '1e35d0bd-4416-4db0-b638-320afa42af26/S1-GUNW-D-R-054-tops-20220301_20220217-100922-00071W_00022S-PP-9df1-v3_0_0.nc'},
    'size': 49062814,
    'url': 'https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/1e35d0bd-4416-4db0-b638-320afa42af26/S1-GUNW-D-R-054-tops-20220301_20220217-100922-00071W_00022S-PP-9df1-v3_0_0.nc'}],
  'logs': [],
  'browse_images': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/1e35d0bd-4416-4db0-b638-320afa42af26/S1-GUNW-D-R-054-tops-20220301_20220217-100922-00071W_00022S-PP-9df1-v3_0_0.png'],
  'thumbnail_images': [],
  'expiration_time': '2023-11-28T00:00:00+00:00',
  'processing_times': [8282.313, 121.845]},
 {'job_id': 'f34d1b58-f0d9-4628-aeed-5b34e4103a89',
  'job_type': 'INSAR_ISCE_TEST',
  'request_time': '2023-05-31T16:47:08+00:00',
  'status_code': 'SUCCEEDED',
  'user_id': 'cmarshak',
  'name': 'Chile_54_GMAO_0531',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 8304,
   'granules': ['S1A_IW_SLC__1SDV_20220217T100831_20220217T100858_041951_04FEE4_7453',
    'S1A_IW_SLC__1SDV_20220217T100856_20220217T100923_041951_04FEE4_394A'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20220205T100831_20220205T100858_041776_04F8C9_36B3',
    'S1A_IW_SLC__1SDV_20220205T100856_20220205T100923_041776_04F8C9_1FD2'],
   'weather_model': 'GMAO'},
  'files': [{'filename': 'S1-GUNW-D-R-054-tops-20220217_20220205-100857-00071W_00020S-PP-1819-v3_0_0.nc',
    's3': {'bucket': 'hyp3-a19-jpl-contentbucket-1wfnatpznlg8b',
     'key': 'f34d1b58-f0d9-4628-aeed-5b34e4103a89/S1-GUNW-D-R-054-tops-20220217_20220205-100857-00071W_00020S-PP-1819-v3_0_0.nc'},
    'size': 49756522,
    'url': 'https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/f34d1b58-f0d9-4628-aeed-5b34e4103a89/S1-GUNW-D-R-054-tops-20220217_20220205-100857-00071W_00020S-PP-1819-v3_0_0.nc'}],
  'logs': [],
  'browse_images': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/f34d1b58-f0d9-4628-aeed-5b34e4103a89/S1-GUNW-D-R-054-tops-20220217_20220205-100857-00071W_00020S-PP-1819-v3_0_0.png'],
  'thumbnail_images': [],
  'expiration_time': '2023-11-28T00:00:00+00:00',
  'processing_times': [9605.973, 89.116]},
 {'job_id': '8f8877cf-7138-41c9-9380-6da340a47a8b',
  'job_type': 'INSAR_ISCE_TEST',
  'request_time': '2023-05-31T16:47:08+00:00',
  'status_code': 'SUCCEEDED',
  'user_id': 'cmarshak',
  'name': 'Chile_54_GMAO_0531',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 8304,
   'granules': ['S1A_IW_SLC__1SDV_20220217T100831_20220217T100858_041951_04FEE4_7453',
    'S1A_IW_SLC__1SDV_20220217T100856_20220217T100923_041951_04FEE4_394A'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20220124T100831_20220124T100859_041601_04F2CB_2404',
    'S1A_IW_SLC__1SDV_20220124T100857_20220124T100924_041601_04F2CB_D770'],
   'weather_model': 'GMAO'},
  'files': [{'filename': 'S1-GUNW-D-R-054-tops-20220217_20220124-100857-00071W_00020S-PP-c232-v3_0_0.nc',
    's3': {'bucket': 'hyp3-a19-jpl-contentbucket-1wfnatpznlg8b',
     'key': '8f8877cf-7138-41c9-9380-6da340a47a8b/S1-GUNW-D-R-054-tops-20220217_20220124-100857-00071W_00020S-PP-c232-v3_0_0.nc'},
    'size': 49650227,
    'url': 'https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/8f8877cf-7138-41c9-9380-6da340a47a8b/S1-GUNW-D-R-054-tops-20220217_20220124-100857-00071W_00020S-PP-c232-v3_0_0.nc'}],
  'logs': [],
  'browse_images': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/8f8877cf-7138-41c9-9380-6da340a47a8b/S1-GUNW-D-R-054-tops-20220217_20220124-100857-00071W_00020S-PP-c232-v3_0_0.png'],
  'thumbnail_images': [],
  'expiration_time': '2023-11-28T00:00:00+00:00',
  'processing_times': [9389.118, 90.276]},
 {'job_id': '77501421-4ee4-4ce0-b56d-bda5398a55b9',
  'job_type': 'INSAR_ISCE_TEST',
  'request_time': '2023-05-31T16:47:08+00:00',
  'status_code': 'SUCCEEDED',
  'user_id': 'cmarshak',
  'name': 'Chile_54_GMAO_0531',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 8303,
   'granules': ['S1A_IW_SLC__1SDV_20220205T100803_20220205T100833_041776_04F8C9_4B36',
    'S1A_IW_SLC__1SDV_20220205T100831_20220205T100858_041776_04F8C9_36B3'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20220124T100804_20220124T100833_041601_04F2CB_1727',
    'S1A_IW_SLC__1SDV_20220124T100831_20220124T100859_041601_04F2CB_2404'],
   'weather_model': 'GMAO'},
  'files': [{'filename': 'S1-GUNW-D-R-054-tops-20220205_20220124-100830-00071W_00019S-PP-ba59-v3_0_0.nc',
    's3': {'bucket': 'hyp3-a19-jpl-contentbucket-1wfnatpznlg8b',
     'key': '77501421-4ee4-4ce0-b56d-bda5398a55b9/S1-GUNW-D-R-054-tops-20220205_20220124-100830-00071W_00019S-PP-ba59-v3_0_0.nc'},
    'size': 47397434,
    'url': 'https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/77501421-4ee4-4ce0-b56d-bda5398a55b9/S1-GUNW-D-R-054-tops-20220205_20220124-100830-00071W_00019S-PP-ba59-v3_0_0.nc'}],
  'logs': [],
  'browse_images': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/77501421-4ee4-4ce0-b56d-bda5398a55b9/S1-GUNW-D-R-054-tops-20220205_20220124-100830-00071W_00019S-PP-ba59-v3_0_0.png'],
  'thumbnail_images': [],
  'expiration_time': '2023-11-28T00:00:00+00:00',
  'processing_times': [8782.911, 95.489]},
 {'job_id': '09498d55-ee22-42a2-92dc-c6103f55e100',
  'job_type': 'INSAR_ISCE_TEST',
  'request_time': '2023-05-31T16:47:08+00:00',
  'status_code': 'SUCCEEDED',
  'user_id': 'cmarshak',
  'name': 'Chile_54_GMAO_0531',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 8304,
   'granules': ['S1A_IW_SLC__1SDV_20220205T100831_20220205T100858_041776_04F8C9_36B3',
    'S1A_IW_SLC__1SDV_20220205T100856_20220205T100923_041776_04F8C9_1FD2'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20220124T100831_20220124T100859_041601_04F2CB_2404',
    'S1A_IW_SLC__1SDV_20220124T100857_20220124T100924_041601_04F2CB_D770'],
   'weather_model': 'GMAO'},
  'files': [{'filename': 'S1-GUNW-D-R-054-tops-20220205_20220124-100857-00071W_00020S-PP-d9fb-v3_0_0.nc',
    's3': {'bucket': 'hyp3-a19-jpl-contentbucket-1wfnatpznlg8b',
     'key': '09498d55-ee22-42a2-92dc-c6103f55e100/S1-GUNW-D-R-054-tops-20220205_20220124-100857-00071W_00020S-PP-d9fb-v3_0_0.nc'},
    'size': 50000696,
    'url': 'https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/09498d55-ee22-42a2-92dc-c6103f55e100/S1-GUNW-D-R-054-tops-20220205_20220124-100857-00071W_00020S-PP-d9fb-v3_0_0.nc'}],
  'logs': [],
  'browse_images': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/09498d55-ee22-42a2-92dc-c6103f55e100/S1-GUNW-D-R-054-tops-20220205_20220124-100857-00071W_00020S-PP-d9fb-v3_0_0.png'],
  'thumbnail_images': [],
  'expiration_time': '2023-11-28T00:00:00+00:00',
  'processing_times': [8579.592, 96.394]}]

mgovorcin commented 1 year ago

@cmarshak it appears that issue comes from ionosphere computation swath by swath, where one swath is completely over the water and masked out. The iono fails over that swath IW3. Will try to see how to bypass this. The other issue might occur with merging the swaths afterwards

ionSwathBySwath mode is activated due to different Starting Range per Swath between reference and secondary scenes

TODO: need to modify compute_ionosphere step to work with nodata arrays. - SOLVED

mgovorcin commented 1 year ago

@cmarshak @dbekaert ... I am thinking about quality layer for ionosphere, it could be the boolean array of the pixels used in the computation. This could give us an idea of how many pixels and their distribution are used in iono computation (aka constrains to plane fitting involved in filt_ion step).

Before proceeding, would like to hear your opinion on this. Would you like to have something like this in GUNW package next to iono layer?

dbekaert commented 1 year ago

Interesting to see what you think, how it would be used downstream in ts. Is the coherence of the ifg not a good marker too?

A binary layer would be small.

ACCESS-Cloud-Based-InSAR / DockerizedTopsApp

New Ionosphere Updates do not always complete in Anchorage (Track 131) #135