search

ASFHyP3 / hyp3-isce2

HyP3 plugin for ISCE2 processing
Apache License 2.0
10 stars 8 forks source link

hyp3-isce2 can not process the pair which crosses over multiple partition of a GSHHG parquet #158

Open cirrusasf opened 7 months ago

cirrusasf commented 7 months ago

The bug

current hyp3-isce2 create water_mask based on GSHHG parquet file. GSHHG file is from asf-dem-west/WATER_MASK/GSHHG/hyp3_water_mask_20220912. In order to speed up the reading file, parquet store the GSHHG into 8 partitions. When a pair crosses over multiple partition, the hyp3_isce2 can not product the correct water_mask, the program stops with error message. As an example, we pick the spain pair: S1_062521_IW1_20230622T175457_VV_91B3-BURST S1_062521_IW1_20230704T175458_VVC6F8-BURST which crosses over two partitions ( 000-0090 and 000_0000). hyp3_isce2 gave the following errors when it process this pair: Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/home/jiangzhu/projects/work/hyp3/hyp3-isce2/src/hyp3_isce2/main.py", line 51, in main() File "/home/jiangzhu/projects/work/hyp3/hyp3-isce2/src/hyp3_isce2/main.py", line 47, in main sys.exit(process_entry_point.load()()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jiangzhu/projects/work/hyp3/hyp3-isce2/src/hyp3_isce2/insar_tops_burst.py", line 439, in main isce_output_dir = insar_tops_burst( ^^^^^^^^^^^^^^^^^ File "/home/jiangzhu/projects/work/hyp3/hyp3-isce2/src/hyp3_isce2/insar_tops_burst.py", line 136, in insar_tops_burst resample_to_radar_io(water_mask_path, 'merged/lat.rdr', 'merged/lon.rdr', 'merged/water_mask.rdr') File "/home/jiangzhu/projects/work/hyp3/hyp3-isce2/src/hyp3_isce2/utils.py", line 189, in resample_to_radar_io mask = np.reshape(mask, [maskim.coord2.coordSize, maskim.coord1.coordSize]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jiangzhu/apps/mambaforge/envs/hyp3-isce2/lib/python3.11/site-packages/numpy/core/fromnumeric.py", line 285, in reshape return _wrapfunc(a, 'reshape', newshape, order=order) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jiangzhu/apps/mambaforge/envs/hyp3-isce2/lib/python3.11/site-packages/numpy/core/fromnumeric.py", line 59, in _wrapfunc return bound(*args, **kwds) ^^^^^^^^^^^^^^^^^^^^ ValueError: cannot reshape array of size 2 into shape (3600,7200)

The problem is the code can not produce the correct water_mask.

forrestfwilliams commented 7 months ago

The command to recreate this issue is: insar_tops_burst S1_062521_IW1_20230622T175457_VV_91B3-BURST S1_062521_IW1_20230704T175458_VV_C6F8-BURST --looks 20x4 --apply-water-mask True I can confirm the issue.

forrestfwilliams commented 7 months ago

@cirrusasf can you also find an antemeridian example?

forrestfwilliams commented 7 months ago

I think we try to find a way to resolve this issue that still allows us to use the parquet file. When running the shapefile version of the burst workflow on a local machine, creating the watermask is very slow and can take up to 30-40% of the total processing time.

While we typically run these workflow in AWS where downloads are faster, I think we need to consider the local use case here because this is the only workflow in the community for creating single-burst InSAR and people will likely want to use it with their own modifications.