search

opera-adt / DSWX-SAR

Dynamic Surface Water Extent from Synthetic Aperture Radar
Apache License 2.0
9 stars 6 forks source link

`gdal` version needs capping + hanging on `scratch/masking_ancillary_darkland_removed.tif` #61

Closed cmarshak closed 5 months ago

cmarshak commented 6 months ago

In the latest version the preprocessing routines, I was getting:

...
reference_water has invalid values.
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[43], line 1
----> 1 dswx_s1_workflow(cfg)

File ~/bekaert-team/DSWX-SAR/src/dswx_sar/dswx_s1.py:51, in dswx_s1_workflow(cfg)
     49 processing_cfg.polarizations = pol_set
     50 # preprocessing (relocating ancillary data and filtering)
---> 51 pre_processing.run(cfg)
     53 # Estimate threshold for given polarizations
     54 initial_threshold.run(cfg)

File ~/bekaert-team/DSWX-SAR/src/dswx_sar/pre_processing.py:706, in run(cfg)
    703             logger.warning(f'{anc_type} has invalid values.')
    704             # update reference water if any values of reference water
    705             # are invalid.
--> 706             replace_reference_water_nodata_from_ancillary(
    707                 os.path.join(scratch_dir,
    708                              relocated_ancillary_filename_set['reference_water']),
    709                 os.path.join(scratch_dir,
    710                              relocated_ancillary_filename_set['landcover']),
    711                 os.path.join(scratch_dir,
    712                              relocated_ancillary_filename_set['hand']),
    713                 reference_water_max=ref_water_max,
    714                 reference_water_no_data=ref_water_no_data,
    715                 line_per_block=line_per_block
    716             )
    718 # apply SAR filtering
    719 pad_shape = (filter_size, 0)

File ~/bekaert-team/DSWX-SAR/src/dswx_sar/pre_processing.py:537, in replace_reference_water_nodata_from_ancillary(reference_water_path, landcover_path, hand_path, reference_water_max, reference_water_no_data, line_per_block)
    534 landcover_label = get_label_landcover_esa_10()
    536 for block_param in block_params:
--> 537     ref_water_block, landcover_block, hand_block = [
    538         dswx_sar_util.get_raster_block(path, block_param)
    539         for path in [temp_water_path,
    540                      landcover_path, 
    541                      hand_path]]
    543     # Both invalid and no-water areas have zero value
    544     # in reference water. The area with zero values are
    545     # replaced with maximum values where the permanent water
    546     # area in landcover.
    548     replaced_area = np.logical_and(
    549         ref_water_block == reference_water_no_data,
    550         landcover_block == landcover_label['Permanent water bodies'])

File ~/bekaert-team/DSWX-SAR/src/dswx_sar/pre_processing.py:538, in <listcomp>(.0)
    534 landcover_label = get_label_landcover_esa_10()
    536 for block_param in block_params:
    537     ref_water_block, landcover_block, hand_block = [
--> 538         dswx_sar_util.get_raster_block(path, block_param)
    539         for path in [temp_water_path,
    540                      landcover_path, 
    541                      hand_path]]
    543     # Both invalid and no-water areas have zero value
    544     # in reference water. The area with zero values are
    545     # replaced with maximum values where the permanent water
    546     # area in landcover.
    548     replaced_area = np.logical_and(
    549         ref_water_block == reference_water_no_data,
    550         landcover_block == landcover_label['Permanent water bodies'])

File ~/bekaert-team/DSWX-SAR/src/dswx_sar/dswx_sar_util.py:608, in get_raster_block(raster_path, block_param)
    605 ds_data = gdal.Open(raster_path, gdal.GA_Update)
    607 # Number of bands in the raster
--> 608 num_bands = ds_data.RasterCount
    609 # List to store blocks from each band
    610 data_blocks = []

AttributeError: 'NoneType' object has no attribute 'RasterCount'

At first, I thought it had to do with auxiliary data not having enough buffer since it said that the nodata was being carried over into the reprojected reference water map. However, buffering by 1 degree yielded a similar issue.

Specifically, the function validate_gtiff here makes sure there are no nodata values as indicated here: https://github.com/opera-adt/DSWX-SAR/blob/5f4c3209e1ba4e974f56f55485dce7e1c8fa017e/src/dswx_sar/pre_processing.py#L688-L691. The particular reprojected Pekel occurence map (this reprojection was done running the dswx-s1 workflow) looked like this: image.

First, I removed the no_data in the validation here: https://github.com/opera-adt/DSWX-SAR/blob/5f4c3209e1ba4e974f56f55485dce7e1c8fa017e/src/dswx_sar/pre_processing.py#L688-L691

Still same error.

So I went into dswx_utils and modifying this line:

ds_data = gdal.Open(raster_path, gdal.GA_Update)

to

ds_data = gdal.Open(raster_path)

Allowed the code to continue. Not sure if it's correct.

This seems like it could be gdal issue. I am using version 3.8.3 for the above as indicated with conda list | grep gdal:

gdal                      3.8.3           py311h4a095a9_0    conda-forge
libgdal                   3.8.3                h7e86f1f_0    conda-forge

My issues could be related to the minor release in November and making the runtime environment inconsistent with that which is being developed here. I have not checked this as I had a hard time pinpointing the bug. I am saying this because I had similar issues with DSWx-HLS recently as documented here. It turned out the expected runtime environment required gdal<3.8 and I was similarly using the latest gdal.

Please let me know:

cmarshak commented 5 months ago

I can confirm capping the gdal version with 'gdal<3.8' removes the above problem(s).

However, even so, I am not able to generate products as there is a step in the workflow that does not complete.

The process hangs here:

...
S1A_IW_SLC__1SDV_20240108T232450_20240108T232517_052022_06495E_2EAD/scratch/input_image_test.tif 
100 - done.
0...10...20...30...40...50...60...70...80...90... -- Reading /u/aurora-r0/cmarshak/dswx-s1-workflow-pst/out/S1A_IW_SLC__1SDV_20240108T232450_20240108T232517_052022_06495E_2EAD/scratch/input_image_test.tif ... (8347, 9766)
100 - done.
0...10...20...30...40...50...60...70...80...90...Saving dswx product : /u/aurora-r0/cmarshak/dswx-s1-workflow-pst/out/S1A_IW_SLC__1SDV_20240108T232450_20240108T232517_052022_06495E_2EAD/scratch/split_mask_water_masking.tif 
100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90... -- Reading /u/aurora-r0/cmarshak/dswx-s1-workflow-pst/out/S1A_IW_SLC__1SDV_20240108T232450_20240108T232517_052022_06495E_2EAD/scratch/intermediate_false_water_candidate.tif ... (8347, 9766)
100 - done.
0...10...20...30...40...50...60...70...80...90...Saving dswx product : /u/aurora-r0/cmarshak/dswx-s1-workflow-pst/out/S1A_IW_SLC__1SDV_20240108T232450_20240108T232517_052022_06495E_2EAD/scratch/false_positive_connected_water.tif 
100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90... -- Reading /u/aurora-r0/cmarshak/dswx-s1-workflow-pst/out/S1A_IW_SLC__1SDV_20240108T232450_20240108T232517_052022_06495E_2EAD/scratch/masking_ancillary_darkland_removed.tif ... (8347, 9766)

I have tried this on my local workstation and on the server aurora using the capped gdal version. I am not sure what the issue is or if there are additional environment constraints that might be needed.

Thank you.

cmarshak commented 5 months ago

Dear @oberonia78,

Thank you for your help thus far with getting DSWx-S1 running; per your excellent observation (my output hand.tif had a lot of nodata when using pysheds). I modified my workflow to pull from ASF tiles as you do. However, I am still having trouble running the workflow. These are the different flavors of the workflow that I have tried:

  1. Running the workflow on my local mac workstation using a hand raster that I merge myself from ASF tiles (see [1])
  2. Running the workflow on the shared server using a hand raster that I merge myself from ASF tiles (see [1])
  3. Running the workflow on the shared server using the VRT you provided. (modified [1] to point to the correct VRT)

For 1. and 2., I am still getting the issue I pointed out above, namely, hanging on 

0...10...20...30...40...50...60...70...80...90... -- Reading /u/aurora-r0/cmarshak/dswx-s1-workflow-pst/out/S1A_IW_SLC__1SDV_20240108T232450_20240108T232517_052022_06495E_2EAD/scratch/masking_ancillary_darkland_removed.tif ... (8347, 9766)

Weirdly, when I use the VRT in the shared server you provided as in 3., it gets past this "hanging" portion. However, it still fails I will share the output below past that "hanging area":

...
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90... -- Reading /u/aurora-r0/cmarshak/dswx-s1-workflow-pst/out2/S1A_IW_SLC__1SDV_20240108T232450_20240108T232517_052022_06495E_2EAD/scratch/masking_ancillary_darkland_removed.tif ... (8347, 9766)
Saving dswx product : /u/aurora-r0/cmarshak/dswx-s1-workflow-pst/out2/S1A_IW_SLC__1SDV_20240108T232450_20240108T232517_052022_06495E_2EAD/scratch/refine_landcover_binary_VV_VH.tif 
100 - done.
0...10...20...30...40...50...60...70...80...90... -- Reading /u/aurora-r0/cmarshak/dswx-s1-workflow-pst/out2/S1A_IW_SLC__1SDV_20240108T232450_20240108T232517_052022_06495E_2EAD/scratch/refine_landcover_binary_VV_VH.tif ... (8347, 9766)
 -- Reading /u/aurora-r0/cmarshak/dswx-s1-workflow-pst/out2/S1A_IW_SLC__1SDV_20240108T232450_20240108T232517_052022_06495E_2EAD/scratch/interpolated_landcover.tif ... (8347, 9766)
100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90... -- Reading /u/aurora-r0/cmarshak/dswx-s1-workflow-pst/out2/S1A_IW_SLC__1SDV_20240108T232450_20240108T232517_052022_06495E_2EAD/scratch/merged_removed_false_water_VV.tif ... (8347, 9766)
Saving dswx product : /u/aurora-r0/cmarshak/dswx-s1-workflow-pst/out2/S1A_IW_SLC__1SDV_20240108T232450_20240108T232517_052022_06495E_2EAD/scratch/bimodality_bright_water_VV_VH.tif 
100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90... -- Reading /u/aurora-r0/cmarshak/dswx-s1-workflow-pst/out2/S1A_IW_SLC__1SDV_20240108T232450_20240108T232517_052022_06495E_2EAD/scratch/merged_fill_gap_VV_VH.tif ... (8347, 9766)
Saving dswx product : /u/aurora-r0/cmarshak/dswx-s1-workflow-pst/out2/S1A_IW_SLC__1SDV_20240108T232450_20240108T232517_052022_06495E_2EAD/scratch/bimodality_output_binary_VV_VH.tif 
100 - done.
0...10...20...30...40...50...60...70...80...90...
/u/aurora-r0/cmarshak/miniforge3/envs/dswx_s1/lib/python3.11/site-packages/dswx_sar/pre_processing.py:32: RuntimeWarning: invalid value encountered in divide
  return array1 / array2
>> lee_enhanced_filter
/u/aurora-r0/cmarshak/miniforge3/envs/dswx_s1/lib/python3.11/site-packages/dswx_sar/detect_inundated_vegetation.py:100: RuntimeWarning: divide by zero encountered in log10
  cross_db = 10 * np.log10(
>> lee_enhanced_filter
>> lee_enhanced_filter
>> lee_enhanced_filter
>> lee_enhanced_filter
>> lee_enhanced_filter
>> lee_enhanced_filter
>> lee_enhanced_filter
>> lee_enhanced_filter
>> lee_enhanced_filter
>> lee_enhanced_filter
>> lee_enhanced_filter
>> lee_enhanced_filter
>> lee_enhanced_filter
>> lee_enhanced_filter
>> lee_enhanced_filter
>> lee_enhanced_filter
>> lee_enhanced_filter
>> lee_enhanced_filter
>> lee_enhanced_filter
>> lee_enhanced_filter
>> lee_enhanced_filter
>> lee_enhanced_filter
>> lee_enhanced_filter
>> lee_enhanced_filter
>> lee_enhanced_filter
>> lee_enhanced_filter
>> lee_enhanced_filter
100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...
WARNING: product version was not provided.
WARNING: Either the MGRS tile or the MGRS collection database was not provided.
 -- Reading /u/aurora-r0/cmarshak/dswx-s1-workflow-pst/out2/S1A_IW_SLC__1SDV_20240108T232450_20240108T232517_052022_06495E_2EAD/scratch/bimodality_output_binary_VV_VH.tif ... (8347, 9766)
 -- Reading /u/aurora-r0/cmarshak/dswx-s1-workflow-pst/out2/S1A_IW_SLC__1SDV_20240108T232450_20240108T232517_052022_06495E_2EAD/scratch/no_data_area_VV_VH.tif ... (8347, 9766)
 -- Reading /u/aurora-r0/cmarshak/dswx-s1-workflow-pst/out2/S1A_IW_SLC__1SDV_20240108T232450_20240108T232517_052022_06495E_2EAD/scratch/mosaic_layovershadow_mask.tif ... (8347, 9766)
 -- Reading /u/aurora-r0/cmarshak/dswx-s1-workflow-pst/out2/S1A_IW_SLC__1SDV_20240108T232450_20240108T232517_052022_06495E_2EAD/scratch/interpolated_hand.tif ... (8347, 9766)
 -- Reading /u/aurora-r0/cmarshak/dswx-s1-workflow-pst/out2/S1A_IW_SLC__1SDV_20240108T232450_20240108T232517_052022_06495E_2EAD/scratch/temp_inundated_vegetation_VV_VH.tif ... (8347, 9766)
 -- Reading /u/aurora-r0/cmarshak/dswx-s1-workflow-pst/out2/S1A_IW_SLC__1SDV_20240108T232450_20240108T232517_052022_06495E_2EAD/scratch/region_growing_output_binary_VV_VH.tif ... (8347, 9766)
 -- Reading /u/aurora-r0/cmarshak/dswx-s1-workflow-pst/out2/S1A_IW_SLC__1SDV_20240108T232450_20240108T232517_052022_06495E_2EAD/scratch/refine_landcover_binary_VV_VH.tif ... (8347, 9766)
100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90... -- Reading /u/aurora-r0/cmarshak/dswx-s1-workflow-pst/out2/S1A_IW_SLC__1SDV_20240108T232450_20240108T232517_052022_06495E_2EAD/scratch/fuzzy_image_VV_VH.tif ... (8347, 9766)
100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...
fail to find the overlapped rtc
100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...
fail to find the overlapped rtc
100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...
fail to find the overlapped rtc
100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...
fail to find the overlapped rtc
100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...

I compared the hand vrt you provided and the one I generated. Mine is slightly smaller (still buffered by 1 degree across the slc extents). When I reproject into the VRT you provided using nearest neighbor interpolation, I get the following difference:

image

If I see where precisely 0s occur I get this difference where the value of 1 indicates 0 (yellow) and blue the opposite.

image

Questions:

  1. Were you able to reinstall the environment to get the DSWx-S1 product to be generated?
  2. Are there ways of running DSWx-S1 to provide warnings and other verbose debugging output?
  3. You said that you saw an error with my previous run related to HAND, can you indicate what error that was and how that was discovered?

[1] https://github.com/OPERA-Cal-Val/dswx-s1-workflow-pst/blob/dev/1_DSWx-SAR-Workflow-Using-SLC-ID.ipynb

oberonia78 commented 5 months ago

Hi @cmarshak, Thanks for posting the issues you have. I'll investigate your issues. But before that, I suggest you to process with the database together. I'll share the files with LFT. The problem seems to happen in the last step of making OPERA packages.

  1. I'm still using the environment I shared. I agree that GDAL3.8 is not compatible with DSWx-S1 as you pointed out. Similarly, I heard that DSWx-HLS and OPERA RTC-S1 are struggling the similar issues.
  2. There is an option to generate more intermediate files. dswx_s1.py <your runconfig> --debug_mode
  3. When I tried to reproduce your issues, I got stuck here : 0...10...20...30...40...50...60...70...80...90... -- Reading /u/aurora-r0/cmarshak/dswx-s1-workflow-pst/out/S1A_IW_SLC__1SDV_20240108T232450_20240108T232517_052022_06495E_2EAD/scratch/masking_ancillary_darkland_removed.tif ... (8347, 9766) This part is taking the HAND data and refining the water boundaries using the height level. So, I investigated your HAND files and found there were many no-data values. By the way, we have a function for downloading HAND and making VRT files. It was not merged yet, but it is still usable. (https://github.com/opera-adt/DSWX-SAR/pull/17)
cmarshak commented 5 months ago

Following up on each:

  1. The environment is still causing me some headache.

The main difference I see between the environment you shared (i.e. the text file with conda list > ....txt and the one I have is that your has gdal version is 3.7.2 and mine is 3.7.3. I see the patch versions of numpy and scipy are similar. That could be an issue. I could cap gdal and see how the environment changes further.

Ideally, you could provide instructions to replicate your environment on a new machine.

And just for posterity, the instructions here do not work on either my mac or aurora (the shared server): https://github.com/opera-adt/DSWX-SAR/blob/main/docker/lock_package.lock#L2

  1. Is there a way to specify debug mode from dswx_workflow? I get slightly better error handling if I call the functions directly from python. Specifically, I am running the workflow like this (so I can avoid passing a command to terminal, see the notebook) :
run_config_path = out_directory / 'run_config.yml'
with open(run_config_path, 'w') as run_config_yaml:
    yaml.dump(input_dict, run_config_yaml)
cfg = RunConfig.load_from_yaml(str(run_config_path), 'dswx_s1', args)
dswx_s1_workflow(cfg)

Would it just be args.debug_mode = True? Or could I add it somewhere in the runconfig?

  1. I am doing something similarly in practice generating geotiff as indicated in the snippet below - still confused as to why the hanging occurs.
from tile_mate import get_raster_from_tiles

X_hand, p_hand = get_raster_from_tiles(bounds_buffered, tile_shortname='hand')
hand_path = out_directory / f'hand.tif'

with rasterio.open(hand_path, 'w', **p_hand) as ds:
    ds.write(X_hand)

My hand tif is just slightly smaller.

cmarshak commented 5 months ago

We can follow up next week.

cmarshak commented 5 months ago

So to my surprise - 3. (using the VRT provided) does work!

And it saves to sas folder. The products look good.

So, the HAND and environment issues still need to be sorted out.

Thanks @oberonia78 !!

cmarshak commented 5 months ago

Dear @oberonia78 and dswx-sar team,

I can confirm that the issue regarding the hanging step related to masking_ancillary_darkland_removed.tif was solved increasing the hand dataset coverage area. In my case, I used 1 degree around the SLC ID whose bursts I was using. When I increased that to 3 degrees, the error went away. I have a vague understanding that the code takes the bursts and uses all the overlapping MGRS tiles either using a database or the python library mgrs. Since the MGRS tiles overlapping a given area may be much, much larger, it's unclear how much hand is required.

Not sure if this could be more elegantly solved providing a global vrt with ASF urls but don't know what the code is doing. You mentioned the relevant areas of the code are found in this file.

Thank you, Junkgyo!

cmarshak commented 5 months ago

This is assuming that gdal<3.8 is used :)