[Bug]: Process hanging indefinitely with apply_shadow_masking : true

[abradley60]

Checked for duplicates

Yes - I've already checked

Describe the bug

I am creating some backscatter products over the Antarctic region (coastal and inland). The workflow works as expected for a wide range of products I have tested. However, when I configure the workflow to apply and save the layover shadow mask, the process seems to hang indefinitely on certain bursts. This will only happen on some products.

E.g. Here are the logs for a test scene where I encountered the issue below. I let the process sit for approx. 10 hours. There was still high CPU usage so the process seemed to be ongoing.

Identification:
    product type: RTC_S1
    product version: 1.0.1
Ancillary input(s):
    DEM file: /data/dem/glo_30/S1B_IW_SLC__1SSH_20210223T233056_20210223T233124_025740_031194_E7BE_dem.tif
Processing parameters:
    apply RTC: True
    apply thermal noise correction: True
    apply absolute radiometric correction: True
    apply valid samples sub-swath masking: True
    apply shadow masking: True
    apply bistatic delay correction: False
    apply static tropospheric delay correction: False
    skip if already processed: False
    scratch dir: /data/OPERA/scratch/temp_1707445567.2457469/temp_1707452886.070618
    output dir: /data/OPERA/outdir/S1B_IW_SLC__1SSH_20210223T233056_20210223T233124_025740_031194_E7BE
    save bursts: True
    save mosaics: False
    save browse: True
    output imagery format: COG
    output imagery compression: DEFLATE
    output imagery nbits: 32
    save secondary layers as HDF5 files: True
    check ancillary coverage: False
Save layers:
    Mask Layer: True
    RTC area normalization factor: True
    RTC area normalization factor Gamma0 to Beta0: True
    Number of Looks: True
    Incidence Angle: False
    Local Incidence Angle: True
    Projection Angle: False
    RTC Area Normalization Factor Gamma0 to Beta0 (Projection Angle - ProjectionAngle): True
    Range Slope: False
    Digital Elevation Model (DEM): True
Browse images:
    burst height: 2048
    burst width: None
Check ancillary inputs' coverage:
    DEM file coverage: (not tested)
Number of bursts to process: 1
Processing burst: t164_351771_iw2 (1/1)
    product ID: OPERA_L2_RTC-S1B_IW_SLC__1SSH_20210223T233056_20210223T233124_025740_031194_E7BE
    burst geogrid:
        start X: -732300.000000
        end X: -700140.000000
        start Y: 1519860.000000
        end Y: 1417680.000000
        spacing X: 20.000000
        spacing Y: -20.000000
        width: 1608
        length: 5109
        epsg: 3031
    reading burst SLCs
    applying thermal noise correction to burst SLC
    applying absolute radiometric correction to burst SLC
    computing layover shadow mask for t164_351771_iw2
journal:
 -- DEM EPSG: 4326
 -- Output EPSG: 4326
journal:
 -- Processing block: 1 
 --   - line start: 0
 --   - line end  : 1000
 --   - dopplers near mid far: 0 0 0 
journal:
 -- Actual DEM bounds used:
 -- Top Left: -27.2192 -74.4911
 -- Bottom Right: -24.6192 -75.6261 
 -- Spacing: 0.000833333 -0.000277778
 -- Dimensions: 3121 4087

Topo progress (block 1/2): 100%

it hung here.

What did you expect?

I ran the same workflow again with both apply_shadow_masking : false and save_mask : false set and it worked as expected:

apply_shadow_masking : false and save_mask : false -> SUCCESS (approx 50 mins) apply_shadow_masking : true and save_mask: true -> HANGING (process still ongoing after 10 hours)

If I had to guess I thought it could potentially be a DEM coverage issue? In some areas there can be a large amount of no-data in the DEM.

Here is the .yaml file I am using. Details on the scene can be found here.

runconfig:
  name: rtc_s1_workflow_default

  groups:
      primary_executable:
          product_type: RTC_S1

      pge_name_group:
          pge_name: RTC_S1_PGE

      input_file_group:
          # Required. List of SAFE files (min=1)
          safe_file_path: [/data/scenes/S1B_IW_SLC__1SSH_20210223T233056_20210223T233124_025740_031194_E7BE.SAFE]

          orbit_file_path: [/data/osv/POEORB/S1B_OPER_AUX_POEORB_OPOD_20210315T112322_V20210222T225942_20210224T005942.EOF]

          # Optional. Burst ID to process (empty for all bursts)
          # burst_id: [t069_147173_iw1, t069_147174_iw1]
          # burst_id: [t071_151225_iw1, t071_151226_iw1]

      dynamic_ancillary_file_group:
          # Digital elevation model
          # dem_file: /data/dem/glo_30/S1B_IW_SLC__1SSH_20190223T222639_20190223T222706_015079_01C2E9_1D63_dem.tif
          dem_file: /data/dem/glo_30/S1B_IW_SLC__1SSH_20210223T233056_20210223T233124_025740_031194_E7BE_dem.tif

          # Digital elevation model description
          dem_file_description:

      static_ancillary_file_group:

          # burst database sqlite file
          burst_database_file:

      product_group:
          processing_type: 'CUSTOM'

          # Directory where PGE will place results
          product_path: .

          # Directory where SAS writes temporary data
          scratch_path: /data/OPERA/scratch

          # If option `save_bursts` is set, output bursts are saved to:
          #     {output_dir}/{burst_id}/{product_id}{suffix}.{ext}
          # If option `save_mosaics` is set, output mosaics are saved to:
          #     {output_dir}/{product_id}{suffix}.{ext}
          # If the field `product_id`` is left empty, the prefix "rtc_product"
          # will be used instead.
          # `suffix` is only used when there are multiple output files.
          # `ext` is determined by geocoding_options.output_imagery_format.
          output_dir: /data/OPERA/outdir/S1B_IW_SLC__1SSH_20210223T233056_20210223T233124_025740_031194_E7BE
          product_id: OPERA_L2_RTC-S1B_IW_SLC__1SSH_20210223T233056_20210223T233124_025740_031194_E7BE

          save_bursts: True
          save_mosaics: True
          output_imagery_format: COG
          output_imagery_compression: DEFLATE
          output_imagery_nbits: 32

          # Optional. Save secondary layers (e.g., inc. angle) within 
          # the HDF5 file
          save_secondary_layers_as_hdf5: True

          # Save RTC-S1 metadata in the HDF5 format
          # Optional for `output_imagery_format` equal to 'ENVI', 'GTiff', or
          # 'COG', and enabled by default for `output_imagery_format` equal
          # to 'HDF5' or 'NETCDF' or `save_secondary_layers_as_hdf5` is True
          save_metadata: True

      processing:

          # Check if ancillary input cover entirely output products
          check_ancillary_inputs_coverage: False

          polarization: co-pol

          # Options to run geo2rdr
          geo2rdr:
              threshold: 1.0e-8
              numiter: 25

          # Options to run rdr2geo
          rdr2geo:
              threshold: 1.0e-7
              numiter: 25

          # Apply absolute radiometric correction
          apply_absolute_radiometric_correction: True

          # Apply thermal noise correction
          apply_thermal_noise_correction: True

          # OPTIONAL - Apply RTC
          apply_rtc: True

          # Apply bistatic delay correction
          apply_bistatic_delay_correction: False

          # Apply static tropospheric delay correction
          apply_static_tropospheric_delay_correction: False

          # DEM interpolation method
          # Choices - 'sinc', 'bilinear', 'bicubic', 'nearest', 'biquintic'
          dem_interpolation_method: bilinear

          # OPTIONAL - to control behavior of RTC module
          # (only applicable if geocode.apply_rtc is True)
          rtc:
              # OPTIONAL - Choices:
              # "gamma0" (default)
              # "sigma0"
              output_type: gamma0

              # OPTIONAL - Choices:
              # "bilinear_distribution" (default)
              # "area_projection"
              algorithm_type: area_projection

              # OPTIONAL - Choices:
              # "beta0" (default)
              # "sigma0"
              input_terrain_radiometry: beta0

              # OPTIONAL - Minimum RTC area factor in dB
              rtc_min_value_db: -30

              # RTC DEM upsampling
              dem_upsampling: 2

          # OPTIONAL - Mechanism to specify output posting and DEM
          geocoding:

              # OPTIONAL - Apply valid-samples sub-swath masking
              apply_valid_samples_sub_swath_masking: True

              # OPTIONAL - Apply shadow masking
              apply_shadow_masking: True

              # OPTIONAL -
              algorithm_type: area_projection

              # OPTIONAL - Choices: "single_block", "geogrid", "geogrid_radargrid", and "auto" (default)
              memory_mode: auto

              # OPTIONAL - Processing upsampling factor applied to input geogrid
              geogrid_upsampling: 2

              # Save the incidence angle
              save_incidence_angle: False

              # Save the local-incidence angle
              save_local_inc_angle: True

              # Save the projection angle
              save_projection_angle: False

              # Save the RTC ANF compuated with the projection angle method
              save_rtc_anf_projection_angle: True

              # Save the range slope angle
              save_range_slope: False

              # Save the number of looks used to compute RTC-S1
              save_nlooks: True

              # Save the RTC area normalization factor (ANF) used to generate
              # the RTC product
              save_rtc_anf: True

              # Save the RTC area normalization factor (ANF) gamma0 to sigma0
              save_rtc_anf_gamma0_to_sigma0: True

              # Save interpolated DEM used to compute RTC-S1
              save_dem: True

              # Save layover shadow mask
              save_mask: True

              # OPTIONAL - Absolute radiometric correction
              abs_rad_cal: 1

              # OPTIONAL - Clip values above threshold
              clip_max:

              # OPTIONAL - Clip values below threshold
              clip_min:

              # OPTIONAL - Double sampling of the radar-grid
              # input sampling in the range direction
              upsample_radargrid: False

              # Fields to populate the products' metadata required by
              # CEOS Analysis Ready Data specifications
              estimated_geometric_accuracy_bias_x:
              estimated_geometric_accuracy_bias_y:
              estimated_geometric_accuracy_stddev_x:
              estimated_geometric_accuracy_stddev_y:

              bursts_geogrid:
                  output_epsg: 3031
                  x_posting: 20
                  y_posting: 20
                  x_snap: 20
                  y_snap: 20
                  top_left:
                      x:
                      y:
                  bottom_right:
                      x:
                      y:

          mosaicking:
              # OPTIONAL - Choices: "average", "first", "bursts_center" (default)
              mosaic_mode: first
              mosaic_geogrid: 
                  output_epsg: 3031
                  x_posting: 20
                  y_posting: 20
                  x_snap: 20
                  y_snap: 20
                  top_left:
                      x:
                      y:
                  bottom_right:
                      x:
                      y:

Reproducible steps

- Download SLC and associated metadata to necessary folders 
- Cop 30m DEM downloaded with - https://pypi.org/project/dem-stitcher/ and padded with no data if the scene extent wasn't covered. The DEM can be found here - https://drive.google.com/file/d/19Z-vbNhoGelBQ_uDwmer0ADYKXG4B50F/view
- docker run rtc_s1.py {path_to_above_config}.yaml

Environment

- docker image - opera/rtc:final_1.0.1
- fedora linux
- aws t2.xlarge instance (4xCPU, 8GB Ram)

[gshiroma]

Thank you, @abradley60 , for reporting this issue. Yes, the problem could be in the DEM coverage and/or the DEM no-data values. I noticed that you have disabled the flag check_ancillary_coverage in your runconfig. We require that flag to be enabled in production, and we do not guarantee that the process will succeed if the flag is disabled. Also, the DEM we use in production does not have no-data values (including in ocean areas). Maybe you can try to substitute your DEM no-data values with 0 and see if the processing succeeds.

[abradley60]

Thanks for the quick response @gshiroma. I will change the check_ancillary_coverage to true and replace the no-data values to 0 to match your workflow. I'll report back if that solves the problem.

[abradley60]

@gshiroma that looks to have solved the issue! I am now filling the missing glo_30 data with zero's (mainly off the Antarctic coast) and the products are processing as expected. Interesting that they would process with no-data filling provided apply_shadow_masking : false . Thanks again for your help! Great to have tools like these available and open.