search

ASFHyP3 / hyp3-autorift

A HyP3 plugin for feature tracking processing with AutoRIFT-ISCE
BSD 3-Clause "New" or "Revised" License
7 stars 2 forks source link

Don't crop when there is no data #254

Closed jhkennedy closed 3 months ago

jhkennedy commented 3 months ago

4612/4636 AutoRIFT job failures in the last two weeks are because they are attempting to crop a data file with no valid data, resulting in an error that looks like this:

Traceback (most recent call last):
  File "/opt/conda/envs/hyp3-autorift/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/conda/envs/hyp3-autorift/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/opt/conda/envs/hyp3-autorift/lib/python3.9/site-packages/hyp3_autorift/__main__.py", line 47, in <module>
    main()
  File "/opt/conda/envs/hyp3-autorift/lib/python3.9/site-packages/hyp3_autorift/__main__.py", line 42, in main
    process_entry_point.load()()
  File "/opt/conda/envs/hyp3-autorift/lib/python3.9/site-packages/hyp3_autorift/process.py", line 595, in main
    product_file, browse_file = process(g1, g2, parameter_file=args.parameter_file, naming_scheme=args.naming_scheme)
  File "/opt/conda/envs/hyp3-autorift/lib/python3.9/site-packages/hyp3_autorift/process.py", line 543, in process
    cropped_file = crop_netcdf_product(netcdf_file)
  File "/opt/conda/envs/hyp3-autorift/lib/python3.9/site-packages/hyp3_autorift/crop.py", line 68, in crop_netcdf_product
    grid_x_min, grid_x_max = x_values.min(), x_values.max()
  File "/opt/conda/envs/hyp3-autorift/lib/python3.9/site-packages/numpy/core/_methods.py", line 44, in _amin
    return umr_minimum(a, axis, None, out, keepdims, initial, where)
ValueError: zero-size array to reduction operation minimum which has no identity

Scenes with no valid data will end with the suffix P000.nc, where P000 represents the % valid pixels in the scene.

You can reproduce this failure by running this from the development branch:

hyp3_autorift LC09_L1GT_209113_20240325_20240325_02_T2 LC09_L1GT_209113_20230915_20230915_02_T2

and show it's resolved by re-running from this branch.

This PR

This PR looks at the file name for the uncropped file and only crops if there is a non-zero % of valid pixels. Otherwise it uploads the file as-is

(ITS_LIVE had been uploading them before we added cropping to mark "successfully processed" so those scenes wouldn't get re-processed).

This PR also: