4612/4636 AutoRIFT job failures in the last two weeks are because they are attempting to crop a data file with no valid data, resulting in an error that looks like this:
Traceback (most recent call last):
File "/opt/conda/envs/hyp3-autorift/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/envs/hyp3-autorift/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/opt/conda/envs/hyp3-autorift/lib/python3.9/site-packages/hyp3_autorift/__main__.py", line 47, in <module>
main()
File "/opt/conda/envs/hyp3-autorift/lib/python3.9/site-packages/hyp3_autorift/__main__.py", line 42, in main
process_entry_point.load()()
File "/opt/conda/envs/hyp3-autorift/lib/python3.9/site-packages/hyp3_autorift/process.py", line 595, in main
product_file, browse_file = process(g1, g2, parameter_file=args.parameter_file, naming_scheme=args.naming_scheme)
File "/opt/conda/envs/hyp3-autorift/lib/python3.9/site-packages/hyp3_autorift/process.py", line 543, in process
cropped_file = crop_netcdf_product(netcdf_file)
File "/opt/conda/envs/hyp3-autorift/lib/python3.9/site-packages/hyp3_autorift/crop.py", line 68, in crop_netcdf_product
grid_x_min, grid_x_max = x_values.min(), x_values.max()
File "/opt/conda/envs/hyp3-autorift/lib/python3.9/site-packages/numpy/core/_methods.py", line 44, in _amin
return umr_minimum(a, axis, None, out, keepdims, initial, where)
ValueError: zero-size array to reduction operation minimum which has no identity
Scenes with no valid data will end with the suffix P000.nc, where P000 represents the % valid pixels in the scene.
You can reproduce this failure by running this from the development branch:
4612/4636 AutoRIFT job failures in the last two weeks are because they are attempting to crop a data file with no valid data, resulting in an error that looks like this:
Scenes with no valid data will end with the suffix
P000.nc
, whereP000
represents the % valid pixels in the scene.You can reproduce this failure by running this from the development branch:
and show it's resolved by re-running from this branch.
This PR
This PR looks at the file name for the uncropped file and only crops if there is a non-zero % of valid pixels. Otherwise it uploads the file as-is
(ITS_LIVE had been uploading them before we added cropping to mark "successfully processed" so those scenes wouldn't get re-processed).
This PR also:
.io
to.utils
to prevent shadowing the builtin io and make flake8 happyprocess
next to the browse image creation