BUG: ValueError: can't extend empty axis 0 using modes other than 'constant' or 'empty'

Describe the bug During inference, certain image raise a ValueError: "can't extend empty axis 0 using modes other than 'constant' or 'empty'" This seems to be related to the image, not the model, since it occurs for a same image, no matter the model.

See log on HPC: /space/partner/nrcan/geobase/work/transfer/work/deep_learning/operationalization_dev/geosys-jobs/logs/NB_Bakerlake_2017_INF_FORE_2023-07-13T16h46_inference.out

To Reproduce Steps to reproduce the behavior on HPC (replace [root_dir] with path to "operationnalization dev" directory):

apptainer exec --cleanenv --nv --bind [root_dir] [root_dir]/singularity_images/gdl-cuda11-v2_7_1.sif \
/bin/bash -c 'mkdir -p [root_dir]/inferences/NB_Bakerlake_2017; python /home/gdl_user/geo-deep-learning/GDL.py \
--config-path=[root_dir]/geo-deep-learning-conf/config --config-name=ccmeo_production mode=inference \
inference.root_dir=[root_dir]/inferences/NB_Bakerlake_2017 \
inference.state_dict_path=[root_dir]/checkpoints/RGB_4class_NB-leaf-off_VA_20210824.pth.tar \
inference.input_stac_item=[root_dir]/data/NB_Bakerlake_2017.json \
inference.output_path=[root_dir]/inferences/NB_Bakerlake_2017/RGB_4class_NB-leaf-off_VA_20210824.pth.tar/NB_Bakerlake_2017_4classes_raw.tif \
debug=False'

Expected behavior Complete inference without error ;)

Screenshots

Inferring on patches: 100%|█████████▉| 129000/129516 [24:28:14<05:52,  1.46it/s]
Inferring from images:   0%|          | 0/1 [24:28:14<?, ?it/s]
Error executing job with overrides: ['mode=inference', 'inference.root_dir=/space/partner/nrcan/geobase/work/transfer/work/deep_learning/operationalization_dev/inferences/NB_Dalhousie_2017', 'inference.state_dict_path=/space/partner/nrcan/geobase/work/transfer/work/deep_learning/operationalization_dev/checkpoints/RGB_4class_NB-leaf-off_VA_20210824.pth.tar', 'inference.input_stac_item=/space/partner/nrcan/geobase/work/transfer/work/deep_learning/operationalization_dev/data/NB_Dalhousie_2017.json', 'inference.output_path=/space/partner/nrcan/geobase/work/transfer/work/deep_learning/operationalization_dev/inferences/NB_Dalhousie_2017/RGB_4class_NB-leaf-off_VA_20210824.pth.tar/NB_Dalhousie_2017_4classes_raw.tif', 'debug=False']
Traceback (most recent call last):
  File "/home/gdl_user/geo-deep-learning/GDL.py", line 62, in run_gdl
    task(cfg)
  File "/home/gdl_user/geo-deep-learning/inference_segmentation.py", line 442, in main
    pred_heatmap = segmentation(
  File "/opt/conda/envs/geo_deep_env/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/home/gdl_user/geo-deep-learning/inference_segmentation.py", line 174, in segmentation
    for sub_image, h_idxs, w_idxs, hann_win in tqdm(
  File "/opt/conda/envs/geo_deep_env/lib/python3.10/site-packages/tqdm/std.py", line 1178, in __iter__
    for obj in iterable:
  File "/home/gdl_user/geo-deep-learning/inference_segmentation.py", line 122, in gen_img_samples
    patch_array = _pad(patch_array, chunk_size)
  File "/home/gdl_user/geo-deep-learning/inference_segmentation.py", line 47, in _pad
    padded_arr = np.pad(arr, ((0, w_diff), (0, h_diff), (0, 0)), mode="reflect")
  File "/opt/conda/envs/geo_deep_env/lib/python3.10/site-packages/numpy/lib/arraypad.py", line 819, in pad
    raise ValueError(
ValueError: can't extend empty axis 0 using modes other than 'constant' or 'empty'

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

Additional context Other problematic images:

NB_Dalhousie_2017
NL_Meadows_GE01_20190627

NRCan / geo-deep-learning

BUG: ValueError: can't extend empty axis 0 using modes other than 'constant' or 'empty' #537