Describe the bug
During inference, certain image raise a ValueError: "can't extend empty axis 0 using modes other than 'constant' or 'empty'"
This seems to be related to the image, not the model, since it occurs for a same image, no matter the model.
See log on HPC: /space/partner/nrcan/geobase/work/transfer/work/deep_learning/operationalization_dev/geosys-jobs/logs/NB_Bakerlake_2017_INF_FORE_2023-07-13T16h46_inference.out
To Reproduce
Steps to reproduce the behavior on HPC (replace [root_dir] with path to "operationnalization dev" directory):
Expected behavior
Complete inference without error ;)
Screenshots
Inferring on patches: 100%|█████████▉| 129000/129516 [24:28:14<05:52, 1.46it/s]
Inferring from images: 0%| | 0/1 [24:28:14<?, ?it/s]
Error executing job with overrides: ['mode=inference', 'inference.root_dir=/space/partner/nrcan/geobase/work/transfer/work/deep_learning/operationalization_dev/inferences/NB_Dalhousie_2017', 'inference.state_dict_path=/space/partner/nrcan/geobase/work/transfer/work/deep_learning/operationalization_dev/checkpoints/RGB_4class_NB-leaf-off_VA_20210824.pth.tar', 'inference.input_stac_item=/space/partner/nrcan/geobase/work/transfer/work/deep_learning/operationalization_dev/data/NB_Dalhousie_2017.json', 'inference.output_path=/space/partner/nrcan/geobase/work/transfer/work/deep_learning/operationalization_dev/inferences/NB_Dalhousie_2017/RGB_4class_NB-leaf-off_VA_20210824.pth.tar/NB_Dalhousie_2017_4classes_raw.tif', 'debug=False']
Traceback (most recent call last):
File "/home/gdl_user/geo-deep-learning/GDL.py", line 62, in run_gdl
task(cfg)
File "/home/gdl_user/geo-deep-learning/inference_segmentation.py", line 442, in main
pred_heatmap = segmentation(
File "/opt/conda/envs/geo_deep_env/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/home/gdl_user/geo-deep-learning/inference_segmentation.py", line 174, in segmentation
for sub_image, h_idxs, w_idxs, hann_win in tqdm(
File "/opt/conda/envs/geo_deep_env/lib/python3.10/site-packages/tqdm/std.py", line 1178, in __iter__
for obj in iterable:
File "/home/gdl_user/geo-deep-learning/inference_segmentation.py", line 122, in gen_img_samples
patch_array = _pad(patch_array, chunk_size)
File "/home/gdl_user/geo-deep-learning/inference_segmentation.py", line 47, in _pad
padded_arr = np.pad(arr, ((0, w_diff), (0, h_diff), (0, 0)), mode="reflect")
File "/opt/conda/envs/geo_deep_env/lib/python3.10/site-packages/numpy/lib/arraypad.py", line 819, in pad
raise ValueError(
ValueError: can't extend empty axis 0 using modes other than 'constant' or 'empty'
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
Describe the bug During inference, certain image raise a ValueError: "can't extend empty axis 0 using modes other than 'constant' or 'empty'" This seems to be related to the image, not the model, since it occurs for a same image, no matter the model.
See log on HPC: /space/partner/nrcan/geobase/work/transfer/work/deep_learning/operationalization_dev/geosys-jobs/logs/NB_Bakerlake_2017_INF_FORE_2023-07-13T16h46_inference.out
To Reproduce Steps to reproduce the behavior on HPC (replace [root_dir] with path to "operationnalization dev" directory):
Expected behavior Complete inference without error ;)
Screenshots
Additional context Other problematic images: