Problems with models where shape of the output image does not match the shape of the input image

[Saareem]

I'm trying a model in Deepness plugin which has Tensor[float32] of shape (1,5,512,512) as the input and it returns a tensor of same data type but cropped to 448 pixels in vertical and horizontal dimension, i.e. output has size (1,2,448,448). Deepness seems to have some trouble in processing the tiles with these kind of models or then I'm doing something incorrectly.

I'm using overlap 25% as that seems to at least make the model run. But when running it processes couple of images correctly and then at some point if fails with the following error message:

Error! Processing error: "Unhandled exception occurred. See Python Console for details"!

Details: ValueError - could not broadcast input array from shape (410,346) into shape (410,410) Last Traceback: File "C:\Users/USERPROFILE/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\deepness\processing\tile_params.py", line 152, in set_mask_on_full_img full_result_img[roi_slice_on_full_image] = tile_result[roi_slice_on_tile_image]

I've also tried setting the overlap to 0 %, but that fails even faster with a rather similar error message:

Error! Processing error: "Unhandled exception occurred. See Python Console for details"!

Details: ValueError - could not broadcast input array from shape (448,448) into shape (512,512) Last Traceback: File "C:\Users/ESAARELAINEN/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\deepness\processing\tile_params.py", line 152, in set_mask_on_full_img full_result_img[roi_slice_on_full_image] = tile_result[roi_slice_on_tile_image]

Is this a bug in the code or is the plugin just lacking an implementation for models where output shape doesn't match the input shape? In the latter case, I can always use the plugin as I library, so it's not that big of a deal.

About the overlap By the way, I find it very confusing that the overlap is given in percentages. Many models have overlap that is some exponent of 2, for example 32, and sometimes it's impossible to set those using percentages. Moreover, it seems that you can not set float percentages with the spinner, only integers. Maybe it's possible using the model metadata?

[bartoszptak]

Hi @Saareem, thank you for your issue.

1. As you suggested, the plugin only supports models that have the same input and output size currently. I suppose your model crops borders and you compensate them by the overlap. We can consider adding support for segmentation models which have smaller output masks than model input by placing them in the middle. Would this solve the problem?

2. The suggestion to declare offset in pixels instead of percentage is a very good idea. We will try to add this option next to the current approach.

If you have any questions, feel free to ask. We'll let you know when we make changes to the code. We also encourage you to contribute.

[Saareem]

You guessed correctly. The model is cropping 32 pixels from each side so I tried to compensate for that by having overlap. I also tried adding padding to the model itself but so far haven't been able to do it without a need for retraining, which would be a rather massive endeavour. Placing the mask in the middle would most likely solve the problem.

[bartoszptak]

Hi @Saareem, we are working on your issue. Could you share with me your model and/or map, with a short description of what results the model should produce? If you don't want to make it public, you can find email on my profile.

[bartoszptak]

Hi @Saareem, We updated the plugin and checked the model you sent by mail and It looks to operate fine now. Could you test it by yourself and get a response?

The newest developer version of the plugin is available here to download. You should remove the current version from QGIS and load it from a ZIP file. A more detailed description is available here.

[bartoszptak]

113