Erratic Image Segmentation and Interpretation of Polygons

[jayshrivastava0]

We are applying the Pretrained satlas model sentinel2_resnet50_mi_ms.pth with the corresponding config file on a set of 8 sentinel2 images stacked together in (N*C, H, W) shape (see attached RGB visualization) to do image segmentation. The output from the given template custom inference code produces the following output: Output Image It is not clear where this large black area stems from. Based on the choice of the crop size paramter, the size of black area changes, but not as expected. Is this area being segmented as unknown? So we tried some different values for crop size Output Image with crop size = 512 We also tried experimenting with crop size (2000) that is more than the size of input image. Expecting it to throw an error. Output Image with crop size = 2000

This led us to look deeper at the custom inference code and it seems that the crop size is actually the window size upon which various operation are performed. And not the actual cropping size of the input image. Is that the case?

What is the relationship between the size of the input image and the crop-size parameter?

Because we believe this crop size parameter is causing this large black area. If you think that is not the case can you please point us where this large black area might be coming from.

One another question.. How can one interpret the colored polygons? The file 'dataset.py' has the same color codes for different land cover types. For example in crop_type there is [0, 0, 255], # (blue) rice and land_cover there is [0, 0, 255], # (blue) water.

[favyen2]

The crop size just determines the size of the crops/patches passed to the model. If it is smaller, then the image is divided into more smaller crops/patches, and if it is bigger, then the image is divided into fewer larger crops/patches. But either way the entire image should be processed.

I wasn't able to replicate the issue where most of the output is black. Which head are you using (controlled by head_idx in the inference code).

Regarding the colors, the duplicate blue color you mentioned come from different heads. The output visualization created by the example custom inference code only shows the outputs for one head. So you should look in the section of the configuration file or dataset.py that corresponds to the head/task you're running.

If you clarify the head_idx you're using and provide the .npy input then I can check further.

[jayshrivastava0]

We appreciate your prompt reply to this matter, Favyen.

We are using head_idx = 0. This is our npy file that we are using. We also tried to run the inference while changing the head_idx and crop_size. We are still getting the black region.

Thank you for clarifying the confusion on color.

[favyen2]

I'm not able to reproduce the issue with this numpy file where the output only appears on one side of the image.

I do see issue where, because the size is not a multiple of 32, it can run into errors for some of the heads. But if I crop it to e.g. 1024x1024 then it does not have a problem.

Here is example output with head_idx=2 (this is the land cover output that you had mentioned).

[jayshrivastava0]

Thank you Favyen. We are now able to reproduce the output as well.