Panoptic Segmentation - Something wrong with Number of Masks and Quality of Predictions

Dicko87 commented 2 years ago

Hi folks, I am having a few problems with using DETR for panoptic segmentation. Before using the panoptic model I trained a DETR model for object detection and achieved great results! I then created the panoptic dataset and included it as mentioned on facebook/detr (having a coco_panoptic and mycoco folders). I then trained the model using the script: The model trained ok, and after inspecting the log.txt file I got the following maximum scores at the shown number of epochs. A plot of the training and validation curves are shown below. The curves do not look good to be honest and they do not converge. I subsequently chose to use the model from epoch 11 which saved as checkpoint0010.pth. To load the mode, I created the following function in hubconf.py: I the loaded the model using: Now my dataset is confidential but detects tree trunks and signs places on the trees. I put the model into evaluation mode and then ran inference over a single image. I then tried to display the results using the code from the DETR panoptic notebook: This is the ground truth png image: The blue shows the tree trunk and the yellow and green show the signs on the trunk. Now, rather than showing me an image for each of the three detected masks, I am being shown one in with all masks, repeated three times! There should be one pole (ID 0) and two signs (ID 1), one is missed which is ok. This is the result I get: I printed out the tensors (the attn_masks and the Mask log list (same thing really)): Since the mask image appears at ¼ of the original image size, I upscaled the image to be the same as the original image. When overlaying the ground truth with the mask image (yellow), the mask does not match the image very well. It leans the opposite way. I plot the predicted bounding boxes and they look great, one signs is missing but the boxes are in the right place. I tried to run the panoptic segmentation prediction on the image and got rubbish: I used this code: And got this result Obviously something is wrong somewhere and I would love to be able to fix this soon as I am on a very tight deadline. Thank you in advance.