Keypoint Visualization

[Ananya21162]

Dear Authors,

Thank you for releasing the code for your work, it is really helpful. Can you please help me with keypoint visualization done after Stage 2 from the feature map generated by USAM. The feature map generated for one image (3x224x224) is of the size 256x56x56. So, in order to extract keypoints using threshold, were these feature maps converted to 2D shape let's say by taking mean and then min-max normalisation and thresholding was applied? Further, how were these keypoints mapped back to original input image since its size differ from that of the feature map.

Eagerly waiting for your response, really appreciate your help.

[AggMan96]

Thank you for your interest in our work. Sorry for a late reply. In our paper, we use the ''att'' feature map for visualization, which is from USAM in code. The shape of the feature map is 1x56x56. And then min-max normalization and thresholding are applied in the feature map. Lastly, resizing the feature map to 1x224x224 for mapping back to original input image, and using 'cv2.applyColorMap' to highlight the feature map. It is noted that the feature map extracted by USAM is only used to enhance the feature discrimination of images rather than to detect actual mapping, and we don't use keypoint annotations for training. Hope this reply can help you.

[Ananya21162]

Thank you so much for your reply, helps a lot! I used the same methodology to extract keypoints feature map of size same as image (resizing using bilinear interpolation) and then mapping them back to original image, this works but the kepoints at different epochs for both drone & satellite are mostly coming out to be on boundary. The training works fine, I am able to reproduce results for Drone--> Satellite recall and mAP, but the keypoints are not moving inside. Did you face any similar challenge, would be a great help if you could suggest something here? Also, I am wondering how the mapping was done for visulaisation in Fig 5 (Red points & yellow correspondences)? Really appreciate the work you have done, it is indeed interesting. Thank you for your help!

[Ananya21162]

Keypoint feature map for drone/satellite is extracted from respective view using ft_resnet_50 block at S2 USC.

 output,_= extract_kpts_desc(model_ft, inputs1, device) # function extracts keypts of shape [8x1x56x56]
 #min-max normalisation
 min_val = output.min(-1)[0].min(-1)[0]
 max_val = output.max(-1)[0].max(-1)[0]
 image = (output-min_val[:,:,None,None])/(max_val[:,:,None,None]-min_val[:,:,None,None])
 for i in range(image[n][0].shape[0]):
     for j in range(image[n][0].shape[1]):
            if image[n][0][i,j]>threshold:
                image[n][0][i,j]=255
                count+=1
            else:
                image[n][0][i,j]=0

 new_image = F.interpolate(image, size=[h, w],mode='bilinear', align_corners=True) #blow up the binary feature map to original image size
 print(f'Count is {count}')
 N,C,H,W = new_image.shape #8x1x224x224; batch=8
 new_image = new_image.reshape(N,H,W)   #8x224x224
 highlighted_img= cv2.applyColorMap(new_image[n].cpu().detach().numpy().astype(np.uint8), cv2.COLORMAP_JET) #highlight feature map
 [cv2.imwrite(path, highlighted_img)

Using the above code I am getting this as my feature map, as you can see the keypoints are coming out to be in corners, which is not improving even after training the model using train.py

Can you please help in this? Waiting for your revert! @AggMan96

[AggMan96]

Thanks for your recognition of our work. We use the ''att'' feature map before "output" for keypoints visualization. Also, you can use "cv2.threshold" to adjust different thresholds for visulization. As for Fig 5, the red points and yellow correspondences are used to illustrate that USAM can extract the salient keypoints from different views. Hope this can help you.