kemaloksuz / RankSortLoss

Official PyTorch Implementation of Rank & Sort Loss for Object Detection and Instance Segmentation [ICCV2021]
Apache License 2.0
240 stars 26 forks source link

LVIS validation issue #1

Closed xinqiaozhao closed 3 years ago

xinqiaozhao commented 3 years ago

Hi, I met an issue when I try to do validation on LVIS v1 dataset. I follow your work and use 4 V100 to do the validation. However, the validation speed is limited to 0.7 img/second ,it is very slow. Is that normal and could you give me a hint about your validation speed? It will be so helpful! Thank you for sharing your work :)

kemaloksuz commented 3 years ago

Hi,

Thank you for your interest.

RS Loss tends to yield higher confidence scores for the predictions than the standard Cross Entropy Loss, hence the NMS score threshold needs to be adjusted accordingly for efficiency. For LVIS dataset, you may be using the default value, which is 0.0001, which causes slower inference speed when the model is trained by RS Loss. Following table presents a comparison of RS-Mask R-CNN and Mask R-CNN regarding this score threshold:

image

As a result, we recommend setting this threshold to 0.60 for LVIS with RS-Mask R-CNN. We added the LVIS configuration file to the repository yesterday and if you are working with the previous version of the repo, then you should set this score threshold as presented in the following line:

https://github.com/kemaloksuz/RankSortLoss/blob/4afcbc0df65481d0ca4833ed4dc8f39e9975e716/configs/ranksort_loss/ranksort_mask_rcnn_r50_fpn_1x_lvis_v1.py#L36

xinqiaozhao commented 3 years ago

Thank you so much!!