the known detection accuracy to decrease

[YH-2023]

@Went-Liang The probability of unknown objects is very high, higher than the probability of known objects, causing the known detection accuracy to decrease. Why is this?

[RockJim2001]

Hi, my friend. I wonder that how to use the pretrain models at the training stage.The probability of known objects is closed to the paper, but the probability of unknow objects is very low.I'd appreciate it if you could let me know！

[RockJim2001]

the picture is the result i got!

[YH-2023]

@RockJim2001 Is this what you are referring to? https://github.com/Went-Liang/UnSniffer/blob/dad023f7089b4b63c33723ef2ca0860782185bb8/detection/configs/VOC-Detection/faster-rcnn/UnSniffer.yaml#L4

I am not using the data set in the paper. In your reproduction process, what are the probabilities of known classes and unknown classes? When I use the GOC module for inference, the probabilities of known and unknown classes are very different, resulting in the calculated Unknown overconfidence.The probability of the unknown is very high, and the probability of the known is very low.

[RockJim2001]

@YH-2023 I'm glad i can communicate with you! I refer to the following section: I wonder how to use this pretraining models to unduplicate the code of the paper. Just now, i only used the pretraining model to evaluate according to the README file. the below is the result:

The sixth line is the result gotten by training the whole model without the pretrain model. The seventh line is the result gotten by only evaluating the performance of the model with the pretrain model.

[YH-2023]

@YH-2023 I'm glad i can communicate with you! I refer to the following section: I wonder how to use this pretraining models to unduplicate the code of the paper. Just now, i only used the pretraining model to evaluate according to the README file. the below is the result:

The sixth line is the result gotten by training the whole model without the pretrain model. The seventh line is the result gotten by only evaluating the performance of the model with the pretrain model.

Do you mean that directly using the weights in the README cannot achieve the indicators in the paper? Did you reproduce it directly?

[RockJim2001]

Yes，you are right！ Directly evaluting the model with the weights in the README without training. Next, I plan to load the pre-training model, and then start the second stage of training from 12000 times to see if the results of the paper can be replicated！

[Went-Liang]

I have tested the code and model from the repository, and these are the test screenshots on COCO-OOD. I'm not sure if this response can solve your issue.

[YH-2023]

Hi@Went-Liang When detecting unknowns, will it affect the known ones? Now I only added the GOC module. The unknown indicators are not bad, but the known indicators have become low. Why is this? Did you encounter any problems during the training process? Have you ever encountered this problem?

[Went-Liang]

As the results in the paper demonstrate, our model indeed experiences a decrease of approximately 2% on known objects. We attribute this decrease to the combined effects of VOS, GOC, and negative energy suppression. VOS impairs the model's generalization ability. Both GOC and negative energy suppression also conflict with the training methodology of the original detector. However, for now, we believe that VOS plays the dominant role, as our model's performance on known object detection is very similar to that of VOS2.

[YH-2023]

As the results in the paper demonstrate, our model indeed experiences a decrease of approximately 2% on known objects. We attribute this decrease to the combined effects of VOS, GOC, and negative energy suppression. VOS impairs the model's generalization ability. 正如论文中的结果所示，我们的模型确实在已知对象上减少了约 2%。我们将这种下降归因于 VOS、GOC 和负能量抑制的综合影响。 VOS 损害了模型的泛化能力。Both GOC and negative energy suppression also conflict with the training methodology of the original detector. However, for now, we believe that VOS plays the dominant role, as our model's performance on known object detection is very similar to that of VOS2.GOC和负能量抑制也与原始探测器的训练方法相冲突。然而，目前我们认为 VOS 发挥主导作用，因为我们的模型在已知物体检测上的性能与 VOS2 非常相似。

Is there any way to alleviate the impact on known classes? Or, at the expense of unknown accuracy, keep the known ones basically unchanged (a small decrease is okay)

[Went-Liang]

Perhaps you can replace VOS with MSP, as this method does not involve training the model.

[YH-2023]

Perhaps you can replace VOS with MSP, as this method does not involve training the model.也许你可以用 MSP 代替 VOS，因为这种方法不涉及训练模型。

I did not use the virtual points in VOS, and only added a GOC module to my model.Which MSP paper are you talking about?

[Went-Liang]

MSP is the contrastive algorithm mentioned in the paper. It uses the maximum value in the classification head as the basis for classifying known and unknown objects (the maximum classification probability for unknown objects is relatively lower). As for why GOC leads to a drop in known accuracy, I am not sure because GOC only provides supervision for the boxes around known objects and should not have such a significant impact.