search

matejgrcic / DenseHybrid

Official implementation of paper "DenseHybrid: Hybrid Anomaly Detection for Dense Open-set Recognition"
GNU General Public License v2.0
36 stars 2 forks source link

mIoU for task models #2

Open gudovskiyd opened 2 years ago

gudovskiyd commented 2 years ago

Hi @matejgrcic, thanks for your work! I have couple of questions about task model accuracy on FS & SMIYC test results

  1. As I understand you used WRN38 for FS, which suppose to have 83.5% mIoU. Table 2 reports 81.0% for DenseHybrid and 81.4 for SynBoost. Does it mean that DenseHybrid sacrifices 83.5-81.0=2.5% accuracy? Also, do you know why SynBoost is not 83.5% since it doesn't retrain the model?
  2. Could tell me the LDN-121 mIoU before and after DenseHybrid retraining that has been used in SMIYC benchmark?
matejgrcic commented 2 years ago

Hi,

thank you for your interest.

  1. We initialize our network form cityscapes_cv0_wideresnet38_nosdcaug.pth checkpoint taken from this NVIDIA repo. Evaluation with our code shows that it achieves 81.4% mIoU. After the DenseHybrid finetuning we have 81.0%. Hence, the fine-tuning sacrifices around 0.4% mIoU. Contacting the authors of SynBoost might resolve your concerns with their model.
  2. LDN-121 is not a SotA model. It doesn't deliver strong classification performance (e.g. 80+ mIoU on cityscapes). Still, it has a small memory footprint and can be trained quickly so it is ideal for prototyping.
gudovskiyd commented 2 years ago

@matejgrcic thanks for answers

  1. I found the discrepancy: 83.5% is for test and 81.4% is for val Cityscapes split.
  2. NFlowJS with LDN-121 reported 77.4% mIoU. Sad that we don't have such numbers for DenseHybrid