search

ZJCV / KnowledgeReview

[CVPR 2021] Distilling Knowledge via Knowledge Review
Apache License 2.0
7 stars 1 forks source link

Did you reimplement for object detection? #5

Open twmht opened 2 years ago

twmht commented 2 years ago

Hi,

Did you reimplement for object detection? I have tried ReviewKD for my own dataset and my own model, but found out it's not good.

zjykzj commented 2 years ago

Hi,

Did you reimplement for object detection? I have tried ReviewKD for my own dataset and my own model, but found out it's not good.

No related plan. In my practice, the effectiveness of distillation depends very much on the data set used. by the way, I also recommend you to try another distillation algorithm - https://github.com/ZJCV/overhaul

twmht commented 2 years ago

@zjykzj

Yup. Recently I have tried many distilltion methods which work on COCO well, but for widerface I can't reproduce a better student. it's very sensitive with the learning rate and other hyper parameters. It would be great if you can share your experience on this.

overhaul is on going.

zjykzj commented 2 years ago

What scene are you working in? Beside distillation, i also recommend you to try network pruning, which works for me. You can check another two repositories that i did for network pruning.

  1. ZJCV/NetworkSlimming
  2. ZJCV/SSL
twmht commented 2 years ago

Well, I have also tried network slimming or other pruning methods(https://github.com/microsoft/nni) before, but the result is the same as https://arxiv.org/abs/1810.05270. So I am using anytime network to do the pruning now, it's much easier to implement. but I also want to migrate distilling after pruning a network structure.

I have experimented KD like overhaul and Knowledge Review on object detection, but I found out they are not stable when training, we may have loss divergence in the middle of the training.

other methods like https://github.com/yzd-v/FGD, which is specific for object detection but very hard to tune on my own dataset.