search

ZJCV / KnowledgeReview

[CVPR 2021] Distilling Knowledge via Knowledge Review
Apache License 2.0
7 stars 1 forks source link

feature after relu or before relu? #4

Closed twmht closed 2 years ago

twmht commented 2 years ago

Hi,

I found out that you extract the feature before relu (https://github.com/ZJCV/KnowledgeReview/blob/master/rfd/model/resnet/resnet.py#L35).

But from the offical repo they extract the feature after relu (https://github.com/dvlab-research/ReviewKD/blob/master/CIFAR-100/model/resnet_cifar.py#L186)

Why did you make this difference?

zjykzj commented 2 years ago

Hi,

I found out that you extract the feature before relu (https://github.com/ZJCV/KnowledgeReview/blob/master/rfd/model/resnet/resnet.py#L35).

But from the offical repo they extract the feature after relu (https://github.com/dvlab-research/ReviewKD/blob/master/CIFAR-100/model/resnet_cifar.py#L186)

Why did you make this difference?

In original implementation, it is also get the feature before relu, see

  1. https://github.com/dvlab-research/ReviewKD/blob/cede6ea6387ae9b6127de0e561507177bf19c11e/CIFAR-100/model/resnet_cifar.py#L95
  2. https://github.com/dvlab-research/ReviewKD/blob/cede6ea6387ae9b6127de0e561507177bf19c11e/CIFAR-100/model/resnet_cifar.py#L52

This operation is derived from A Comprehensive Overhaul of Feature Distillation. Because relu will filter all negative values, and this may be useful for learning.

twmht commented 2 years ago

@zjykzj

yup. I was wrong so I have closed this issue. thanks for your comment.