search

adobe / antialiased-cnns

pip install antialiased-cnns to improve stability and accuracy
https://richzhang.github.io/antialiased-cnns/
Other
1.65k stars 203 forks source link

Why do deeper CNNs have better shift consistency? #52

Open vadik6666 opened 1 year ago

vadik6666 commented 1 year ago

With baseline CNNs with no anti-aliasing, we see better shift consistency if we increase the CNN's depth, e.g. VGG11 -> VGG19, Resnet18 -> Resnet152. Why is that so?

richzhang commented 1 year ago

Good question. Higher accuracy naturally lends itself to better shift consistency. A classifier that gets 100% accuracy will be consistent across shifts, even if has no shift-invariant inductive bias to begin with

vadik6666 commented 1 year ago

Thank you @richzhang for a quick response!

  1. Suppose we use an ensemble of very deep CNNs as a teacher and do knowledge distillation to a small student CNN e.g. MobileNetv2. All CNNs in here are not anti-aliased CNNs. Do you think such a student model will have shift consistency?
  2. Same situation as in (1) but CNNs in the ensemble are anti-aliased versions, but student CNN isn't (it is again default MobileNetv2). What do you think about shift consistency of a student model?