[2021] Do Vision Transformers See Like Convolutional Neural Networks?

[gatheluck]

論文リンク

• arXiv

公開日（yyyy/mm/dd）

2021/08/19 Google Research Brain Team

概要

TeX

% yyyy/mm/dd
@article{
    raghu2021do,
    title={Do Vision Transformers See Like Convolutional Neural Networks?},
    author={Maithra Raghu and Thomas Unterthiner and Simon Kornblith and Chiyuan Zhang and Alexey Dosovitskiy},
    journal=arXiv # "2108.08810",
    year={2021}
}

[gatheluck]

1.どんなもの(what is it?)

ViTとCNNが獲得している特徴表現の違いと、その違いは何故生まれるのかを調査した研究。

A study investigating the differences in feature representations acquired by ViT and CNN, and where these differences come from.

[gatheluck]

2. 先行研究と比べてどこがすごい？(What is the advantage over previous studies?)

とにかく実験量が多く、計算リソースとJFT-300Mを使って上から殴ってくるスタイルの研究。

The amount of experiments is huge, and they are using unlimited computational resources and JFT-300M to beat others.

[gatheluck]

3. 技術や手法のキモはどこ？

複数の実験によって下記のような点を明らかにしたこと。 ViTが獲得している特徴表現は、CNNが獲得しているものと比較して下記の点が顕著に異なっていた。

• ViTはより均一な特徴表現を獲得しており、浅層と深層の特徴表現の類似度が大きかった。
• ViTはResNetと比較して浅層から大域情報を多く取り入れている。一方で、浅層が局所情報を考慮することは依然として重要であり、大規模な事前学習は浅層の注意機構がこのことを学習するのに役立っている。
• ViTにおいてはskip connectionの影響がCNNよりも大きく、性能と表現の類似性に大きく寄与している。
• ViTはCNNと比較して入力された空間情報をより上手く保存している。

Clarifing following things by multiple experiments. The feature representation acquired by ViT is significantly different from those acquired by CNN in the following aspects.

• ViT having more uniform representations, with greater similarity between lower and higher layers.
• ViT incorporates more global information than ResNet at lower layers. Nevertheless, incorporating local information at lower layers remains vital, with large-scale pre-training data helping early attention layers learn to do this.
• Skip connections in ViT are even more influential than in ResNets, having strong effects on performance and representation similarity.
• ViT stores the input spatial information better than CNN.

[gatheluck]

4.どうやって有効だと検証した？(How did you verify that it works?)

To analyze representation similarity, they used the centered kernel alignment (CKA). CKA takes representations (activation matrices) of two layers as input and its output is similarity of input representations. (Figure 1 is visualization result of CKA of ViT and ResNet.)

And they also computing the CKA between ViT and ResNet. The result shows that the lower half of ResNet layers are similar to around the lowest quarter of ViT layers.

[gatheluck]

Plotting attention head mean distances shows lower ViT layers attend both locally and globally when they are trained by Mega dataset.

However, if the number of dataset becomes small (like ImageNet), ViT does not learn to attend locally in earlier layers.

[gatheluck]

Lower layer representations of ResNet are most similar to representations corresponding to local attention heads of ViT.