Class token (self.cls_token) in patchembedding block?

Project-MONAI / MONAI

AI Toolkit for Healthcare Imaging

https://monai.io/

Apache License 2.0

5.71k stars 1.04k forks source link

Class token (self.cls_token) in patchembedding block? #2677

Closed saruarlive closed 3 years ago

saruarlive commented 3 years ago

Hi, I have found that class embedding (token, self.cls_token ) is not applied in patchembedding block used in vit class. The vision transformer article mentions it. Is there any reason not to add/concatenate the class token? source: here

Nic-Ma commented 3 years ago

Hi @ahatamiz ,

Could you please help share some comments about this question as you are expert on it?

Thanks in advance.

ahatamiz commented 3 years ago

Hi @saruarlive thank you for your comment. This version of ViT model was designed to serve as backbone for segmentation models and in particular the UNETR model. So it does not utilize class tokens.

I will submit a new PR to support classification applications for ViT model and hence also add the class token.

wyli commented 3 years ago

Tracked by https://github.com/Project-MONAI/MONAI/issues/2682, I'm closing this.