Chap 3: Fault about TransformerForSequenceClassification

[S3nnyK]

Information

The question or comment is about chapter:

• [ ] Introduction
• [ ] Text Classification
• [x] Transformer Anatomy
• [ ] Multilingual Named Entity Recognition
• [ ] Text Generation
• [ ] Summarization
• [ ] Question Answering
• [ ] Making Transformers Efficient in Production
• [ ] Dealing with Few to No Labels
• [ ] Training Transformers from Scratch
• [ ] Future Directions

Question or comment

In TransformerForSequenceClassification, x = self.encoder(x)[:, 0, :] means [CLS] token is included in the inputs. However, in the beginning of this chapter, inputs is defined as tokenizer(text, return_tensors="pt", add_special_tokens=False), without special_tokens. Hence, the 0-th is "time", not "[CLS]".