search

mit-han-lab / streaming-llm

[ICLR 2024] Efficient Streaming Language Models with Attention Sinks
https://arxiv.org/abs/2309.17453
MIT License
6.59k stars 361 forks source link

Question about attention sink arising in pretrained models #68

Open kevinli573 opened 10 months ago

kevinli573 commented 10 months ago

In section 3.3 of the paper, it mentions 160M parameter models were pretrained. For these models, do you know roughly when during the training, e.g., after how many steps/tokens trained, did the attention sink phenomenon started to arise?

Thanks!