why seq_len of llm is 128?

MrYxJ / calculate-flops.pytorch

The calflops is designed to calculate FLOPs、MACs and Parameters in all various neural networks, such as Linear、 CNN、 RNN、 GCN、Transformer(Bert、LlaMA etc Large Language Model)

https://pypi.org/project/calflops/

MIT License

508 stars 16 forks source link

why seq_len of llm is 128? #8

Closed LinglingGreat closed 11 months ago

LinglingGreat commented 12 months ago

Is it too small?

MrYxJ commented 11 months ago

Yeah it is small for large language model, 128 is just referring to the tool ptflops example which lists many example on the CV Models.