search

microsoft / MInference

To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.
https://aka.ms/MInference
MIT License
284 stars 11 forks source link

issues

Newest
Newest Most commented Recently updated Oldest Least commented Least recently updated