issues
search
microsoft
/
LLMLingua
To speed up LLMs' inference and enhance LLM's perceive of key information, compress the prompt and KV-Cache, which achieves up to 20x compression with minimal performance loss.
https://llmlingua.com/
MIT License
4.42k
stars
241
forks
source link
Fix (LLMLingua): support mps, fix keep_flag out of dimension.
#27
Closed
iofu728
closed
8 months ago
iofu728
commented
8 months ago
add support for 'mps' within the device mapping configuration.
correct the calculation for repeats at a coarse level.
resolve the issue where the 'keep_flag' variable exceeds its intended dimensions.