[Question]: LLMLingua2 query condition and dynamic ratio

microsoft / LLMLingua

To speed up LLMs' inference and enhance LLM's perceive of key information, compress the prompt and KV-Cache, which achieves up to 20x compression with minimal performance loss.

https://llmlingua.com/

MIT License

4.18k stars 222 forks source link

[Question]: LLMLingua2 query condition and dynamic ratio #128

Open DomStan opened 3 months ago

DomStan commented 3 months ago

Describe the issue

Hi again and thanks for your continued work on this!

Similar question to before, regarding LLMLingua2 support for conditioning on a given query and applying a dynamic compression ratio, currently I see that the corresponding parameters are ignored in the compress prompt function.

Are there any plans on the roadmap to add support for this with LLMLingua2?

iofu728 commented 3 months ago

Hi @DomStan, thanks for your support. Currently, LLMLingua-2 does not support query conditioning and dynamic ratio. We will consider adding support for these features in future versions. Thanks again for your support.