To speed up LLMs' inference and enhance LLM's perceive of key information, compress the prompt and KV-Cache, which achieves up to 20x compression with minimal performance loss.
This pull request introduces detailed docstrings for the PromptCompressor class and its primary method compress_prompt in our codebase. These additions are intended to enhance the understandability and maintainability of our prompt compression feature. #51
This pull request introduces detailed docstrings for the PromptCompressor class and its primary method compress_prompt in our codebase. These additions are intended to enhance the understandability and maintainability of our prompt compression feature. #51