Closed mirth closed 9 months ago
Yes, the past_key_values
parameter is used to reduce redundant computations during Iterative token-level prompt compression.
If you're directly invoking llm_lingua.get_ppl
, then there's no need to use this parameter at all.
If you don't want to activate the KV Cache during the compression process, you can try removing the relevant code.
I hope this answers your question.
Does manipulations with
past_key_values
present only to increase speed? Can I remove code which operatespast_key_values
without lowering the performance?