Closed acnagle closed 6 months ago
Hi @acnagle, thank you for your support of LLMLingua.
This is a great question, and I believe other users may have similar queries.
The actual compression ratio indeed has a certain relationship with the iterative_size
. Specifically, the calculation of (\gamma_j) is based on the global Perplexity (PPL) distribution, determined by calculating the quantile according to the compression ratio, as detailed https://github.com/microsoft/LLMLingua/blob/main/llmlingua/prompt_compressor.py#L621.
However, since it's challenging to directly obtain the actual global PPL distribution after compression, it's also difficult to get a true and accurate quantile. Thus, we update the PPL distribution per segment. A smaller iterative_size
results in more sampling points for this estimation, leading to more accurate compression. This is particularly evident when the original prompt size is smaller.
I believe that by improving the get_estimate_threshold_base_distribution
, the estimation of the PPL distribution can be made more accurate.
I hope this answers your question, and thank you again for your support.
Hello again, I was reviewing the code and realized that I might not have understood your explanation as well as I thought. Just to make sure I understand: by using a smaller iterative_size
, we are able to get a better approximation for p(\tilde{s}_j) according to eq. 5 in the paper. And if we have a better approximation then our chosen \gamma_j quantile will be more faithful to the compression ratio that we passed in. Is this correct?
You also mentioned in your comment that there might be a way to improve the get_estimate_threshold_base_distribution
. Do you have any suggestions in this direction?
Hi @acnagle,
Yes, the purpose of iterative compression is to minimize the approximation loss in eq. 5. This approximation can be improved in two ways:
Thank you for your response. I'm looking forward to the followup work!
Thank you for the interesting work, and making the code easily accessible. I have some confusion on the relationship between the
ratio
anditerative_size
parameters.In the case I am interested, there is a single demonstration that I want to compress using only the token-level compression approach. I've noticed that, in general, the final ratio between the compressed and original length can vary quite a bit for large enough' ratio' values. However, I noticed that when I make the
iterative_size
parameter small, e.g. 10, the final compressed ratio is more truthful to the value specified for theratio
parameter.I'm confused as to why this is the case. From the paper, my understanding was that \gamma_j threshold for segment s_j (whose length is defined by the
iterative_size
parameter), was based primarily on theratio
parameter. Meaning that, regardless of theiterative_size
, LLMLingua would always pruneratio
percentage of the tokens in that segment.Any clarifications of this would be useful, including where in the code \gamma_j is computed.