FasterDecoding / SnapKV

141 stars 4 forks source link

Can snapkv compress kv in case different user questions are posed towards the same context? #18

Open namespace-Pt opened 1 week ago

namespace-Pt commented 1 week ago

Say there is a long document, then two users ask two different questions based on the document. These two questions are no way similar, targeting on different part of the document. In this case, can snapkv compress the context robustly?