Adding code for evaluation, along with Squality and refernce-based metrics.

AnswerDotAI / cold-compress

Cold Compress is a hackable, lightweight, and open-source toolkit for creating and benchmarking cache compression methods built on top of GPT-Fast, a simple, PyTorch-native generation codebase.

https://www.answer.ai/posts/2024-08-01-cold-compress.html

BSD 3-Clause "New" or "Revised" License

85 stars 8 forks source link

Adding code for evaluation, along with Squality and refernce-based metrics. #16

Closed fladhak closed 4 months ago

fladhak commented 4 months ago

Here's some initial eval code for you guys to take a look and give feedback. I'll push some additional commit(s) for supporting Dolomites, MuSiQue, and QMSum.

griff4692 commented 4 months ago

LGTM - please run ruff format *.py before merging to master -- otherwise, good to go