Benchmark prompt summarizers with frontier LLMs (GPT-4o / Opus)

AnswerDotAI / cold-compress

Cold Compress is a hackable, lightweight, and open-source toolkit for creating and benchmarking cache compression methods built on top of GPT-Fast, a simple, PyTorch-native generation codebase.

https://www.answer.ai/posts/2024-08-01-cold-compress.html

BSD 3-Clause "New" or "Revised" License

86 stars 8 forks source link

Benchmark prompt summarizers with frontier LLMs (GPT-4o / Opus) #1

Closed griff4692 closed 5 months ago

griff4692 commented 5 months ago

See this section of the writeup

Inspiration

RECOMP

The goal is to test out different methods for taking an existing prompt and shortening it.

This just involves a prompt along the lines of "summarize this prompt".

Evaluation metrics:

Compression Ratio
Benchmark performance (Lm-eval-harness is integrated into gpt-fast).
This should be tested across settings (RAG, n-shot, short prompt, long prompt, etc.)

griff4692 commented 5 months ago

De-prioritized