search

lm-sys / RouteLLM

A framework for serving and evaluating LLM routers - save LLM costs without compromising quality!
Apache License 2.0
3.3k stars 250 forks source link

How to set threshold value? #36

Closed Harinisri29 closed 3 months ago

Harinisri29 commented 4 months ago

What this threshold value actually means? routed_model = client.route( prompt="What's the squareroot of 144?", router="bert", threshold=0.4066, ) print(f"Prompt should be routed to {routed_model}")

[For 50.0% strong model calls for bert, threshold = 0.4066] what does it exactly implies? Does it mean that among 10 queries 50% will be routed to strong models?

iojw commented 4 months ago

We described what threshold are here: https://github.com/lm-sys/RouteLLM?tab=readme-ov-file#threshold-calibration, let me know if you have any questions!

Specifically, what it means is that for the Chatbot Arena dataset which we used to calibrate, 50% of the queries were routed to the strong model using that threshold. Depending on how similar your input queries are to the Chatbot Arena dataset, the actual % might be greater or smaller.

iojw commented 3 months ago

Closing this for now, let me know if you have any other questions!