A/B Test LLMs?

[krrishdholakia]

Hi @Fluder-Paradyne @Tarraann @jedan2506

In production, we don't know if Llama2 is going to provide:

• good results
• quickly

Would it be helpful to provide a way to easily A/B test between new models in production?

Context - I'm working on LiteLLM and we recently released a way to a/b test straight from the completion endpoint:

Tutorial: https://docs.litellm.ai/docs/tutorials/ab_test_llms

[jedan2506]

Thank you for your valuation feedback. We will surely look into the information you provided and have a detailed discussion on it.