Support my own judge model? --custom judge model

langchain-ai / langsmith-docs

Documentation for langsmith

https://docs.smith.langchain.com/

MIT License

76 stars 32 forks source link

Support my own judge model? --custom judge model #342

Open aiyinyuedejustin opened 2 months ago

aiyinyuedejustin commented 2 months ago

Hi there, I am wondering does the llm-as-a-judge evaluation from LangSmith support customized my own model as a judge? I wish to develop my custom prompts for my own judge model through langsmith.

I see currently there are only OpenAI models supported.

hinthornw commented 1 month ago

It supports any llm you'd like, though we mostly benchmarked OAI/Anthropic/Gemini when we wrote the off-the-shelf prompts