search

langchain-ai / langsmith-sdk

LangSmith Client SDK Implementations
https://smith.langchain.com/
MIT License
371 stars 67 forks source link

Load Prompt not from hub when creating an evaluator #965

Closed Lukecn1 closed 5 days ago

Lukecn1 commented 2 weeks ago

Feature request

As the title suggest, I want to be able to use a prompt I have saved and committed, but havent published on the public hub.

Motivation

The reason is that we need to keep our prompts private.

hinthornw commented 1 week ago

So a private prompt? It should be possible already: what errors are you seeing?

Lukecn1 commented 1 week ago

I dont see an error - In the UI when creating an evaluator for a dataset, I cant select a private prompt as the template for the evaluator - only public prompts from the hub.

hinthornw commented 1 week ago

Oh that's indeed frustrating. I'll forward to the team.

@Lukecn1 this is occurring when creating evaluators in the UI? (not showing up in the drop-down list)

Lukecn1 commented 1 week ago

Yes exactly, when I go to select a prompt from the hub, I am not able to select any other than public ones, hence those I have , which are private, are not listed.

hinthornw commented 1 week ago

Hi @Lukecn1 FYi the prompt has to be a structure prompt FYI but I believe this should be supported today. Could you confirm that the prompt you're looking to access is a structured prompt?

Lukecn1 commented 5 days ago

That works yes, thanks for clearing that up. I was not aware that I needed to specify and output schema for it to be used as a evaluator.

Thanks for the responses :)