search

petals-infra / chat.petals.dev

💬 Chatbot web app + HTTP and Websocket endpoints for LLM inference with the Petals client
https://chat.petals.dev
308 stars 79 forks source link

Integrate with Langchain #36

Open Sharpz7 opened 1 year ago

Sharpz7 commented 1 year ago

Just wanted to add this here.

https://github.com/langchain-ai/langchain/issues/8563

Ideally we would write it into langchain in a way that the user can choose the URL of the endpoint they want to use (Since its not recommended to use chat.petals.dev)

borzunov commented 1 year ago

For the record, there is an existing integration by Langchain devs that runs the native Petals client: https://python.langchain.com/docs/integrations/llms/petals This connects to the swarm directly (without using this API endpoint), but requires downloading input/output embeddings of the model (a few GB) before running.

@Sharpz7, do you think Petals API support will be still helpful, even though Langchain provides integration for the native client?

Sharpz7 commented 1 year ago

Hmm. I think for applications where space is limited potentially? I think having more options rather than less is still useful to have.

Also just to lower the barrier to entry, that is something that using the petals API directly would give you.