search

SJTU-IPADS / PowerInfer

High-speed Large Language Model Serving on PCs with Consumer-grade GPUs
MIT License
7.9k stars 406 forks source link

How to integrate with LangChain? #64

Open tigerinus opened 9 months ago

tigerinus commented 9 months ago

Does this framework provide any interoperability with existing ecosystem like LangChain?

hodlen commented 9 months ago

We are currently in the early stages of development, so integrating into the ecosystem may not be ideal at this moment. Our plan is to support orchestration and user interfaces more robustly after achieving feature stability and full platform support, which we expect to happen in the coming weeks.

However, I have created a llama-cpp-python fork for PowerInfer. This fork provides Python bindings and could potentially act as a backend for LangChain, as well as other applications and frameworks. It is 100% experimental at this stage, but we encourage you to try it out and share your feedback with us!