LangchainRouter not using chain's built-in memory

[bllchmbrs]

I certainly could be doing something wrong, but there's so much "magic" in the LangchainRouter I don't know what I should be doing.

Side Note: A worked example that builds a function, parses input, runs a model, then returns the streaming result would be helpful. Using that to then introduce LangchainRouter as syntatic sugar would helpful for 2 reasons: (1) Users can debug issues themselves, going step by step. (2) it teaches people how the system works, facilitating contributions by others.

It works fine when I supply the chat history, I just don't want to manage that client side and don't see the value in doing so when Langchain should have it integrated.

Scenario

Here is my LangchainRouter:

langchain_router = LangchainRouter(
    langchain_url="/api/v0/streaming",
    langchain_object=ml.get_chat_model(),
    streaming_mode=1
)

Here is my chat model definition:

def get_chat_model():
    """
    llm = ChatOpenAI(model='gpt-3.5-turbo',
                     temperature=0.1,
                     streaming=True)
    retriever = gets_a_retriever()
    memory = ConversationBufferMemory(
        memory_key="chat_history", return_messages=True)
    chain = ConversationalRetrievalChain.from_llm(
        llm=llm,
        retriever=retriever,
        memory=memory,
        verbose=True
    )
    return chain

When I make the API Call:

curl -N -X POST \
-H "Accept: text/event-stream" -H "Content-Type: application/json" \
-d '{"question": "write me a song about sparkling water" }' \
http://localhost:8000/api/v0/streaming

Actual result

{"detail":[{"loc":["body","chat_history"],"msg":"field required","type":"value_error.missing"}]}

Expected result

I would expect that it would use the chain's memory and not require the API to supply it.

Acceptance criteria

• [ ]

[ajndkr]

Hi @bllchmbrs! Totally agree on the working example point. The documentation is definitely lacking at the moment.

About the chat history issue, I initially intended the API endpoints to be stateless. To allow langchain's built-in memory work as expected, we'll need the endpoint to be stateful. maybe a websocket is more suited for your use-case. Have you tried using it?

If the framework doesn't support it, it's something I can look into.

[ajndkr]

closing this issue due to user inactivity.

you can view the new documentation on LangChainRouter here: https://lanarky.ajndkr.com/learn/adapters/langchain/router/

If you are interested to know more about the low-level modules, you can find the documentation for it here: https://lanarky.ajndkr.com/learn/adapters/langchain/fastapi/

please reopen this issue if you'd like to discuss more.