search

samuelint / langchain-openai-api-bridge

A bridge to use Langchain output as an OpenAI-compatible API
MIT License
7 stars 3 forks source link

Configurable event streaming #41

Open etburke opened 3 weeks ago

etburke commented 3 weeks ago

Hello,

I've just used your library to bring some order to my Langchain/FastAPI/Vercel AI SDK project. Super handy - thanks for the hard work/contribution!

In my scenario I have a fairly complex chain with many steps, including multiple chat model calls. Only the output of the final chat model needs to be streamed to the user, though, but all of the "on_chat_model_stream" events are being passed along in LangchainStreamAdapter. I could also see the utility of being able to choose to pass custom events to the user. I'm wondering if an optional lambda could be included in the configuration so https://github.com/samuelint/langchain-openai-api-bridge/blob/dc398f0deab77ff50e18a115f6a8913efa34fa5c/langchain_openai_api_bridge/chat_completion/langchain_stream_adapter.py#L28-L38 could be overridden with custom logic.

I'd be happy to take a shot at implementing, but wanted to get your thoughts first.

Thanks!

samuelint commented 2 weeks ago

Hi @etburke,

indeed, all of the "on_chat_model_stream" are streamed in the latest version. There's currently no workaround for this. I might fix the issue soon since I have the same issue in multiple projects using this library, but I cannot give any timeframe of when it's gonna be fix.

A lambda could definetly to fix the problem. Feel free to contribute & open a Pull Request! It's probably going to be quicker for you :)

Something similar I have thought for long term but did not implemented yet, is the ability to have middleware. This would also work with other type of events LanggraphEventToOpenAIAssistantEventStream and allow full flexibility. Something that would look like:

class MyEventMiddleware(ChatEventMiddleware):

  def does_match(self, event):
    return event["event"] == "on_chat_model_stream"

  def execute(self, event):
    ...

and have every internal adapters a middleware.

Feel free to choose the solution that works best for your use case if you contribute :smile:

Hope this help!