Callback for on_llm_new_token should be invoked before the token is yielded by the model

[eyurtsev]

Privileged issue

• [X] I am a LangChain maintainer, or was asked directly by a LangChain maintainer to create an issue here.

Issue Content

Goal

Improve streaming in LangChain for chat models / language models.

Background

Many chat and language models implement a streaming mode in which they stream tokens one at a time.

LangChain has a callback system that is useful for logging and important APIs like "stream", "stream_log" and "stream_events".

Currently many models incorrectly yield the token (chat generation) before invoking the callback.

Acceptance criteria

For a PR to be accepted and merged, the PR should:

• [ ] Fix the code to make sure that the callback is called before the token is yielded
• [ ] Link to this issue
• [ ] Change ONE and only ONE model
• [ ] FIx sync and async implementation if both are defined

Example PR

Here is an example PR that shows the fix for the OpenAI chat model:

https://github.com/langchain-ai/langchain/pull/16909

Find models that need to be fixed

The easiest way to find places in the code that may need to be fixed is using git grep

git grep -C 5  "\.on_llm_new"

Examine the output to determine whether the callback is called before the token is yielded (correct) or after (needs to be fixed).

[dosubot[bot]]

🤖

It's great to see your interest in improving the streaming functionality in LangChain for chat models and language models. Your proposed goal and acceptance criteria for the PR are clear and well-defined. If you have a solution in mind and are planning to work on a fix, we encourage you to go ahead and create a pull request. Your contribution will be valuable in enhancing the LangChain platform. Thank you for your willingness to improve the codebase!

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[samjove]

Working on this, can I get it assigned?

[efriis]

@samjove feel free to start working on it for a specific model and reference this issue!