dosubot[bot]
commented
5 months ago To address the issue of Replacement Characters (�) appearing in outputs with multibyte characters from Google VertexAI, it's crucial to handle UTF-8 character encoding properly, especially when dealing with streams. The core of the problem lies in how chunks are processed and converted to strings, potentially splitting multibyte characters across chunks.
Here's a more direct approach to solve this:
-
Accumulate Data in a Buffer: Instead of converting each chunk to a string immediately, accumulate the data in a Buffer. This ensures that you're not splitting any multibyte characters.
-
Use
StringDecoderfor Conversion: Utilize theStringDecoderfrom Node.js'sstring_decodermodule to convert the entire Buffer to a string after all data has been received.StringDecoderis designed to handle multibyte characters that may be split across multiple chunks.
Here is a simplified example:
const { StringDecoder } = require('string_decoder');
const decoder = new StringDecoder('utf8');
let buffer = Buffer.alloc(0); // Initialize an empty buffer
// Example: Simulate receiving chunks of data
const chunks = [Buffer.from([0xE3, 0x81, 0x82]), Buffer.from([0xE3, 0x81, 0x84])];
chunks.forEach((chunk) => {
buffer = Buffer.concat([buffer, chunk]); // Accumulate chunks
});
// Convert buffer to string, correctly handling multibyte characters
const str = decoder.write(buffer) + decoder.end();
console.log(str); // Correctly prints characters without replacement charactersThis approach ensures that you're not inadvertently introducing replacement characters by splitting multibyte characters across chunks.
For contributing to the LangChain.js codebase with a fix for this issue:
- Fork and clone the repository.
- Create a new branch for your changes.
- Implement the fix, ensuring you handle multibyte characters correctly as described.
- Add tests to verify your changes.
- Commit and push your changes to your fork, then open a pull request to the original repository with a detailed description of your fix.
Make sure to review the LangChain.js contribution guidelines in their CONTRIBUTING.md file for more detailed instructions on how to contribute effectively.
#### Details
**Sources** - [langchain/src/util/tests/googlevertexai-connection.test.ts](https://github.com/langchain-ai/langchainjs/blob/main/langchain/src/util/tests/googlevertexai-connection.test.ts) - [libs/langchain-google-vertexai-web/src/tests/chat_models.int.test.ts](https://github.com/langchain-ai/langchainjs/blob/main/libs/langchain-google-vertexai-web/src/tests/chat_models.int.test.ts) - [langchain-core/src/output_parsers/tests/output_parser.test.ts](https://github.com/langchain-ai/langchainjs/blob/main/langchain-core/src/output_parsers/tests/output_parser.test.ts) - [examples/src/llms/googlevertexai-streaming.ts](https://github.com/langchain-ai/langchainjs/blob/main/examples/src/llms/googlevertexai-streaming.ts) - [libs/langchain-community/src/llms/googlevertexai/common.ts](https://github.com/langchain-ai/langchainjs/blob/main/libs/langchain-community/src/llms/googlevertexai/common.ts) - [libs/langchain-google-common/src/utils/stream.ts](https://github.com/langchain-ai/langchainjs/blob/main/libs/langchain-google-common/src/utils/stream.ts) - [CONTRIBUTING.md](https://github.com/langchain-ai/langchainjs/blob/main/CONTRIBUTING.md) - [.github/pull_request_template.md](https://github.com/langchain-ai/langchainjs/blob/main/.github/pull_request_template.md)**About Dosu** This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.
To continue the conversation, mention @dosu.
pokutuna
rossanodr
matzkoh
Checked other resources
Example Code
Make the model output long texts containing multibyte characters as a stream.
Error Message and Stack Trace (if applicable)
(No errors or stack traces occur)
Output Example: Includes Replacement Characters (�)
Description
This issue occurs when requesting outputs from the model in languages that include multibyte characters, such as Japanese, Chinese, Russian, Greek, and various other languages, or in texts that include emojis 😎.
This issue occurs due to the handling of streams containing multibyte characters and the behavior of
buffer.toString()method in Node. https://github.com/langchain-ai/langchainjs/blob/a1ed4fee2d8ebf8d26e1979a4d463c96c38cd177/libs/langchain-google-gauth/src/auth.ts#L15When receiving a stream containing multibyte characters, the point at which a chunk (
readable.on('data', ...)is executed) is may be in the middle of a character’s byte sequence. For instance, the emoji "👋" is represented in UTF-8 as0xF0 0x9F 0x91 0x8B. The callback might be executed after only0xF0 0x9Fhas been received.buffer.toString()attempts to decode byte sequences assuming UTF-8 encoding. If the bytes are invalid, it does not throw an error, instead silently outputs a REPLACEMENT CHARACTER (�). https://nodejs.org/api/buffer.html#buffers-and-character-encodingsTo resolve this, use
TextDecoderwith thestreamoption. https://developer.mozilla.org/en-US/docs/Web/API/TextDecoder/decodeRelated Issues
The issue has been reported below, but it persists even in the latest version.
The same issue occurred when using Google Cloud's client libraries instead of LangChain, but it has been fixed.
I will send a Pull Request later, but I am not familiar with this codebase, and there are many google-related packages under libs/ which I have not grasped enough. Any advice would be appreciated.
System Info