search

langchain-ai / langchain

🦜🔗 Build context-aware reasoning applications
https://python.langchain.com
MIT License
90.2k stars 14.27k forks source link

Garbled characters and errors in Japanese streaming output with LangChain and watsonx.ai #19637

Open onoyu1012 opened 4 months ago

onoyu1012 commented 4 months ago

Checked other resources

Example Code

# %%
import os
from langchain_ibm.llms import WatsonxLLM
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
from langchain.callbacks.manager import CallbackManager

os.environ['WATSONX_APIKEY'] = WASONX_APIKEY

model_id = 'ibm-mistralai/mixtral-8x7b-instruct-v01-q'
project_id = PROJECT_ID
url = 'https://us-south.ml.cloud.ibm.com'
params = {
    'decoding_metho': 'greedy',
    'max_new_tokens': 4096,
    'repetition_penalty': 1.1,
    "stop_sequences": [],
}

llm = WatsonxLLM(model_id=model_id, url=url, project_id=project_id, params=params, streaming=True, callbacks=CallbackManager([StreamingStdOutCallbackHandler()]))

#%%
query = 'IBMとはどういう会社ですか？必ず日本語で回答して下さい！'
llm.invoke(query)

Error Message and Stack Trace (if applicable)

IBMï¼ã¢ã¤ã»ãã¼ã»ã¨ã ï¼ã¯ãã¢ã¡ãªã«åè¡åã«ããTraceback (most recent call last):
  File "/home/onoyu1012/.local/lib/python3.10/site-packages/ibm_watsonx_ai/foundation_models/inference/base_model_inference.py", line 225, in _generate_stream_with_url
    parsed_response = json.loads(response)
  File "/usr/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.10/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 126 (char 125)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/onoyu1012/work/udemy/test/streamingtestbywatsonxai.py", line 23, in <module>
    llm.invoke(query)
  File "/home/onoyu1012/.local/lib/python3.10/site-packages/langchain_core/language_models/llms.py", line 248, in invoke
    self.generate_prompt(
  File "/home/onoyu1012/.local/lib/python3.10/site-packages/langchain_core/language_models/llms.py", line 569, in generate_prompt
    return self.generate(prompt_strings, stop=stop, callbacks=callbacks, **kwargs)
  File "/home/onoyu1012/.local/lib/python3.10/site-packages/langchain_core/language_models/llms.py", line 748, in generate
    output = self._generate_helper(
  File "/home/onoyu1012/.local/lib/python3.10/site-packages/langchain_core/language_models/llms.py", line 606, in _generate_helper
    raise e
  File "/home/onoyu1012/.local/lib/python3.10/site-packages/langchain_core/language_models/llms.py", line 593, in _generate_helper
    self._generate(
  File "/home/onoyu1012/.local/lib/python3.10/site-packages/langchain_ibm/llms.py", line 374, in _generate
    for chunk in stream_iter:
  File "/home/onoyu1012/.local/lib/python3.10/site-packages/langchain_ibm/llms.py", line 412, in _stream
    for stream_resp in self.watsonx_model.generate_text_stream(
  File "/home/onoyu1012/.local/lib/python3.10/site-packages/ibm_watsonx_ai/foundation_models/inference/base_model_inference.py", line 227, in _generate_stream_with_url
    raise Exception(f"Could not parse {response} as json")
Exception: Could not parse {"model_id":"ibm-mistralai/mixtral-8x7b-instruct-v01-q","created_at":"2024-03-27T07:21:39.943Z","results":[{"generated_text":"æ as json

Description

I'm trying to output Japanese by streaming using LangChain ​​and watsonx.ai. I'm hoping that Japanese will be output via streaming. However, the characters are garbled and errors occur. I tried streaming Japanese using Langchain and OpenAI with the following Python script. It was successful.

# %%
from langchain_openai.llms import OpenAI
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
from langchain.callbacks.manager import CallbackManager
import os

os.environ['OPENAI_API_KEY'] = OPENAI_API_KEY
llm = OpenAI(streaming=True, callback_manager=CallbackManager([StreamingStdOutCallbackHandler()]), verbose=True, temperature=0)
#
llm.invoke('IBMとはどういう会社ですか？必ず日本語で回答して下さい。')
$ python3 streamingtestbyopenai.py
IBMは、国際的な情報技術企業であり、コンピューターやソフトウェア、クラウドサービス、人工知能などの分野で活躍しています。1911年にアメリカで創業し、現在は世界各国に拠点を持ち、多様な業界や企業に対して革新的なソリューションを提供しています。日本でも長年にわたり事業を展開し、多くの企業や組織にITの力でビジネスを支援しています。また、社会貢献活動や環境保護活動にも積極的に取り組んでおり、持続可能な社会の実現にも貢献しています。

System Info

"pip freeze | grep langchain"

$ pip freeze | grep langchain
langchain==0.1.13
langchain-community==0.0.29
langchain-core==0.1.33
langchain-ibm==0.1.3
langchain-openai==0.1.1
langchain-text-splitters==0.0.1

platform (windows/WSL)

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

python version

$ python3 --version
Python 3.10.12
MateuszOssGit commented 2 months ago

I tried to reproduce this issue using the same code as in the example but i was not able to. I am getting fully response without any error. @onoyu1012 Could you please update langchain_ibm version and try again run your code?

llm.invoke(query)

output:

IBM（アイ・ビー・エム）は、アメリカ合衆国にある情報技術企業です。1911年に設立され、世界中の多くの国に事業を展開しています。IBMは、コンピューター、ソフトウェア、サービスなどの情報技術関連の製品やサービスを提供しています。彼らは、クラウドコンピューティング、人工知能、データ分析、セキュリティなど、最新のITテクノロジーについて深い知識を持っています。IBMは、企業や政府機関などの客先でも活用されている、高度なIT技術を開発し、提供しています。彼らは、創造的な解決策やイノベーションに貢献することを目指しています。

System Info

pip list | grep langchain

langchain                               0.1.17
langchain-community                     0.0.37
langchain-core                          0.1.52
langchain-ibm                           0.1.5

python version

Python 3.10.13