[BUG/Help] openai_api.py中缺少了usage，导致langchain执行失败

walker83 commented 1 year ago

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior

openai 的api中没有封装usage，导致langchain执行chat模型失败
raise e File "/home/langchain-trilium/.venv/lib/python3.10/site-packages/langchain/llms/base.py", line 163, in _generate_helper self._generate( File "/home/langchain-trilium/.venv/lib/python3.10/site-packages/langchain/llms/openai.py", line 824, in _generate "token_usage": full_response["usage"], KeyError: 'usage'

Expected Behavior

No response

Steps To Reproduce

增加了一行后修复了 usage: Optional[int] = 0 class ChatCompletionResponse(BaseModel): model: str object: Literal["chat.completion", "chat.completion.chunk"] choices: List[Union[ChatCompletionResponseChoice, ChatCompletionResponseStreamChoice]] created: Optional[int] = Field(default_factory=lambda: int(time.time())) usage: Optional[int] = 0

Environment

- OS:
- Python:
- Transformers:
- PyTorch:
- CUDA Support (`python -c "import torch; print(torch.cuda.is_available())"`) :

Anything else?

No response

Tsukumizu commented 1 year ago

你好，能否分享下怎么在langchain中按照ChatModel的形式调用chatglm呢？非常感谢

Tsukumizu commented 1 year ago

已经解决了，运行openai_api.py之后，采用langchain的ChatOpenAI调用： llm_chatglm2 = ChatOpenAI( openai_api_base="http://localhost:8000/v1", openai_api_key="none", streaming=True, temperature=0.7 ) 能让Chatglm无缝对接langchain的ChatModel，可以配合ChatPromptTemplate一起使用。

language model直接from langchain.llms import ChatGLM即可

WilliamChen-luckbob commented 1 year ago

已经解决了，运行openai_api.py之后，采用langchain的ChatOpenAI调用： llm_chatglm2 = ChatOpenAI( openai_api_base="http://localhost:8000/v1", openai_api_key="none", streaming=True, temperature=0.7 ) 能让Chatglm无缝对接langchain的ChatModel，可以配合ChatPromptTemplate一起使用。

language model直接from langchain.llms import ChatGLM即可

弱弱的问一下，这个的实现方式就是url和出参入参和openai一致，然后只是openai_api_base指向自己部署的chatglm服务，实际上通过http通信实现的吧？

Tsukumizu commented 1 year ago

@WilliamChen-luckbob 非常正确，就是用OpenAI的Message封装框架发送一个POST请求，到本地部署的API服务，能无缝对接使用ChatGPT的所有场景

mortalGeorge commented 11 months ago

@WilliamChen-luckbob 非常正确，就是用OpenAI的Message封装框架发送一个POST请求，到本地部署的API服务，能无缝对接使用ChatGPT的所有场景

用于agent好用吗，我用from langchain.llms import ChatGLM封装llm，然后使用agent = initialize_agent(llm=llm)，怎么调prompt都没法达到text-davinci-003的效果:(

Tsukumizu commented 11 months ago

@mortalGeorge 可以参考这篇文章：定制基于 ChatGLM-6B 的本地知识库+联网检索能力的 LangChain Agent - 拔赤的文章 - 知乎 https://zhuanlan.zhihu.com/p/635724707 Langchain内置的agent prompt基本是为chatgpt设计的，不太适合chatglm

dbtzy commented 9 months ago

已经解决了，运行openai_api.py之后，采用langchain的ChatOpenAI调用： llm_chatglm2 = ChatOpenAI( openai_api_base="http://localhost:8000/v1", openai_api_key="none", streaming=True, temperature=0.7 ) 能让Chatglm无缝对接langchain的ChatModel，可以配合ChatPromptTemplate一起使用。

language model直接from langchain.llms import ChatGLM即可

想请教一下用from langchain.llms import ChatGLM之后还需要配置什么吗？

我用命令行POST测试有正确回应

curl -X POST "http://127.0.0.1:8000/v1/chat/completions" \
-H "Content-Type: application/json" \
-d "{\"model\": \"chatglm2-6b\", \"messages\": [{\"role\": \"system\", \"content\": \"You are ChatGLM2-6B, a large language model trained by Zhipu.AI. Follow the user's instructions carefully. Respond using markdown.\"}, {\"role\": \"user\", \"content\": \"你好，给我讲一个故事，大概100字\"}], \"stream\": false, \"max_tokens\": 100, \"temperature\": 0.8, \"top_p\": 0.8}"

{"id":"chatcmpl-3nF7JbjqCtGaC9VBc8gVzo","object":"chat.completion","created":1701090590,"model":"chatglm2-6b","choices":[{"index":0,"message":{"role":"assistant","content":"这是一个古老的寓言，叫做《乌龟和兔子》。\n\n从前有一只兔子，他非常快，经常骄傲地说自己是最快的动物。一天，他遇到了一只缓慢而稳定的乌龟。他们互相挑衅，兔子嘲笑乌龟说他从来没有机会赢得比赛。\n\n乌龟感到受到了冒犯，于是他向兔子提出了一个挑战：在一场比赛中，看看谁能先到达终点。兔子非常高兴地接受了挑战，并开玩笑说他不需要全力以赴就能轻易赢得比赛"},"finish_reason":"stop"}],"usage":{"prompt_tokens":65,"total_tokens":165,"completion_tokens":100}}

然后看Langchain的文档想实现下面的代码，报错显示404

from langchain.chains import LLMChain
from langchain.llms import ChatGLM
from langchain.prompts import PromptTemplate

llm = ChatGLM(endpoint_url="http://127.0.0.1:8000/v1")

template = """{question}"""
prompt = PromptTemplate(template=template, input_variables=["question"])

llm_chain = LLMChain(prompt=prompt, llm=llm)
llm_chain.run("北京和上海两座城市有什么不同？")

Traceback (most recent call last):
  File "/home/dxxm/langchain-sql-test/langchain_sql_test.py", line 11, in <module>
    llm_chain.run("北京和上海两座城市有什么不同？")
  File "/usr/local/iCompute/lib/python3.10/site-packages/langchain/chains/base.py", line 505, in run
    return self(args[0], callbacks=callbacks, tags=tags, metadata=metadata)[
  File "/usr/local/iCompute/lib/python3.10/site-packages/langchain/chains/base.py", line 310, in __call__
    raise e
  File "/usr/local/iCompute/lib/python3.10/site-packages/langchain/chains/base.py", line 304, in __call__
    self._call(inputs, run_manager=run_manager)
  File "/usr/local/iCompute/lib/python3.10/site-packages/langchain/chains/llm.py", line 108, in _call
    response = self.generate([inputs], run_manager=run_manager)
  File "/usr/local/iCompute/lib/python3.10/site-packages/langchain/chains/llm.py", line 120, in generate
    return self.llm.generate_prompt(
  File "/usr/local/iCompute/lib/python3.10/site-packages/langchain/llms/base.py", line 507, in generate_prompt
    return self.generate(prompt_strings, stop=stop, callbacks=callbacks, **kwargs)
  File "/usr/local/iCompute/lib/python3.10/site-packages/langchain/llms/base.py", line 656, in generate
    output = self._generate_helper(
  File "/usr/local/iCompute/lib/python3.10/site-packages/langchain/llms/base.py", line 544, in _generate_helper
    raise e
  File "/usr/local/iCompute/lib/python3.10/site-packages/langchain/llms/base.py", line 531, in _generate_helper
    self._generate(
  File "/usr/local/iCompute/lib/python3.10/site-packages/langchain/llms/base.py", line 1053, in _generate
    self._call(prompt, stop=stop, run_manager=run_manager, **kwargs)
  File "/usr/local/iCompute/lib/python3.10/site-packages/langchain/llms/chatglm.py", line 104, in _call
    raise ValueError(f"Failed with response: {response}")
ValueError: Failed with response: <Response [404]>

Tsukumizu commented 9 months ago

@dbtzy 如果http://127.0.0.1:8000/v1/chat/completions正确运行了，那可以这样试下 from langchain.chat_models import ChatOpenAI from langchain.schema import ( HumanMessage, ) from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler chat = ChatOpenAI( openai_api_base="http://localhost:8000/v1", openai_api_key="none", streaming=True, callbacks=[StreamingStdOutCallbackHandler()], temperature=0 )

resp = chat([HumanMessage(content="写一首诗歌。")])

dbtzy commented 9 months ago

感谢，用from langchain.chat_models import ChatOpenAI可以运行

THUDM / ChatGLM2-6B