xusenlinzy / api-for-open-llm

Openai style api for open large language models, using LLMs just as chatgpt! Support for LLaMA, LLaMA-2, BLOOM, Falcon, Baichuan, Qwen, Xverse, SqlCoder, CodeLLaMA, ChatGLM, ChatGLM2, ChatGLM3 etc. 开源大模型的统一后端接口

Apache License 2.0

2.16k stars 252 forks source link

请问它支持langchain中这种方式调用吗：from langchain.llms import OpenAI #213

Closed whm233 closed 6 months ago

whm233 commented 6 months ago

提交前必须检查以下项目 | The following items must be checked before submission

[X] 请确保使用的是仓库最新代码（git pull），一些问题已被解决和修复。 | Make sure you are using the latest code from the repository (git pull), some issues have already been addressed and fixed.
[X] 我已阅读项目文档和FAQ章节并且已在Issue中对问题进行了搜索，没有找到相似问题和解决方案 | I have searched the existing issues / discussions

问题类型 | Type of problem

None

操作系统 | Operating system

None

详细描述问题 | Detailed description of the problem

from langchain.llms import OpenAI from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler

llm = OpenAI(streaming=True, callbacks=[StreamingStdOutCallbackHandler()], temperature=0) resp = llm("Write me a song about sparkling water.")。

Dependencies

# 请在此处粘贴依赖情况
# Please paste the dependencies here

运行日志或截图 | Runtime logs or screenshots

# 请在此处粘贴运行日志
# Please paste the run log here

xusenlinzy commented 6 months ago

在此项目中使用OpenAI是补全接口，ChatOpenAI是对话接口，一般使用ChatOpenAI

whm233 commented 6 months ago

在此项目中使用OpenAI是补全接口，ChatOpenAI是对话接口，一般使用ChatOpenAI

In [5]: from langchain.chat_models import ChatOpenAI ...: from langchain.embeddings import OpenAIEmbeddings ...: from langchain.schema import HumanMessage ...: ...: text = "你好" ...: messages = [HumanMessage(content=text)] ...: ...: llm = ChatOpenAI(openai_api_key="xxx", openai_api_base="http://0.0.0.0:8000/v1/") ...: ...: print(llm(messages)) ...: ...: embedding = OpenAIEmbeddings(openai_api_key="xxx", openai_api_base="http://0.0.0.0:8000/v1/") ...: print(embedding.embed_documents(["你好"])) 在测试时报错：DEBUG | api.routes.chat:create_chat_completion:45 - ==== request ==== {'model': 'gpt-3.5-turbo', 'frequency_penalty': 0.0, 'function_call': None, 'functions': None, 'logit_bias': None, 'max_tokens': 1024, 'n': 1, 'presence_penalty': 0.0, 'response_format': None, 'seed': None, 'stop': ['<|im_end|>', '<|endoftext|>'], 'temperature': 0.7, 'tool_choice': None, 'tools': None, 'top_p': 1.0, 'user': None, 'stream': False, 'prompt_or_messages': [{'content': '你好', 'role': 'user'}], 'echo': False, 'stop_token_ids': [151643, 151644, 151645]} INFO: 127.0.0.1:41372 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error ERROR: Exception in ASGI application 请问是怎么回事

whm233 commented 6 months ago

环境的配置应该也没有问题： "host": "0.0.0.0", "port": 8000, "api_prefix": "/v1", "engine": "default", "model_name": "qwen", "model_path": "/root/.cache/modelscope/hub/qwen/Qwen-7B-Chat-Int4", "adapter_model_path": null, "resize_embeddings": false, "dtype": "half", "device": "cuda", "device_map": "auto", "gpus": null, "num_gpus": 1, "only_embedding": false, "embedding_name": "/qwen/embedding_model/text2vec-large-chinese"

xusenlinzy commented 6 months ago

https://github.com/xusenlinzy/api-for-open-llm/blob/master/tests/chat.py

这个文件测试有问题吗？

whm233 commented 6 months ago

https://github.com/xusenlinzy/api-for-open-llm/blob/master/tests/chat.py

这个文件测试有问题吗？也是一样的问题，我使用的是Qwen-7B-Chat-Int4，和这个有关系吗。日志如下 INFO: Started server process [39973] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit) INFO: 127.0.0.1:59212 - "GET /v1/models HTTP/1.1" 200 OK 2023-12-25 04:47:40.184 | DEBUG | api.routes.chat:create_chat_completion:45 - ==== request ==== {'model': 'gpt-3.5-turbo', 'frequency_penalty': 0.0, 'function_call': None, 'functions': None, 'logit_bias': None, 'max_tokens': 1024, 'n': 1, 'presence_penalty': 0.0, 'response_format': None, 'seed': None, 'stop': ['<|endoftext|>', '<|im_end|>'], 'temperature': 0.9, 'tool_choice': None, 'tools': None, 'top_p': 1.0, 'user': None, 'stream': False, 'prompt_or_messages': [{'content': '感冒了怎么办', 'role': 'user'}], 'echo': False, 'stop_token_ids': [151643, 151644, 151645]} INFO: 127.0.0.1:59212 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error

xusenlinzy commented 6 months ago

auto-gptq这个包安装了吗

whm233 commented 6 months ago

auto-gptq

是的安装了

whm233 commented 6 months ago

auto-gptq这个包安装了吗

from openai import OpenAI

client = OpenAI( api_key="EMPTY", base_url="http://0.0.0.0:8000/v1/", )

Chat completion API

completion = client.completions.create( model="gpt-3.5-turbo", prompt="你好", ) print(completion) 再使用这个案例测试时返回的是下面的内容： Completion(id='cmpl-0e886133-a46c-436b-a94c-07c2cb5f1daa', choices=[CompletionChoice(finish_reason='stop', index=0, logprobs=None, text=' решил его не возвращать и продал его в лондонский уч')], created=1703666994, model='gpt-3.5-turbo', object='text_completion', system_fingerprint=None, usage=CompletionUsage(completion_tokens=15, prompt_tokens=1, total_tokens=16))

whm233 commented 6 months ago

auto-gptq这个包安装了吗

from openai import OpenAI

client = OpenAI( api_key="EMPTY", base_url="http://0.0.0.0:8000/v1/", )

Chat completion API

completion = client.completions.create( model="gpt-3.5-turbo", prompt="你好", ) print(completion) 再使用这个案例测试时返回的是下面的内容： Completion(id='cmpl-0e886133-a46c-436b-a94c-07c2cb5f1daa', choices=[CompletionChoice(finish_reason='stop', index=0, logprobs=None, text=' решил его не возвращать и продал его в лондонский уч')], created=1703666994, model='gpt-3.5-turbo', object='text_completion', system_fingerprint=None, usage=CompletionUsage(completion_tokens=15, prompt_tokens=1, total_tokens=16))

似乎是资源的问题，谢谢大佬。测试通过了但是有点不稳定