search

eosphoros-ai / DB-GPT

AI Native Data App Development framework with AWEL(Agentic Workflow Expression Language) and Agents
http://docs.dbgpt.cn
MIT License
13.24k stars 1.75k forks source link

[Document][model] Connect to model serving of DB-GPT cluster in AWEL? #1604

Closed lastshogunshin closed 3 months ago

lastshogunshin commented 3 months ago

Search before asking

Operating system information

Linux

Python version information

3.10

DB-GPT version

main

Related scenes

Installation Information

Device information

GPU 32G

Models information

LLM：Qwen1.5-14B-Chat Embedding: text2vec-large-chinese

What happened

我已经成功在本地linux系统上启动了以下服务（llm，text2vec） image 现在我想在AWEL的示例代码里使用这两项服务实现文本向量化和大模型调用，但是失败了，代码和错误如下： image requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://0.0.0.0:8001/api/v1/embeddings

What you expected to happen

请问如何修改AWEL脚本里的设置才能正确使用cluster中提供的服务？

How to reproduce

1.

Additional context

No response

Are you willing to submit PR?

fangyinc commented 3 months ago

Hi, @lastshogunshin. I suggest you deploy a API Server for the model serving, here is simple command to deploy a API Server:

dbgpt start apiserver \
--controller_addr http://127.0.0.1:8000 \
--api_keys EMPTY \
--port 8100

Note: The controller_addr is the address of the Model Controller, please replace it with the real address of the Model Controller.

API Server will provide a HTTP API which is compatible with OpenAI's API.

Then you can use the DefaultEmbeddingFactory and OpenAILLMClient to connect to the remote DB-GPT model serving.

from dbgpt.core import Embeddings
from dbgpt.model.proxy import OpenAILLMClient
from dbgpt.rag.embedding import DefaultEmbeddingFactory

llm_client = OpenAILLMClient(
    api_key="EMPTY", 
    api_base="http://127.0.0.1:8100/api/v1", 
    model_alias="Qwen1.5-14B-Chat"
)

embeddings: Embeddings = DefaultEmbeddingFactory.remote(
    api_url="http://127.0.0.1:8100/api/v1/embeddings", 
    api_key="EMPTY", 
    model_name="text2vec"
)
lastshogunshin commented 3 months ago

Thank for your help! I was just thinking of setting up a API server may help, and this answer came up. I will try this out and see if it works for this issue, thanks again.

lastshogunshin commented 3 months ago

Thank for your help! I was just thinking of setting up a API server may help, and this answer came up. I will try this out and see if it works for this issue, thanks again.

Sorry just tried it out, the embedding model works just fine, but LLM came up with this error:

ValueError: Model server error!code=1, error msg is **LLMServer Generate Error, Please CheckErrorInfo.**: Error code: 400 - {'object': 'error', 'message': 'Only Qwen1.5-14B-Chat allowed now, your model gpt-3.5-turbo', 'code': 40301}

I check the source code and not sure if gpt-3.5-turbo is the default LLM for OpenAILLMClient(), is there a way to also use Qwen1.5-14B-Chat here without causing any error? Or should I change another client to use? image

fangyinc commented 3 months ago

Thank for your help! I was just thinking of setting up a API server may help, and this answer came up. I will try this out and see if it works for this issue, thanks again.

Sorry just tried it out, the embedding model works just fine, but LLM came up with this error:

ValueError: Model server error!code=1, error msg is **LLMServer Generate Error, Please CheckErrorInfo.**: Error code: 400 - {'object': 'error', 'message': 'Only Qwen1.5-14B-Chat allowed now, your model gpt-3.5-turbo', 'code': 40301}

I check the source code and not sure if gpt-3.5-turbo is the default LLM for OpenAILLMClient(), is there a way to also use Qwen1.5-14B-Chat here without causing any error? Or should I change another client to use? image

Please provide some code or trace logs.

lastshogunshin commented 3 months ago

Please provide some code or trace logs.

Sure, here is the code with error.
image

This is the full log for this error.

`/home/feida/anaconda3/envs/DBGPT/lib/python3.10/site-packages/pydantic/_internal/_fields.py:160: UserWarning: Field "modelname" has conflict with protected namespace "model".

You may be able to resolve this warning by setting model_config['protected_namespaces'] = (). warnings.warn( [Chunk(content='user(id, name, age)', metadata={'source': 'database'}, chunk_id='6ca35ceb-acc9-4dd0-a730-34cfcfd1f7de', score=0.0, summary='', separator='\n')] Traceback (most recent call last): File "/storage/DBGPT/DB-GPT/examples/sdk/chat_data_with_awel.py", line 278, in final_result = asyncio.run( File "/home/feida/anaconda3/envs/DBGPT/lib/python3.10/asyncio/runners.py", line 44, in run return loop.run_until_complete(main) File "/home/feida/anaconda3/envs/DBGPT/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete return future.result() File "/storage/DBGPT/DB-GPT/dbgpt/core/awel/operators/base.py", line 218, in call out_ctx = await self._runner.execute_workflow( File "/storage/DBGPT/DB-GPT/dbgpt/core/awel/runner/local_runner.py", line 92, in execute_workflow await self._execute_node( File "/storage/DBGPT/DB-GPT/dbgpt/core/awel/runner/local_runner.py", line 121, in _execute_node await self._execute_node( File "/storage/DBGPT/DB-GPT/dbgpt/core/awel/runner/local_runner.py", line 121, in _execute_node await self._execute_node( File "/storage/DBGPT/DB-GPT/dbgpt/core/awel/runner/local_runner.py", line 121, in _execute_node await self._execute_node( [Previous line repeated 3 more times] File "/storage/DBGPT/DB-GPT/dbgpt/core/awel/runner/local_runner.py", line 177, in _execute_node raise e File "/storage/DBGPT/DB-GPT/dbgpt/core/awel/runner/local_runner.py", line 160, in _execute_node await node._run(dag_ctx, task_ctx.log_id) File "/storage/DBGPT/DB-GPT/dbgpt/core/awel/operators/base.py", line 186, in _run return await self._do_run(dag_ctx) File "/storage/DBGPT/DB-GPT/dbgpt/core/awel/operators/common_operator.py", line 162, in _do_run input_ctx: InputContext = await curr_task_ctx.task_input.map(map_function) File "/storage/DBGPT/DB-GPT/dbgpt/core/awel/task/task_impl.py", line 538, in map new_outputs, results = await self._apply_func(map_func) File "/storage/DBGPT/DB-GPT/dbgpt/core/awel/task/task_impl.py", line 533, in _apply_func results = await asyncio.gather(*map_tasks) File "/storage/DBGPT/DB-GPT/dbgpt/core/awel/task/task_impl.py", line 126, in map out = await self._apply_func(map_func) File "/storage/DBGPT/DB-GPT/dbgpt/core/awel/task/task_impl.py", line 112, in _apply_func out = await func(self._data) File "/storage/DBGPT/DB-GPT/dbgpt/core/interface/output_parser.py", line 274, in map return self.parse_model_nostream_resp(input_value, "###") File "/storage/DBGPT/DB-GPT/dbgpt/core/interface/output_parser.py", line 302, in parse_model_nostream_resp model_out_text = super().parse_model_nostream_resp(response, sep) File "/storage/DBGPT/DB-GPT/dbgpt/core/interface/output_parser.py", line 138, in parse_model_nostream_resp raise ValueError( ValueError: Model server error!code=1, error msg is LLMServer Generate Error, Please CheckErrorInfo.: Error code: 400 - {'object': 'error', 'message': 'Only Qwen1.5-14B-Chat allowed now, your model gpt-3.5-turbo', 'code': 40301} `

lastshogunshin commented 3 months ago

This issue happened when I try to run "Write Your Own Chat Data With AWEL" tutorial locally: https://docs.dbgpt.site/docs/latest/awel/cookbook/write_your_chat_database

lastshogunshin commented 3 months ago

Fixed it, just need to modify the model config here to local model. Thanks for looking into it！ image