使用remote方式运行tests.test_query_gradio报下面的错误

acwwt commented 2 months ago

Traceback (most recent call last): File "/root/.conda/envs/InternLM2_Huixiangdou/lib/python3.10/site-packages/aiohttp/web_protocol.py", line 452, in _handle_request resp = await request_handler(request) File "/root/.conda/envs/InternLM2_Huixiangdou/lib/python3.10/site-packages/aiohttp/web_app.py", line 543, in _handle resp = await handler(request) File "/root/huixiangdou/huixiangdou/service/llm_server_hybrid.py", line 574, in inference text = server.generate_response(prompt=prompt, File "/root/huixiangdou/huixiangdou/service/llm_server_hybrid.py", line 487, in generate_response output_text = self.inference.chat(prompt, history) AttributeError: 'HybridLLMServer' object has no attribute 'inference'

tpoisonooo commented 2 months ago

某个 API 在调用本地的 LLM ，我看下 qaq

tpoisonooo commented 2 months ago

我已经运行过一切正常。按如下方法自查：

1. 更新代码

git pull 代码，我的 commit-id 是 af08486036b496a547fe7b5b57550264f917308d

2. 检查 config.ini

这是我的纯 remote 配置，其中 TOKEN 和 bce 模型路径需要用自己的

[feature_store]
reject_throttle = 0.37044240854913324
embedding_model_path = "/data2/khj/bce-embedding-base_v1/"
reranker_model_path = "/data2/khj/bce-reranker-base_v1"
work_dir = "workdir"

[web_search]
x_api_key = "${YOUR-API-KEY}"
domain_partial_order = ["openai.com", "pytorch.org", "readthedocs.io", "nvidia.com", "stackoverflow.com", "juejin.cn", "zhuanlan.zhihu.com", "www.cnblogs.com"]
save_dir = "logs/web_search_result"

[llm]
enable_local = 0
enable_remote = 1
client_url = "http://127.0.0.1:18888/inference"

[llm.server]
local_llm_path = "/data2/khj/internlm2-chat-7b"
local_llm_max_text_length = 3000
local_llm_bind_port = 18888
remote_type = "kimi"
remote_api_key = "你自己的 TOKEN"
remote_llm_max_text_length = 128000
remote_llm_model = "moonshot-v1-128k"
rpm = 30

[worker]
enable_sg_search = 0
save_path = "logs/work.txt"

[worker.time]
start = "00:00:00"
end = "23:59:59"
has_weekday = 0

[sg_search]
binary_src_path = "/usr/local/bin/src"
src_access_token = "${YOUR-SRC-ACCESS-TOKEN}"

[sg_search.opencompass]
github_repo_id = "open-compass/opencompass"
introduction = "用于评测大型语言模型（LLM）. 它提供了完整的开源可复现的评测框架，支持大语言模型、多模态模型的一站式评测，基于分布式技术，对大参数量模型亦能实
现高效评测。评测方向汇总为知识、语言、理解、推理、考试五大能力维度，整合集纳了超过70个评测数据集，合计提供了超过40万个模型评测问题，并提供长文本、安全、代码3类大模型特色技术能力评测。"

[sg_search.lmdeploy]
github_repo_id = "internlm/lmdeploy"
introduction = "lmdeploy 是一个用于压缩、部署和服务 LLM（Large Language Model）的工具包。是一个服务端场景下，transformer 结构 LLM 部署工具，支持 GPU 服务端部
署，速度有保障，支持 Tensor Parallel，多并发优化，功能全面，包括模型转换、缓存历史会话的 cache feature 等. 它还提供了 WebUI、命令行和 gRPC 客户端接入。"

[frontend]
type = "lark"
webhook_url = "https://open.feishu.cn/open-apis/bot/v2/hook/xxxxxxxxxxxxxxx"

3. 启动

我的 config 名叫 .github/scripts/config-ci.ini

python3 -m tests.test_query_gradio --config_path .github/scripts/config-ci.ini

tpoisonooo commented 2 months ago

@acwwt 代码里加了句检查，如果用 remote only 却又调用了 local LLM，自动修复成 remote 继续执行，但会红色报错。

InternLM / HuixiangDou