使用 Higress + LobeChat 快速搭建私人GPT助理（支持 RAG、联网）

johnlanni commented 1 month ago

第一步：创建一个名为 `docker-compose.yml` 的文件，并填入以下内容：

注意：

YOUR_DASHSCOPE_API_KEY 需要替换为你自己的通义千问的 API Key；

/path-to-local-config-folder 需要替换为一个本地可以用来保存配置文件的目录路径。

CODE=访问密码 这里换成你自己的密码

version: '3.9'

networks:
  higress-net:
    external: false

services:
  higress:
    image: higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:1.4.1
    environment:
      - CONFIG_TEMPLATE=ai-proxy
      - DEFAULT_AI_SERVICE=qwen
      - DASHSCOPE_API_KEY=YOUR_DASHSCOPE_API_KEY
    networks:
      - higress-net
    ports:
      - "8080:8080/tcp"
      - "8001:8001/tcp"
    volumes:
      - /path-to-local-config-folder:/data
    restart: always
  lobechat:
    # docker hub 如果访问不了，可以改用这个地址：registry.cn-hangzhou.aliyuncs.com/2456868764/lobe-chat:v1.1.3
    image: lobehub/lobe-chat
    environment:
      - CODE=访问密码
      - OPENAI_API_KEY=unused
      - OPENAI_PROXY_URL=http://higress:8080/v1
    networks:
      - higress-net
    ports:
      - "3210:3210/tcp"
    restart: always

第二步：在命令行中运行以下命令，启动 docker compose 项目：

docker compose -p higress-ai up -d

第三步：在浏览器里访问 http://localhost:3210/ 打开 LobeChat 页面；

Tips 1：开启联网搜索能力

在浏览器里访问 http://localhost:8001/ 打开 Higress 控制台

在这里做配置调整：

provider:
  apiTokens:
  - "你的Dashscope API Key"
  # openai 模型到 qwen 的模型映射，qwen-long 最便宜，百万token 5毛钱，qwen-max 效果最好
  modelMapping:
    '*': "qwen-long"
    gpt-4: "qwen-max"
    gpt-4-turbo: "qwen-max"
    gpt-4o: "qwen-max"
  type: "qwen"
  # 开启联网搜索能力
  qwenEnableSearch: true

Tips 2：开启 RAG 能力

在浏览器里访问 http://localhost:8001/ 打开 Higress 控制台

上传一份文件用于 RAG (详细可以查看这里的文档):

curl --location --request POST 'https://dashscope.aliyuncs.com/compatible-mode/v1/files' \
  --header 'Authorization: Bearer 这里填APIKey' \
  --form 'file=@" ./doc.md"' \
  --form 'purpose="file-extract"'

# 返回内容：
{"id":"file-fe-xxxxxxx","object":"file","bytes":596687,"created_at":1716635947,"filename":"doc.md","purpose":"file-extract","status":"processed"}

在这里做配置调整：

provider:
  apiTokens:
  - "你的Dashscope API Key"
  # openai 模型到 qwen 的模型映射，qwen-long 最便宜，百万token 5毛钱，qwen-max 效果最好
  modelMapping:
    '*': "qwen-long"
    gpt-4: "qwen-max"
    gpt-4-turbo: "qwen-max"
    gpt-4o: "qwen-max"
  type: "qwen"
  # 在这里配置上面的 file id
  qwenFileIds:
  - "file-fe-xxxxxxx"

注意 RAG 只有 qwen-long 支持，且文件内容转变为 token 数，每次请求都会加上进行计费

其他

Higress AI 相关插件的详细文档可以查阅：https://higress.io/zh-cn/docs/plugins/ai/ai-proxy

dpy013 commented 1 month ago

😀 17:11:56 介绍一下江苏省 🤯 17:14:58 警告请求 OpenAI 服务出错，请根据以下信息排查或重试 json { "error" : { "cause" : { "errno" : -111 , "code" : "ECONNREFUSED" , "syscall" : "connect" , "address" : "172.29.0.2" , "port" : 8080 } }, "endpoint" : "http://higress:****/v1" , "provider" : "openai" }

johnlanni commented 1 month ago

@dpy013 进higress容器看下/var/log/higress/gateway.log

johnlanni commented 1 month ago

@dpy013 你是ARM处理器吗，刚刚支持了下多架构镜像，可以重新拉取一下镜像试试

dpy013 commented 1 month ago

不是，是x86的腾讯云vps

johnlanni commented 1 month ago

@dpy013 给错日志了，docker exec -it ai-higress-1 bash

然后cat /var/log/higress/gateway.log

dpy013 commented 1 month ago

gateway.log 这日志里的错误可能修复吗？

CH3CHO commented 4 weeks ago

gateway.log 这日志里的错误可能修复吗？

没看到什么错误啊

johnlanni commented 4 weeks ago

@dpy013 怀疑是配置文件没有权限写入，你按这个步骤试一下：

# 创建一个工作目录，以及其下的配置目录
mkdir -p higress/data
# 进入工作目录
cd higress

在工作目录下创建这份 docker-compose.yaml:

version: '3.9'

networks:
  higress-net:
    external: false

services:
  higress:
    image: higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:1.4.0
    environment:
      - CONFIG_TEMPLATE=ai-proxy
      - DEFAULT_AI_SERVICE=qwen
      - DASHSCOPE_API_KEY=YOUR_DASHSCOPE_API_KEY
    networks:
      - higress-net
    ports:
      - "8080:8080/tcp"
      - "8001:8001/tcp"
    volumes:
      - ./data:/data
    restart: always
  lobechat:
    image: lobehub/lobe-chat
    environment:
      - CODE=访问密码
      - OPENAI_API_KEY=unused
      - OPENAI_PROXY_URL=http://higress:8080/v1
    networks:
      - higress-net
    ports:
      - "3210:3210/tcp"
    restart: always

haifzhu commented 3 weeks ago

还是8080端口没有开启。

johnlanni commented 3 weeks ago

@haifzhu 定位到是新版本镜像的配置问题，你重新执行一下 docker pull higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:1.4.0 ，再启动试试

johnlanni commented 2 weeks ago

参加编程挑战赛的朋友可以用这个issue里的方式做本地测试，给AI代理插件配上以下配置：

provider:
  apiTokens:
  - "你的Dashscope API Key"
  modelMapping:
    '*': "qwen-long"
    gpt-4: "qwen-max"
    gpt-4-turbo: "qwen-max"
    gpt-4o: "qwen-max"
  type: "qwen"
  qwenFileIds:
  # 比赛使用的文档，需要自己上传获取fileid，具体看赛事说明文档
  - "file-fe-xxxxx"

使用lobechat 对话的效果（注意选择gpt-3.5，才能映射到qwen-long）：

pepesi commented 1 week ago

有直接操作ingress添加ai代理的例子吗？

johnlanni commented 1 week ago

@pepesi 你用这个模式启动，在ingresses目录下就是ingress文件了，mcpbridges目录下是服务配置，wasmplugins目录下是插件配置，都跟k8s下的配置一样的

pepesi commented 1 week ago

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.io/backend-protocol: HTTPS
    higress.io/destination: dashscope.aliyuncs.com
    higress.io/proxy-ssl-name: dashscope.dns
    higress.io/proxy-ssl-server-name: "on"
  labels:
    higress.io/resource-definer: higress
  name: dashscope
  namespace: higress-system
spec:
  ingressClassName: higress
  rules:
  - host: qwen.test.com
    http:
      paths:
      - backend:
          resource:
            apiGroup: networking.higress.io
            kind: McpBridge
            name: default
        path: /
        pathType: Prefix
---
apiVersion: networking.higress.io/v1
kind: McpBridge
metadata:
  name: default
  namespace: higress-system
spec:
  registries:
  - domain: dashscope.aliyuncs.com
    name: dashscope
    port: 443
    type: dns

报错400, 从错误日志看，似乎是网关返回的，没到上游去

{"authority":"-","bytes_received":"0","bytes_sent":"11","downstream_local_address":"127.0.0.1:80","downstream_remote_address":"127.0.0.1:51594","duration":"0","istio_policy_status":"-","method":"-","path":"-","protocol":"HTTP/1.1","request_id":"-","requested_server_name":"-","response_code":"400","response_flags":"DPE","route_name":"-","start_time":"2024-06-21T09:57:35.421Z","trace_id":"-","upstream_cluster":"-","upstream_host":"-","upstream_local_address":"-","upstream_service_time":"-","upstream_transport_failure_reason":"-","user_agent":"-","x_forwarded_for":"-"}

johnlanni commented 1 week ago

@pepesi 访问日志发来看看

pepesi commented 1 week ago

添加了header和content-type等，似乎可以了，但是上游503，这个我还是没理解为啥

pepesi commented 1 week ago

johnlanni commented 1 week ago

response flag = DPE 是 downstream protocol 客户端协议错误

response flag = NC 是 No Cluster 对应路由的目标服务发现不了

pepesi commented 1 week ago

现在是UF了…… {"authority":"qwen.test.com","bytes_received":"259","bytes_sent":"91","downstream_local_address":"127.0.0.1:80","downstream_remote_address":"127.0.0.1:33932","duration":"241","istio_policy_status":"-","method":"POST","path":"/v1/chat/completions","protocol":"HTTP/1.1","request_id":"adb9059e-c0a9-425c-a724-f42c050a012f","requested_server_name":"-","response_code":"503","response_flags":"UF","route_name":"dashscope","start_time":"2024-06-21T10:24:45.652Z","trace_id":"d6e782aafb01c02b3bff9c20264866e5","upstream_cluster":"outbound|443||dashscope.dns","upstream_host":"47.93.243.29:443","upstream_local_address":"-","upstream_service_time":"-","upstream_transport_failure_reason":"-","user_agent":"curl/8.6.0","x_forwarded_for":"10.244.1.5"}

pepesi commented 1 week ago

可以了，是我的注解写错了导致的完整如下

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.io/backend-protocol: HTTPS
    higress.io/destination: dashscope.dns
    higress.io/proxy-ssl-name: dashscope.aliyuncs.com
    higress.io/proxy-ssl-server-name: "on"
  labels:
    higress.io/resource-definer: higress
  name: dashscope
  namespace: higress-system
spec:
  ingressClassName: higress
  rules:
  - host: qwen.test.com
    http:
      paths:
      - backend:
          resource:
            apiGroup: networking.higress.io
            kind: McpBridge
            name: default
        path: /
        pathType: Prefix
---
apiVersion: networking.higress.io/v1
kind: McpBridge
metadata:
  name: default
  namespace: higress-system
spec:
  registries:
  - domain: dashscope.aliyuncs.com
    name: dashscope
    port: 443
    type: dns

Suchun-sv commented 1 week ago

想问一下，我基于该环境测试。并且为了对接云服务，封装了http请求之后（用的net/http)，编译不通过，看样子可能是使用协程导致的（如下图）。这个有什么简单的解决方法呢，我看到有tinygo封装的http库，但是似乎需要注册到higress的第三方库列表里比较好对吗？

johnlanni commented 1 week ago

@Suchun-sv 现在 tinygo 编译 wasm 目标还不支持开启schedule，没法支持goroutine

插件里调用外部服务请参考：https://higress.io/zh-cn/docs/user/wasm-go#%E5%9C%A8%E6%8F%92%E4%BB%B6%E4%B8%AD%E8%AF%B7%E6%B1%82%E5%A4%96%E9%83%A8%E6%9C%8D%E5%8A%A1

Suchun-sv commented 1 week ago

@Suchun-sv 现在 tinygo 编译 wasm 目标还不支持开启schedule，没法支持goroutine

插件里调用外部服务请参考：https://higress.io/zh-cn/docs/user/wasm-go#%E5%9C%A8%E6%8F%92%E4%BB%B6%E4%B8%AD%E8%AF%B7%E6%B1%82%E5%A4%96%E9%83%A8%E6%9C%8D%E5%8A%A1

哦哦，明白了，感谢！

2456868764 commented 1 week ago

更新一下 lobe-chat 镜像到最新版本 registry.cn-hangzhou.aliyuncs.com/2456868764/lobe-chat:v1.1.3，避免下载不了 docker.io 镜像

# 创建一个工作目录，以及其下的配置目录
mkdir -p higress/data
# 进入工作目录
cd higress

在工作目录下创建这份 docker-compose.yaml:

version: '3.9'

networks:
  higress-net:
    external: false

services:
  higress:
    image: higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:1.4.1
    environment:
      - CONFIG_TEMPLATE=ai-proxy
      - DEFAULT_AI_SERVICE=qwen
      - DASHSCOPE_API_KEY=YOUR_DASHSCOPE_API_KEY
    networks:
      - higress-net
    ports:
      - "8080:8080/tcp"
      - "8001:8001/tcp"
    volumes:
      - ./data:/data
    restart: always
  lobechat:
    image: registry.cn-hangzhou.aliyuncs.com/2456868764/lobe-chat:v1.1.3
    environment:
      - ACCESS_CODE=lobe66
      - OPENAI_API_KEY=sk-unused
      - OPENAI_PROXY_URL=http://higress:8080/v1
    networks:
      - higress-net
    ports:
      - "3210:3210/tcp"
    restart: always

jaymie9019 commented 1 week ago

我按照上面的教程部署了 higress 以及 lobechat，但是在使用的时候，如果回答的文本过长（其实也没有特比长）就会报错

@johnlanni

johnlanni commented 1 week ago

@jaymie9019 现在higress的默认参数，请求body超过32kb会有问题，可以先参考这个文档，给configmaps目录下的higress-config设置connectionBufferLimits：

后续会在插件内缓存，不用网关的buffer，更灵活一些 cc @CH3CHO

johnlanni commented 1 week ago

@jaymie9019 你可以进容器，看下/var/log/higress/gateway.log的日志确认下，response_code是否是413

jaymie9019 commented 1 week ago

@jaymie9019 你可以进容器，看下/var/log/higress/gateway.log的日志确认下，response_code是否是413

已经配置，并且重启了 higress 的 pod，问题并没有得到解决

日志中的 response_code 正常，仍然是 200

root@higress-all-in-one-9fb5f859d-28btt:/var/log/higress# tail -f gateway.log 
2024-06-23T09:18:35.093157Z     info    Epoch 0 starting
2024-06-23T09:18:35.093204Z     info    Envoy command: [-c etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --drain-strategy immediate --parent-shutdown-time-s 60 --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --log-format [Envoy (Epoch 0)] [%Y-%m-%d %T.%e][%t][%l][%n] %v -l warning --component-log-level misc:error]
2024-06-23T09:18:35.095158Z     info    sds     Starting SDS grpc server
2024-06-23T09:18:35.095387Z     info    starting Http service at 127.0.0.1:15004
2024-06-23T09:18:35.195971Z     info    cache   adding watcher for file certificate etc/certs/cert-chain.pem
2024-06-23T09:18:35.196012Z     info    cache   read certificate from file      resource=default
2024-06-23T09:18:35.196100Z     info    cache   adding watcher for file certificate etc/certs/root-cert.pem
2024-06-23T09:18:35.196110Z     info    cache   read certificate from file      resource=ROOTCA
2024-06-23T09:18:35.391047Z     info    xdsproxy        connected to upstream XDS server: 127.0.0.1:15012
{"authority":"higress-all-in-one","bytes_received":"982","bytes_sent":"17506","downstream_local_address":"172.19.0.18:8080","downstream_remote_address":"172.19.96.54:51340","duration":"16725","istio_policy_status":"-","method":"POST","path":"/v1/chat/completions","protocol":"HTTP/1.1","request_id":"0b43a7fd-52b7-4bbb-b314-83a49e09e44e","requested_server_name":"-","response_code":"200","response_flags":"-","route_name":"qwen","start_time":"2024-06-23T09:18:42.731Z","trace_id":"e2661cbc039358258612805b367eb292","upstream_cluster":"outbound|443||qwen.dns","upstream_host":"47.93.243.29:443","upstream_local_address":"172.19.0.18:51922","upstream_service_time":"565","upstream_transport_failure_reason":"-","user_agent":"sx/JS 4.52.0","x_forwarded_for":"172.19.96.54"}

johnlanni commented 1 week ago

@jaymie9019 给容器加一下这个环境变量：GATEWAY_COMPONENT_LOG_LEVEL=misc:error,wasm:debug

请求后会有日志输出，注意下日志内容脱敏后发一下看看响应

johnlanni commented 1 week ago

@jaymie9019 你的higress镜像是 higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:1.4.1 吗，我试了多次没问题：

Suchun-sv commented 6 days ago

基于这套测试环境，目前每次更新代码需要1.重新编译wasm插件，2.打包并发布docker镜像，3.在web界面删除并重新配置插件是吗？其中第一步和第二步都比较方便自动化，第三步有什么比较好的自动化测试思路吗？

CH3CHO commented 6 days ago

基于这套测试环境，目前每次更新代码需要1.重新编译wasm插件，2.打包并发布docker镜像，3.在web界面删除并重新配置插件是吗？其中第一步和第二步都比较方便自动化，第三步有什么比较好的自动化测试思路吗？

不需要这么麻烦，可以参考这篇文档：https://higress.io/zh-cn/docs/user/wasm-go#%E4%B8%89%E6%9C%AC%E5%9C%B0%E8%B0%83%E8%AF%95

johnlanni commented 6 days ago

基于这套测试环境，目前每次更新代码需要1.重新编译wasm插件，2.打包并发布docker镜像，3.在web界面删除并重新配置插件是吗？其中第一步和第二步都比较方便自动化，第三步有什么比较好的自动化测试思路吗？

higress 目录下有个 wasmplugins 子目录，里面有自己插件对应的 yaml 文件，直接改里面的镜像地址就行

Suchun-sv commented 5 days ago

@Suchun-sv 现在 tinygo 编译 wasm 目标还不支持开启schedule，没法支持goroutine

插件里调用外部服务请参考：https://higress.io/zh-cn/docs/user/wasm-go#%E5%9C%A8%E6%8F%92%E4%BB%B6%E4%B8%AD%E8%AF%B7%E6%B1%82%E5%A4%96%E9%83%A8%E6%9C%8D%E5%8A%A1

老师，这个请求外部服务怎么协调多个顺序的服务呢，以请求 A，请求B，请求C为例，我需要的逻辑是等到请求A结束后再开始请求B。然而ai-cache的示例代码里只给出了包含一个异步请求时候的处理逻辑（例如对redis的处理是在外部暂停返回types.ActionPause, 再在回调函数中根据结果ResumeHttpRequest)，如果多个请求是用信号变量来控制，还是可以在main函数里注册多个函数呢？

johnlanni commented 5 days ago

@Suchun-sv 需要在 A 响应的 callback 里发起请求 B

KagaJiankui commented 5 days ago

请问一下ai-proxy插件能支持类似one-api的多渠道分流吗? 例如在同一个provider下配置多个不同的type和参数映射

johnlanni commented 5 days ago

@KagaJiankui 可以的，这个对于网关来说是通用能力，创建路由通过Header匹配Authorization Header走不同路由就好了，不同的路由可以配不同的ai-proxy配置

KagaJiankui commented 5 days ago

@johnlanni 按照配置文件在windows docker-desktop (wsl2) 上docker-compose, lobechat和higress gateway.log报错如下:

{
  "error": {
    "headers": {
      "content-length": "0",
      "date": "Tue, 25 Jun 2024 11:37:47 GMT",
      "server": "istio-envoy"
    },
    "stack": "Error: 404 status code (no body)\n    at eE.generate (/app/.next/server/edge-chunks/465.js:4:1790)\n    at sx.makeStatusError (/app/.next/server/edge-chunks/465.js:4:14583)\n    at sx.makeRequest (/app/.next/server/edge-chunks/465.js:4:15506)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async Object.chat (/app/.next/server/edge-chunks/305.js:7:7669)\n    at async /app/.next/server/app/api/chat/[provider]/route.js:1:1473\n    at async /app/.next/server/edge-chunks/614.js:6:67058\n    at async $.execute (/app/.next/server/edge-chunks/614.js:6:64768)\n    at async $.handle (/app/.next/server/edge-chunks/614.js:6:68325)\n    at async e_.handler (/app/.next/server/edge-chunks/614.js:7:33025)",
    "status": 404
  },
  "endpoint": "http://higress:****/v1",
  "provider": "openai"
}

{
    "authority": "higress:8080",
    "bytes_received": "929",
    "bytes_sent": "0",
    "downstream_local_address": "172.20.0.3:8080",
    "downstream_remote_address": "172.20.0.2:34694",
    "duration": "10",
    "istio_policy_status": "-",
    "method": "POST",
    "path": "/v1/chat/completions",
    "protocol": "HTTP/1.1",
    "request_id": "a5c006fc-6a11-4492-a0ff-e0dc30b7fd45",
    "requested_server_name": "-",
    "response_code": "404",
    "response_flags": "NR",
    "route_name": "-",
    "start_time": "2024-06-25T11:37:47.550Z",
    "trace_id": "-",
    "upstream_cluster": "-",
    "upstream_host": "-",
    "upstream_local_address": "-",
    "upstream_service_time": "-",
    "upstream_transport_failure_reason": "-",
    "user_agent": "sx/JS 4.52.0",
    "x_forwarded_for": "172.20.0.2"
}

AI代理插件配置如下:

provider:
  apiTokens:
  - "KEY"
  modelMapping:
    '*': "qwen-max"
  type: "qwen"
  qwenEnableSearch: true

johnlanni commented 5 days ago

@KagaJiankui qwen这个路由的配置页面截图看看呢

KagaJiankui commented 5 days ago

(怀疑可能是WSL2的回环地址问题)

johnlanni commented 5 days ago

@KagaJiankui 你用的不是上面的配置吧，上面的配置不会加这个精确匹配的 header，你把这个header匹配条件删除就可以了

你应该没加这个环境变量：DEFAULT_AI_SERVICE=qwen

KagaJiankui commented 5 days ago

可能是因为配置时设置了错误的环境变量, 删除bind mount的本地data目录后重新配置无此header匹配条件

alibaba / higress