ai-proxy + request block 断开连接

007gzs commented 2 months ago

Ⅰ. Issue Description

ai-proxy + request block 客户端请求触发黑名单时直接断开连接

Ⅱ. Describe what happened

请求返回：

curl -vvv http://172.20.5.14:20080/api/openai/v1/chat/completions -d '{"messages": [{"role": "user", "content": "helloworld"}], "model": "gpt-3.5"}' -H "content-type: application/json"
* Uses proxy env variable no_proxy == 'localhost,127.0.0.0/8,::1'
*   Trying 172.20.5.14:20080...
* Connected to 172.20.5.14 (172.20.5.14) port 20080
> POST /api/openai/v1/chat/completions HTTP/1.1
> Host: 172.20.5.14:20080
> User-Agent: curl/8.5.0
> Accept: */*
> content-type: application/json
> Content-Length: 77
> 
< HTTP/1.1 200 OK
< content-type: text/plain
< date: Tue, 23 Jul 2024 07:30:34 GMT
< server: istio-envoy
< transfer-encoding: chunked
< 
* transfer closed with outstanding read data remaining
* Closing connection
curl: (18) transfer closed with outstanding read data remaining

Ⅲ. Describe what you expected to happen

返回状态码403,body内容fail

Ⅳ. How to reproduce it (as minimally and precisely as possible)

配置ai-proxy
request block 配置：

block_bodies:
- "helloworld"
block_bodys:
- "helloworld"
blocked_code: 403
blocked_message: "fail"
case_sensitive: false

Ⅴ. Anything else we need to know?

Ⅵ. Environment:

Higress version: 1.4.0 docker-compse版
OS : ubuntu20.04
Others:

007gzs commented 2 months ago

感觉是返回内容非json导致ai-proxy报错，配置修改成json格式后 403可以通过，但是200就会出问题，下面是在全新的1.4.1上测试的：

403

request block 配置：

block_bodies:
- "test"
block_bodys:
- "test"
blocked_code: 403
blocked_message: "{\"id\":\"from-cache\",\"choices\":[{\"index\":0,\"message\":{\"\
  role\":\"assistant\",\"content\": \"测试\"},\"finish_reason\":\"stop\"}],\"model\"\
  :\"gpt-4o\",\"object\":\"chat.completion\",\"usage\":{\"prompt_tokens\":0,\"completion_tokens\"\
  :0,\"total_tokens\":0}}"
case_sensitive: false

curl:

curl -vvv http://172.20.5.14:20080/api/openai/v1/chat/completions -d '{"messages": [{"role": "user", "content": "test"}], "model": "gpt-3.5"}' -H "content-type: application/json"
*   Trying 172.20.5.14:20080...
* Connected to 172.20.5.14 (172.20.5.14) port 20080
> POST /api/openai/v1/chat/completions HTTP/1.1
> Host: 172.20.5.14:20080
> User-Agent: curl/8.5.0
> Accept: */*
> content-type: application/json
> Content-Length: 71
> 
< HTTP/1.1 403 Forbidden
< content-length: 228
< content-type: text/plain
< date: Tue, 23 Jul 2024 10:43:54 GMT
< server: istio-envoy
< 
* Connection #0 to host 172.20.5.14 left intact
{"id":"from-cache","choices":[{"index":0,"message":{"role":"assistant","content": "测试"},"finish_reason":"stop"}],"model":"gpt-4o","object":"chat.completion","usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}

200

request block 配置：

block_bodies:
- "test"
block_bodys:
- "test"
blocked_code: 200
blocked_message: "{\"id\":\"from-cache\",\"choices\":[{\"index\":0,\"message\":{\"\
  role\":\"assistant\",\"content\": \"测试\"},\"finish_reason\":\"stop\"}],\"model\"\
  :\"gpt-4o\",\"object\":\"chat.completion\",\"usage\":{\"prompt_tokens\":0,\"completion_tokens\"\
  :0,\"total_tokens\":0}}"
case_sensitive: false

curl:

curl -vvv http://172.20.5.14:20080/api/openai/v1/chat/completions -d '{"messages": [{"role": "user", "content": "test"}], "model": "gpt-3.5"}' -H "content-type: application/json"
*   Trying 172.20.5.14:20080...
* Connected to 172.20.5.14 (172.20.5.14) port 20080
> POST /api/openai/v1/chat/completions HTTP/1.1
> Host: 172.20.5.14:20080
> User-Agent: curl/8.5.0
> Accept: */*
> content-type: application/json
> Content-Length: 71
> 
< HTTP/1.1 200 OK
< content-type: text/plain
< date: Tue, 23 Jul 2024 10:44:09 GMT
< server: istio-envoy
< transfer-encoding: chunked
< 
* Connection #0 to host 172.20.5.14 left intact
{"choices":[],"created":1721731450,"object":"chat.completion","usage":{}}

CH3CHO commented 2 months ago

那应该降低ai-proxy的优先级，让request_block先执行吧

007gzs commented 2 months ago

状态码403返回符合期望，200就不符合期望感觉不是优先级的问题。目前 ai-proxy priority: "100" request_block priority: "320"

CH3CHO commented 2 months ago

状态码403返回符合期望，200就不符合期望感觉不是优先级的问题。目前 ai-proxy priority: "100" request_block priority: "320"

不能单看优先级数值，还要看阶段的

007gzs commented 2 months ago

request-block:

apiVersion: extensions.higress.io/v1alpha1
kind: WasmPlugin
metadata:
  annotations:
    higress.io/wasm-plugin-description: 基于 URI、请求头等特征屏蔽 HTTP 请求，可以用于防护部分站点资源不对外部暴露
    higress.io/wasm-plugin-icon: https://img.alicdn.com/imgextra/i1/O1CN01jKT9vC1O059vNaq5u_!!6000000001642-2-tps-42-42.png
    higress.io/wasm-plugin-title: Request Block
  creationTimestamp: "2024-07-23T08:03:08Z"
  labels:
    higress.io/resource-definer: higress
    higress.io/wasm-plugin-built-in: "true"
    higress.io/wasm-plugin-category: security
    higress.io/wasm-plugin-name: request-block
    higress.io/wasm-plugin-version: 1.0.0
  managedFields:
  - apiVersion: extensions.higress.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:higress.io/wasm-plugin-description: {}
          f:higress.io/wasm-plugin-icon: {}
          f:higress.io/wasm-plugin-title: {}
        f:labels:
          .: {}
          f:higress.io/resource-definer: {}
          f:higress.io/wasm-plugin-built-in: {}
          f:higress.io/wasm-plugin-category: {}
          f:higress.io/wasm-plugin-name: {}
          f:higress.io/wasm-plugin-version: {}
      f:spec:
        f:defaultConfig:
          .: {}
          f:block_bodies: {}
          f:block_bodys: {}
          f:blocked_code: {}
          f:blocked_message: {}
          f:case_sensitive: {}
        f:matchRules: {}
        f:phase: {}
        f:priority: {}
        f:url: {}
    manager: Kubernetes Java Client
    operation: Update
    time: "2024-07-23T10:44:03Z"
  name: request-block-1.0.0
  namespace: higress-system
spec:
  defaultConfig:
    block_bodies:
    - test
    block_bodys:
    - test
    blocked_code: 200
    blocked_message: '{"id":"from-cache","choices":[{"index":0,"message":{"role":"assistant","content":
      "测试"},"finish_reason":"stop"}],"model":"gpt-4o","object":"chat.completion","usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}'
    case_sensitive: false
  matchRules: []
  phase: AUTHZ
  priority: "320"
  url: oci://higress-registry.cn-hangzhou.cr.aliyuncs.com/plugins/request-block:1.0.0
status: {}

ai-proxy:

apiVersion: extensions.higress.io/v1alpha1
kind: WasmPlugin
metadata:
  annotations:
    higress.io/wasm-plugin-description: 通过AI助手提供智能对话服务
    higress.io/wasm-plugin-icon: https://img.alicdn.com/imgextra/i1/O1CN018iKKih1iVx287RltL_!!6000000004419-2-tps-42-42.png
    higress.io/wasm-plugin-title: AI代理
  creationTimestamp: "2024-07-23T10:14:38Z"
  labels:
    higress.io/resource-definer: higress
    higress.io/wasm-plugin-built-in: "true"
    higress.io/wasm-plugin-category: custom
    higress.io/wasm-plugin-name: ai-proxy
    higress.io/wasm-plugin-version: 1.0.0
  managedFields:
  - apiVersion: extensions.higress.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:higress.io/wasm-plugin-description: {}
          f:higress.io/wasm-plugin-icon: {}
          f:higress.io/wasm-plugin-title: {}
        f:labels:
          .: {}
          f:higress.io/resource-definer: {}
          f:higress.io/wasm-plugin-built-in: {}
          f:higress.io/wasm-plugin-category: {}
          f:higress.io/wasm-plugin-name: {}
          f:higress.io/wasm-plugin-version: {}
      f:spec:
        f:defaultConfig:
          .: {}
          f:provider:
            .: {}
            f:apiTokens: {}
            f:modelMapping:
              .: {}
              f:*: {}
            f:qwenFileIds: {}
            f:type: {}
        f:matchRules: {}
        f:priority: {}
        f:url: {}
    manager: Kubernetes Java Client
    operation: Update
    time: "2024-07-23T10:14:38Z"
  name: ai-proxy-1.0.0
  namespace: higress-system
spec:
  defaultConfig:
    provider:
      apiTokens:
      - sk-**
      modelMapping:
        '*': qwen-long
      qwenFileIds:
      - file-fe-**
      type: qwen
  matchRules: []
  priority: "100"
  url: oci://higress-registry.cn-hangzhou.cr.aliyuncs.com/plugins/ai-proxy:1.0.0
status: {}

johnlanni commented 2 months ago

@007gzs 现在还有问题吗，之前开源镜像仓库里的request block插件有问题，现在我更新了，我本地测没发现问题

007gzs commented 2 months ago

@007gzs 现在还有问题吗，之前开源镜像仓库里的request block插件有问题，现在我更新了，我本地测没发现问题

./bin/shutdown.sh && ./bin/startup.sh 重启之后200还是返回{"choices":[],"created":1721731450,"object":"chat.completion","usage":{}}

johnlanni commented 2 months ago

@007gzs request block如果拦截了请求是不会到 ai-proxy 插件的，你是不是网关上还配了其他插件呀

007gzs commented 2 months ago

block_bodies:

"test" block_bodys:

"test" blocked_code: 200 blocked_message: "{\"id\":\"from-cache\",\"choices\":[{\"index\":0,\"message\":{\"\ role\":\"assistant\",\"content\": \"测试\"},\"finish_reason\":\"stop\"}],\"model\"\ :\"gpt-4o\",\"object\":\"chat.completion\",\"usage\":{\"prompt_tokens\":0,\"completion_tokens\"\ :0,\"total_tokens\":0}}" case_sensitive: false

没有，新部的

johnlanni commented 2 months ago

进容器执行 curl localhost:15000/config_dump 发来看下，注意内容脱敏

johnlanni commented 2 months ago

@007gzs 新部署的如果用了老的data目录，也会有你之前的插件配置

007gzs commented 2 months ago

@007gzs 新部署的如果用了老的data目录，也会有你之前的插件配置

把老的停了新目录里 get-higress.sh

johnlanni commented 2 months ago

现在插件的机制是，只要经过插件的 onRequestHeader阶段，就会经过 onResponseHeader 和 onResponseData 阶段，这里 request-block 插件自定义的响应不是通义千问的协议，所以进过 ai proxy 插件后转换协议失败，返回了上面的响应。

这块可以优化一下，ai proxy 插件在 onResponseHeader 阶段判断 response code detail：

        codeDetails, _ := proxywasm.GetProperty([]string{"response", "code_details"})
        if  codeDetails != “via_upstream” {
                // 设置标记，确保ai proxy跳过对header和body的处理
        }

当 codeDetails 不是 via_upstream 时，表示响应不是后端返回的，ai proxy 不需要进行处理

alibaba / higress