search

alibaba / higress

🤖 AI Gateway | AI Native API Gateway
https://higress.io
Apache License 2.0
2.91k stars 478 forks source link

Config key-cluster-rate-limit plugin failed. #1104

Closed jaggerwang closed 3 months ago

jaggerwang commented 3 months ago

Ⅰ. Issue Description

Config key-cluster-rate-limit plugin as following to limit query per second to 1 time/s, but still can query more than one time in one second.

apiVersion: extensions.higress.io/v1alpha1
kind: WasmPlugin
metadata:
  name: key-cluster-rate-limit
spec:
  url: oci://higress-registry.cn-hangzhou.cr.aliyuncs.com/plugins/key-cluster-rate-limit:1.0.0
  matchRules:
  - domain:
    - open-dev-cloud.alidev.beisai.com
    config:
      rule_name: open_api_rate_limit
      rule_items:
      - limit_by_per_header: x-api-key
        limit_keys:
        - key: "*"
          query_per_second: 1
      show_limit_quota_header: true
      rejected_code: 429
      redis:
        service_name: basicai-redis-master.redis-cluster.svc.cluster.local
        service_port: 6379
        username:
        password: ******
        timeout: 1000

Ⅱ. Describe what happened

higress-controller log

2024-07-09T06:36:18.917374Z info    ads ADS: "10.101.0.15:58440" higress-gateway-5dcb964948-mdfng.higress-system-475 terminated rpc error: code = Canceled desc = context canceled
2024-07-09T06:36:19.027525Z info    ads ADS: new connection for node:higress-gateway-5dcb964948-mdfng.higress-system-476
2024-07-09T14:36:19.027677419+08:00 2024-07-09T06:36:19.027627Z info    ads CDS: PUSH request for node:higress-gateway-5dcb964948-mdfng.higress-system resources:2 size:379B cached:1/1
2024-07-09T06:36:19.028055Z info    ads LDS: PUSH request for node:higress-gateway-5dcb964948-mdfng.higress-system resources:2 size:8.1kB
2024-07-09T14:36:19.028137562+08:00 2024-07-09T06:36:19.028087Z info    ads EDS: PUSH request for node:higress-gateway-5dcb964948-mdfng.higress-system resources:1 size:179B empty:0 cached:1/1
2024-07-09T06:36:19.028128Z info    ads SRDS: PUSH request for node:higress-gateway-5dcb964948-mdfng.higress-system resources:2 size:263B
2024-07-09T14:36:19.029548688+08:00 2024-07-09T06:36:19.029495Z info    ads SDS: PUSH request for node:higress-gateway-5dcb964948-mdfng.higress-system resources:1 size:5.4kB cached:1/1
2024-07-09T14:36:19.029601724+08:00 2024-07-09T06:36:19.029562Z info    ads RDS: PUSH request for node:higress-gateway-5dcb964948-mdfng.higress-system resources:2 size:1.5kB cached:2/2
2024-07-09T14:36:37.135088641+08:00 2024-07-09T06:36:37.134982Z info    ads Push debounce stable[8344] 1 for config ServiceEntry/default/kubelet.default.svc.cluster.local: 100.863736ms since last change, 100.863575ms since last push, full=false
2024-07-09T06:36:37.135027Z info    ads XDS: Incremental Pushing:2024-07-09T01:42:23Z/37 ConnectedEndpoints:1 Version:2024-07-09T01:42:23Z/37
2024-07-09T06:37:00.533316Z info    ads Push debounce stable[8345] 1 for config ServiceEntry/default/kubelet.default.svc.cluster.local: 100.179104ms since last change, 100.178932ms since last push, full=false
2024-07-09T14:37:00.533470129+08:00 2024-07-09T06:37:00.533367Z info    ads XDS: Incremental Pushing:2024-07-09T01:42:23Z/37 ConnectedEndpoints:1 Version:2024-07-09T01:42:23Z/37
2024-07-09T14:40:48.385706997+08:00 2024-07-09T06:40:48.385636Z info    adsc    Received 127.0.0.1:15051 type networking.istio.io/v1alpha3/WorkloadEntry cnt=0 nonce=04ae7a08-01fb-45ab-b4b8-eb34b5bf996c
2024-07-09T06:40:48.385693Z info    adsc    Received 127.0.0.1:15051 type security.istio.io/v1beta1/AuthorizationPolicy cnt=0 nonce=a385c8f0-2e47-41bc-9e6c-74bf9173656f
2024-07-09T06:40:48.385716Z info    adsc    Received 127.0.0.1:15051 type telemetry.istio.io/v1alpha1/Telemetry cnt=0 nonce=b604a8a7-5e4f-420e-ab88-77d457779b86
2024-07-09T14:40:48.385815102+08:00 2024-07-09T06:40:48.385736Z info    adsc    Received 127.0.0.1:15051 type core/v1alpha1/MeshConfig cnt=0 nonce=c891c5e8-f70a-424b-b823-75ae6c9219af
2024-07-09T06:40:48.385750Z info    adsc    Received 127.0.0.1:15051 type networking.istio.io/v1alpha3/DestinationRule cnt=0 nonce=03da12f3-1009-477b-9b92-4dd42c650937
2024-07-09T06:40:48.385771Z info    adsc    Received 127.0.0.1:15051 type networking.istio.io/v1alpha3/ServiceEntry cnt=0 nonce=1267351f-b6f6-4ea0-9c24-1aae2f950117
2024-07-09T06:40:48.385787Z info    adsc    Received 127.0.0.1:15051 type networking.istio.io/v1alpha3/ServiceSubscriptionList cnt=0 nonce=b9091853-26bf-442e-9897-150c5395ec04
2024-07-09T06:40:48.385874Z info    adsc    Received 127.0.0.1:15051 type networking.istio.io/v1alpha3/VirtualService cnt=1 nonce=6cac0cb4-2bb2-4a84-a600-01b7e8290bd2
2024-07-09T06:40:48.385985Z info    adsc    Received 127.0.0.1:15051 type networking.istio.io/v1alpha3/Sidecar cnt=0 nonce=23621178-a261-47fd-b94b-684afcb9792a
2024-07-09T06:40:48.386006Z info    adsc    Received 127.0.0.1:15051 type security.istio.io/v1beta1/PeerAuthentication cnt=0 nonce=00a67bb8-a01f-4e6b-9860-2dcf7e5b8866
2024-07-09T06:40:48.386022Z info    adsc    Received 127.0.0.1:15051 type security.istio.io/v1beta1/RequestAuthentication cnt=0 nonce=e6e81be3-0360-4144-98c7-00b66045ce4f
2024-07-09T06:40:48.386710Z info    adsc    Received 127.0.0.1:15051 type extensions.istio.io/v1alpha1/WasmPlugin cnt=1 nonce=6588db96-0b20-4364-bcdd-652f507c28e7
2024-07-09T06:40:48.396781Z info    adsc    Received 127.0.0.1:15051 type networking.istio.io/v1alpha3/EnvoyFilter cnt=1 nonce=25494d60-9114-4c46-b9e9-c04b5f50695a
2024-07-09T06:40:48.396844Z info    ads full push happen, reason:[config]
2024-07-09T06:40:48.396873Z info    adsc    Received 127.0.0.1:15051 type networking.istio.io/v1alpha3/Gateway cnt=1 nonce=4e703376-792e-4731-8000-5141e85005b3
2024-07-09T06:40:48.396918Z info    adsc    Received 127.0.0.1:15051 type networking.istio.io/v1alpha3/WorkloadGroup cnt=0 nonce=afd11f5e-c8c6-4ef9-80cc-77bd23826cbe
2024-07-09T06:40:48.498018Z info    ads Push debounce stable[8346] 1 for config WasmPlugin/higress-system/key-cluster-rate-limit: 100.721986ms since last change, 100.721609ms since last push, full=true
2024-07-09T14:40:48.508903712+08:00 2024-07-09T06:40:48.508845Z info    ads XDS: Pushing:2024-07-09T06:40:48Z/38 Services:178 ConnectedEndpoints:1  Version:2024-07-09T06:40:48Z/38
2024-07-09T06:40:48.509009Z info    ads CDS: PUSH for node:higress-gateway-5dcb964948-mdfng.higress-system resources:2 size:379B cached:1/1
2024-07-09T06:40:48.509042Z info    ads EDS: PUSH for node:higress-gateway-5dcb964948-mdfng.higress-system resources:1 size:179B empty:0 cached:1/1
2024-07-09T06:40:48.515634Z info    ads LDS: PUSH for node:higress-gateway-5dcb964948-mdfng.higress-system resources:2 size:8.7kB
2024-07-09T14:40:48.515730141+08:00 2024-07-09T06:40:48.515681Z info    ads SRDS: PUSH for node:higress-gateway-5dcb964948-mdfng.higress-system resources:2 size:263B
2024-07-09T06:40:48.515721Z info    ads RDS: PUSH for node:higress-gateway-5dcb964948-mdfng.higress-system resources:2 size:1.5kB cached:2/2
2024-07-09T06:40:48.536860Z info    ads ECDS: PUSH request for node:higress-gateway-5dcb964948-mdfng.higress-system resources:1 size:850B
2024-07-09T06:40:48.846056Z warn    ads ADS:ECDS: ACK ERROR higress-gateway-5dcb964948-mdfng.higress-system-476 Internal:Proto constraint validation failed (WasmValidationError.Config: embedded message failed validation | caused by PluginConfigValidationError.VmConfig: embedded message failed validation | caused by VmConfigValidationError.Code: embedded message failed validation | caused by AsyncDataSourceValidationError.Remote: embedded message failed validation | caused by RemoteDataSourceValidationError.Sha256: value length must be at least 1 characters): config {
2024-07-09T14:40:48.846191171+08:00   name: "higress-system.key-cluster-rate-limit"
  vm_config {
    runtime: "envoy.wasm.runtime.v8"
    code {
      remote {
2024-07-09T14:40:48.846206876+08:00         http_uri {
2024-07-09T14:40:48.846211374+08:00           uri: "oci://higress-registry.cn-hangzhou.cr.aliyuncs.com/plugins/key-cluster-rate-limit:1.0.0"
2024-07-09T14:40:48.846214896+08:00           cluster: "_"
2024-07-09T14:40:48.846218488+08:00           timeout {
2024-07-09T14:40:48.846222198+08:00             seconds: 30
2024-07-09T14:40:48.846225772+08:00           }
        }
      }
    }
  }
  configuration {
    [type.googleapis.com/google.protobuf.StringValue] {
      value: "{\"_rules_\":[{\"_match_domain_\":[\"open-dev-cloud.alidev.beisai.com\"],\"redis\":{\"password\":\"******\",\"service_name\":\"basicai-redis-master.redis-cluster.svc.cluster.local\",\"service_port\":6379,\"timeout\":1000,\"username\":null},\"rejected_code\":429,\"rule_items\":[{\"limit_by_per_header\":\"x-api-key\",\"limit_keys\":[{\"key\":\"*\",\"query_per_second\":1},{\"key\":\"*\",\"query_per_minute\":3000},{\"key\":\"*\",\"query_per_hour\":72000},{\"key\":\"*\",\"query_per_day\":864000}]}],\"rule_name\":\"open_api_rate_limit\",\"show_limit_quota_header\":true}]}"
2024-07-09T14:40:48.846249286+08:00     }
  }
2024-07-09T14:40:48.846263346+08:00   fail_open: true
}

2024-07-09T06:41:49.462557Z info    rootcertrotator Check and rotate root cert.
2024-07-09T14:41:49.467395046+08:00 2024-07-09T06:41:49.467331Z info    rootcertrotator Root cert is not about to expire, skipping root cert rotation.
2024-07-09T06:42:44.957263Z info    ads Push debounce stable[8347] 3 for config ServiceEntry/default/kubelet.default.svc.cluster.local: 100.4699ms since last change, 169.241417ms since last push, full=false
2024-07-09T14:42:44.957392320+08:00 2024-07-09T06:42:44.957331Z info    ads XDS: Incremental Pushing:2024-07-09T06:40:48Z/38 ConnectedEndpoints:1 Version:2024-07-09T06:40:48Z/38

higress-gateway log

{"authority":"-","bytes_received":"0","bytes_sent":"11","downstream_local_address":"10.101.0.15:80","downstream_remote_address":"192.168.3.82:38733","duration":"0","istio_policy_status":"-","method":"-","path":"-","protocol":"HTTP/1.1","request_id":"-","requested_server_name":"-","response_code":"400","response_flags":"DPE","route_name":"-","start_time":"2024-07-09T06:34:19.925Z","trace_id":"-","upstream_cluster":"-","upstream_host":"-","upstream_local_address":"-","upstream_service_time":"-","upstream_transport_failure_reason":"-","user_agent":"-","x_forwarded_for":"-"}
2024-07-09T14:36:19.024415055+08:00 2024-07-09T06:36:19.024298Z info    xdsproxy    connected to upstream XDS server: higress-controller.higress-system.svc:15012
2024-07-09T06:40:48.831897Z error   wasm    cannot fetch Wasm module oci://higress-registry.cn-hangzhou.cr.aliyuncs.com/plugins/key-cluster-rate-limit:1.0.0: could not fetch Wasm OCI image: could not fetch image: GET https://higress-registry.cn-hangzhou.cr.aliyuncs.com/v2/plugins/key-cluster-rate-limit/manifests/1.0.0: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:plugins/key-cluster-rate-limit Type:repository]]
2024-07-09T14:40:48.845498292+08:00 [Envoy (Epoch 0)] [2024-07-09 06:40:48.844][29][warning][config] gRPC config for type.googleapis.com/envoy.config.core.v3.TypedExtensionConfig rejected: Proto constraint validation failed (WasmValidationError.Config: embedded message failed validation | caused by PluginConfigValidationError.VmConfig: embedded message failed validation | caused by VmConfigValidationError.Code: embedded message failed validation | caused by AsyncDataSourceValidationError.Remote: embedded message failed validation | caused by RemoteDataSourceValidationError.Sha256: value length must be at least 1 characters): config {
  name: "higress-system.key-cluster-rate-limit"
2024-07-09T14:40:48.845541584+08:00   vm_config {
    runtime: "envoy.wasm.runtime.v8"
2024-07-09T14:40:48.845548852+08:00     code {
      remote {
        http_uri {
          uri: "oci://higress-registry.cn-hangzhou.cr.aliyuncs.com/plugins/key-cluster-rate-limit:1.0.0"
          cluster: "_"
2024-07-09T14:40:48.845578054+08:00           timeout {
            seconds: 30
          }
        }
2024-07-09T14:40:48.845594158+08:00       }
    }
  }
2024-07-09T14:40:48.845605916+08:00   configuration {
    [type.googleapis.com/google.protobuf.StringValue] {
      value: "{\"_rules_\":[{\"_match_domain_\":[\"open-dev-cloud.alidev.beisai.com\"],\"redis\":{\"password\":\"******\",\"service_name\":\"basicai-redis-master.redis-cluster.svc.cluster.local\",\"service_port\":6379,\"timeout\":1000,\"username\":null},\"rejected_code\":429,\"rule_items\":[{\"limit_by_per_header\":\"x-api-key\",\"limit_keys\":[{\"key\":\"*\",\"query_per_second\":1},{\"key\":\"*\",\"query_per_minute\":3000},{\"key\":\"*\",\"query_per_hour\":72000},{\"key\":\"*\",\"query_per_day\":864000}]}],\"rule_name\":\"open_api_rate_limit\",\"show_limit_quota_header\":true}]}"
    }
2024-07-09T14:40:48.845631025+08:00   }
2024-07-09T14:40:48.845634408+08:00   fail_open: true
}

{"authority":"47.103.200.40:80","bytes_received":"0","bytes_sent":"0","downstream_local_address":"10.101.0.15:80","downstream_remote_address":"192.168.3.77:41221","duration":"0","istio_policy_status":"-","method":"GET","path":"/","protocol":"HTTP/1.1","request_id":"39b08ed3-861e-4a36-a125-631e5be41d4b","requested_server_name":"-","response_code":"404","response_flags":"NR","route_name":"-","start_time":"2024-07-09T06:44:14.609Z","trace_id":"-","upstream_cluster":"-","upstream_host":"-","upstream_local_address":"-","upstream_service_time":"-","upstream_transport_failure_reason":"-","user_agent":"-","x_forwarded_for":"192.168.3.77"}
2024-07-09T14:45:45.919696065+08:00 {"authority":"open-dev-cloud.alidev.beisai.com","bytes_received":"0","bytes_sent":"8","downstream_local_address":"10.101.0.15:443","downstream_remote_address":"192.168.2.249:21488","duration":"2","istio_policy_status":"-","method":"GET","path":"/api/v1/user/test/test1","protocol":"HTTP/2","request_id":"afadb850-3d36-4a7d-8f10-4632319f78e5","requested_server_name":"open-dev-cloud.alidev.beisai.com","response_code":"200","response_flags":"-","route_name":"basicai-backend/user-open","start_time":"2024-07-09T06:45:45.616Z","trace_id":"5519d9920f3e865a92f9518d097e05e6","upstream_cluster":"outbound|80||user.basicai-backend.svc.cluster.local","upstream_host":"10.101.0.84:8080","upstream_local_address":"10.101.0.15:35540","upstream_service_time":"1","upstream_transport_failure_reason":"-","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36","x_forwarded_for":"192.168.2.249"}

It seems that pull key-cluster-rate-limit image failed, but there is no this plugin's address at doc Introduction to the use of Wasm plugins.

Ⅵ. Environment:

cr7258 commented 3 months ago

2024-07-09T06:40:48.831897Z error wasm cannot fetch Wasm module oci://higress-registry.cn-hangzhou.cr.aliyuncs.com/plugins/key-cluster-rate-limit:1.0.0: could not fetch Wasm OCI image: could not fetch image: GET https://higress-registry.cn-hangzhou.cr.aliyuncs.com/v2/plugins/key-cluster-rate-limit/manifests/1.0.0: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:plugins/key-cluster-rate-limit Type:repository]]

应该是这个镜像在仓库中还没有。可以参考这里手动构建一下镜像：https://higress.io/zh-cn/docs/user/wasm-image-spec#4-%E9%95%9C%E5%83%8F%E6%9E%84%E5%BB%BA%E6%96%B9%E5%BC%8F

cr7258 commented 3 months ago

@CH3CHO 自动构建镜像的 pr 已经合并了 https://github.com/alibaba/higress/pull/1069 可以触发把现有插件的镜像都构建一下吗？

CH3CHO commented 3 months ago

@CH3CHO 自动构建镜像的 pr 已经合并了 #1069 可以触发把现有插件的镜像都构建一下吗？

我先针对性的构建一下 key-cluster-rate-limit 吧。

CH3CHO commented 3 months ago

@CH3CHO 自动构建镜像的 pr 已经合并了 #1069 可以触发把现有插件的镜像都构建一下吗？

这个插件我看 main 分支里还没有？

https://github.com/alibaba/higress/tree/main/plugins/wasm-go/extensions

cr7258 commented 3 months ago

@CH3CHO https://github.com/alibaba/higress/tree/main/plugins/wasm-go/extensions/cluster-key-rate-limit

CH3CHO commented 3 months ago

@CH3CHO https://github.com/alibaba/higress/tree/main/plugins/wasm-go/extensions/cluster-key-rate-limit

好吧。那打出来的镜像也会叫 cluster-key-rate-limit 的。

image

cr7258 commented 3 months ago

我等下把文档改成 cluster-key-rate-limit

jaggerwang commented 3 months ago

另外请问一下 key-rate-limit 和 cluster-key-rate-limit 两个插件的区别是什么，文档里没有相关介绍。

hanxiantao commented 3 months ago

我等下把文档改成 cluster-key-rate-limit

https://higress.io/zh-cn/docs/plugins/traffic/cluster-key-rate-limit hi，我看你这个文档的内容之前调整过了，麻烦也帮忙同步过来吧，感谢

hanxiantao commented 3 months ago

另外请问一下 key-rate-limit 和 cluster-key-rate-limit 两个插件的区别是什么，文档里没有相关介绍。

key-rate-limit 插件针对的是单机限流，cluster-key-rate-limit 插件针对的是集群限流，https://mp.weixin.qq.com/s/7yaN1wQSQC0JPdUenE53lg 这篇文章中有相关介绍，可以看下

cr7258 commented 3 months ago

hi，我看你这个文档的内容之前调整过了，麻烦也帮忙同步过来吧，感谢

ok

CH3CHO commented 3 months ago

hi，我看你这个文档的内容之前调整过了，麻烦也帮忙同步过来吧，感谢

ok

建议在文档里注明一下 cluster-key-rate-limit 与 key-rate-limit 的区别。

CH3CHO commented 3 months ago

cluster-key-rate-limit

镜像已经 push 上去了：higress-registry.cn-hangzhou.cr.aliyuncs.com/plugins/cluster-key-rate-limit:1.0.0

@jaggerwang @cr7258 @hanxiantao

jaggerwang commented 3 months ago

插件连接 Redis 失败：

[Envoy (Epoch 0)] [2024-07-10 03:58:11.453][29][critical][wasm] wasm log: failed to init redis: error status returned by host: bad argument
2024-07-10T11:58:11.453791194+08:00 [Envoy (Epoch 0)] [2024-07-10 03:58:11.453][29][warning][wasm] wasm log: [cluster-key-rate-limit] parse rule config failed: error status returned by host: bad argument
[Envoy (Epoch 0)] [2024-07-10 03:58:11.453][29][error][wasm] Wasm VM failed Failed to configure base Wasm plugin
[Envoy (Epoch 0)] [2024-07-10 03:58:11.455][29][warning][config] gRPC config for type.googleapis.com/envoy.config.core.v3.TypedExtensionConfig rejected: Unable to create Wasm HTTP filter higress-system.cluster-key-rate-limit
[Envoy (Epoch 0)] [2024-07-10 03:58:11.455][29][warning][config] gRPC config for type.googleapis.com/envoy.config.core.v3.TypedExtensionConfig rejected: Unable to create Wasm HTTP filter higress-system.cluster-key-rate-limit
[Envoy (Epoch 0)] [2024-07-10 03:58:41.246][29][critical][wasm] wasm log: failed to init redis: error status returned by host: bad argument
2024-07-10T11:58:41.246939056+08:00 [Envoy (Epoch 0)] [2024-07-10 03:58:41.246][29][warning][wasm] wasm log: [cluster-key-rate-limit] parse rule config failed: error status returned by host: bad argument
2024-07-10T11:58:41.246944224+08:00 [Envoy (Epoch 0)] [2024-07-10 03:58:41.246][29][error][wasm] Wasm VM failed Failed to configure base Wasm plugin
2024-07-10T11:58:41.248254058+08:00 [Envoy (Epoch 0)] [2024-07-10 03:58:41.248][29][warning][config] gRPC config for type.googleapis.com/envoy.config.core.v3.TypedExtensionConfig rejected: Unable to create Wasm HTTP filter higress-system.cluster-key-rate-limit
2024-07-10T12:01:58.133559420+08:00 [Envoy (Epoch 0)] [2024-07-10 04:01:58.133][29][critical][wasm] wasm log: failed to init redis: error status returned by host: bad argument
2024-07-10T12:01:58.133594104+08:00 [Envoy (Epoch 0)] [2024-07-10 04:01:58.133][29][warning][wasm] wasm log: [cluster-key-rate-limit] parse rule config failed: error status returned by host: bad argument
[Envoy (Epoch 0)] [2024-07-10 04:01:58.133][29][error][wasm] Wasm VM failed Failed to configure base Wasm plugin
2024-07-10T12:01:58.135050855+08:00 [Envoy (Epoch 0)] [2024-07-10 04:01:58.134][29][warning][config] gRPC config for type.googleapis.com/envoy.config.core.v3.TypedExtensionConfig rejected: Unable to create Wasm HTTP filter higress-system.cluster-key-rate-limit

Redis 配置如下，使用 redis-cli 命令确认过地址、端口和密码都是正确的（注意无用户名）。

apiVersion: extensions.higress.io/v1alpha1
kind: WasmPlugin
metadata:
  name: cluster-key-rate-limit
spec:
  url: oci://higress-registry.cn-hangzhou.cr.aliyuncs.com/plugins/cluster-key-rate-limit:1.0.0
  matchRules:
  - domain:
    - open-dev-cloud.alidev.beisai.com
    - open-test-cloud.alidev.beisai.com
    config:
      rule_name: open_api_rate_limit
      rule_items:
      - limit_by_per_header: x-api-key
        limit_keys:
        - key: "*"
          query_per_second: 1
      show_limit_quota_header: true
      rejected_code: 429
      redis:
        service_name: basicai-redis-master.redis-cluster.svc.cluster.local
        service_port: 6379
        password: ******

即便配置 username 为空也不行。

hanxiantao commented 3 months ago

插件连接 Redis 失败：

[Envoy (Epoch 0)] [2024-07-10 03:58:11.453][29][critical][wasm] wasm log: failed to init redis: error status returned by host: bad argument
2024-07-10T11:58:11.453791194+08:00 [Envoy (Epoch 0)] [2024-07-10 03:58:11.453][29][warning][wasm] wasm log: [cluster-key-rate-limit] parse rule config failed: error status returned by host: bad argument
[Envoy (Epoch 0)] [2024-07-10 03:58:11.453][29][error][wasm] Wasm VM failed Failed to configure base Wasm plugin
[Envoy (Epoch 0)] [2024-07-10 03:58:11.455][29][warning][config] gRPC config for type.googleapis.com/envoy.config.core.v3.TypedExtensionConfig rejected: Unable to create Wasm HTTP filter higress-system.cluster-key-rate-limit
[Envoy (Epoch 0)] [2024-07-10 03:58:11.455][29][warning][config] gRPC config for type.googleapis.com/envoy.config.core.v3.TypedExtensionConfig rejected: Unable to create Wasm HTTP filter higress-system.cluster-key-rate-limit
[Envoy (Epoch 0)] [2024-07-10 03:58:41.246][29][critical][wasm] wasm log: failed to init redis: error status returned by host: bad argument
2024-07-10T11:58:41.246939056+08:00 [Envoy (Epoch 0)] [2024-07-10 03:58:41.246][29][warning][wasm] wasm log: [cluster-key-rate-limit] parse rule config failed: error status returned by host: bad argument
2024-07-10T11:58:41.246944224+08:00 [Envoy (Epoch 0)] [2024-07-10 03:58:41.246][29][error][wasm] Wasm VM failed Failed to configure base Wasm plugin
2024-07-10T11:58:41.248254058+08:00 [Envoy (Epoch 0)] [2024-07-10 03:58:41.248][29][warning][config] gRPC config for type.googleapis.com/envoy.config.core.v3.TypedExtensionConfig rejected: Unable to create Wasm HTTP filter higress-system.cluster-key-rate-limit
2024-07-10T12:01:58.133559420+08:00 [Envoy (Epoch 0)] [2024-07-10 04:01:58.133][29][critical][wasm] wasm log: failed to init redis: error status returned by host: bad argument
2024-07-10T12:01:58.133594104+08:00 [Envoy (Epoch 0)] [2024-07-10 04:01:58.133][29][warning][wasm] wasm log: [cluster-key-rate-limit] parse rule config failed: error status returned by host: bad argument
[Envoy (Epoch 0)] [2024-07-10 04:01:58.133][29][error][wasm] Wasm VM failed Failed to configure base Wasm plugin
2024-07-10T12:01:58.135050855+08:00 [Envoy (Epoch 0)] [2024-07-10 04:01:58.134][29][warning][config] gRPC config for type.googleapis.com/envoy.config.core.v3.TypedExtensionConfig rejected: Unable to create Wasm HTTP filter higress-system.cluster-key-rate-limit

Redis 配置如下，使用 redis-cli 命令确认过地址、端口和密码都是正确的（注意无用户名）。

apiVersion: extensions.higress.io/v1alpha1
kind: WasmPlugin
metadata:
  name: cluster-key-rate-limit
spec:
  url: oci://higress-registry.cn-hangzhou.cr.aliyuncs.com/plugins/cluster-key-rate-limit:1.0.0
  matchRules:
  - domain:
    - open-dev-cloud.alidev.beisai.com
    - open-test-cloud.alidev.beisai.com
    config:
      rule_name: open_api_rate_limit
      rule_items:
      - limit_by_per_header: x-api-key
        limit_keys:
        - key: "*"
          query_per_second: 1
      show_limit_quota_header: true
      rejected_code: 429
      redis:
        service_name: basicai-redis-master.redis-cluster.svc.cluster.local
        service_port: 6379
        password: ******

即便配置 username 为空也不行。

进到gateway pod里，curl http://localhost:15000/clusters 看看，是否有对应的cluster

默认配置的是只下发有路由的cluster，有两种解决方法：

1.新增一条路由关联redis对应的service

2.修改一下环境变量为false，会下发Kubernetes集群中所有的service对应的cluster

lQLPJwSkjs8mGXtlzQQjsCCkVjYeiIFaBnX7yWIhEQA_1059_101

这个也有挺多人在使用redis的时候遇到

jaggerwang commented 3 months ago

禁用 global.onlyPushRouteCluster 后可以了，谢谢！

hanxiantao commented 3 months ago

禁用 global.onlyPushRouteCluster 后可以了，谢谢！

好的，我这边会尽快完善下如何访问redis的文档，把涉及的case都给出对应示例，感谢提问

cr7258 commented 3 months ago

@CH3CHO @hanxiantao 文档已更新 https://github.com/higress-group/higress-group.github.io/pull/250 https://github.com/alibaba/higress/pull/1113