JWT规则无法及时响应配置变更，经常不生效

listapp99 commented 1 month ago

If you are reporting any crash or any potential security issue, do not open an issue in this repo. Please report the issue via ASRC(Alibaba Security Response Center) where the issue will be triaged appropriately.

[ ] I have searched the issues of this repository and believe that this is not a duplicate.

Ⅰ. Issue Description

JWT规则无法及时响应配置变更，经常不生效

Ⅱ. Describe what happened

根据持续的curl结果，看到jwt配置变更经常不生效，无法根据配置变化动态拦截没有鉴权的请求，对于jwt频繁变更的应用场景，higress无法适用。

Ⅲ. Describe what you expected to happen

higress及时响应jwt配置变更，根据配置变化动态拦截没有鉴权的请求

Ⅳ. How to reproduce it (as minimally and precisely as possible)

1、测试ingress配置

kind: Ingress
metadata:
  name: demo.abc.com
spec:
  ingressClassName: higress
  rules:
  - host: demo.abc.com
    http:
      paths:
      - pathType: Prefix
        path: /
        backend:
          service:
            name: demo-ping
            port: 
              number: 8888

2、测试jwt配置，保存为jwt.yaml

kind: WasmPlugin
metadata:
  name: demo.abc.com
  namespace: higress-system
spec:
  defaultConfig:
    consumers:
    - name: demo.abc.com
      issuer: demo.abc.com
      jwks: |
        { 
          "keys": [
            {
              "alg": "RS256",
              "e": "AQAB",
              "kid": "DHFbpoIUqrY8t2zpA2qXfCmr5VO5ZEr4RzHU_-envvQ",
              "kty": "RSA",
              "n": "xAE7eB6qugXyCAG3yhh7pkDkT65pHymX-P7KfIupjf59vsdo91bSP9C8H07pSAGQO1MV_xFj9VswgsCg4R6otmg5PV2He95lZdHtOcU5DXIg_pbhLdKXbi66GlVeK6ABZOUW3WYtnNHD-91gVuoeJT_DwtGGcp4ignkgXfkiEm4sw-4sfb4qdt5oLbyVpmW6x9cfa7vs2WTfURiCrBoUqgBo_-4WTiULmmHSGZHOjzwa8WtrtOQGsAFjIbno85jp6MnGGGZPYZbDAa_b3y5u-YpW7ypZrvD8BgtKVjgtQgZhLAGezMt0ua3DRrWnKqTZ0BJ_EyxOGuHJrLsn00fnMQ",
              "use": "sig"
            }
          ]
        }
  matchRules:
  - ingress: ["default/demo.abc.com"]
    config:
      allow: ["demo.abc.com"]
  url: oci://higress-registry.cn-hangzhou.cr.aliyuncs.com/plugins/jwt-auth:1.0.0

3、复现操作 for((i=0;i<300;i++));do echo $i;kubectl apply -f jwt.yaml;sleep 2;kubectl delete wasmplugin -n higress-system demo.abc.com;sleep 2;done

4、观察 for((i=0;i<1000;i++));do echo $i;curl -v http://demo.abc.com;sleep 1;done

Ⅴ. Anything else we need to know?

Ⅵ. Environment:

Higress version: 1.3.6
OS : linux，k8s
Others:

johnlanni commented 1 month ago

是说配置生效慢么

listapp99 commented 1 month ago

是说配置生效慢么

有的时候生效慢，有的时候不生效，生效慢的时候要等1分钟以上

johnlanni commented 1 month ago

https://help.aliyun.com/zh/mse/user-guide/jwt-auth-plug-ins?spm=a2c4g.11186623.0.0.60b76af7PvLZfd 企业版的jwt插件目前支持配置remote_jwks，动态更新jwks，能满足你需求吗。插件配置修改生效是秒级的，这个做不到很实时。

johnlanni commented 1 month ago

超过一分钟不会的，这种一般是你的jwks格式有问题，可以看下网关日志有配置解析的报错

listapp99 commented 1 month ago

超过一分钟不会的，这种一般是你的jwks格式有问题，可以看下网关日志有配置解析的报错

配置是正确的，日志没有报错。在单次变更中能够生效，但如果变更次数频繁，比如按照帖子里的复现方法，就会出现长时间不响应配置变更的情况出现

pepesi commented 1 month ago

可以打开debug日志，将启动参数的日志等级改成debug ,grep jwt下，或许能得到有用的信息

johnlanni commented 1 month ago

@listapp99 插件配置初始化需要占用主线程cpu，你看是否变更过于频繁主线程CPU跑满了，可以进gateway 容器执行 top -H -p 看一下线程的cpu情况

listapp99 commented 1 month ago

@listapp99 插件配置初始化需要占用主线程cpu，你看是否变更过于频繁主线程CPU跑满了，可以进gateway 容器执行 top -H -p 看一下线程的cpu情况

cpu是正常的

listapp99 commented 1 month ago

我已定位原因，istio-1.12版本下发xds配置存在并发性bug，higress-1.3.6内核基于istio-1.12，继承了该bug，istio-1.14.1版本已修复。 https://github.com/istio/istio/pull/39172

johnlanni commented 1 month ago

@listapp99 好的我们正在准备升级istio到1.19，会顺带解决这个问题

johnlanni commented 3 days ago

2.0.0-rc.1已经发布，该问题已经修复

alibaba / higress