kmesh-net / kmesh

High Performance ServiceMesh Data Plane Based on Programmable Kernel
https://kmesh.net
Apache License 2.0
361 stars 46 forks source link

Kmesh cannot get new applyed RDS config during runtime #437

Closed lec-bit closed 1 week ago

lec-bit commented 2 weeks ago

What happened: During the runtime of kmesh, when applying the fortio-service.yaml, kmesh is unable to retrieve the latest RDS configuration.

After restarting kmesh, it is able to successfully retrieve the latest RDS configuration. What you expected to happen: During the runtime of kmesh, when applying the fortio-service.yaml, kmesh can get the latest RDS configuration.

How to reproduce it (as minimally and precisely as possible): 1.Start kmesh : helm install kmesh ./deploy/helm -n kmesh-system --create-namespace 2.apply fortio-service.yaml : kubectl apply -f fortio-service.yaml 3.check config : kubectl exec -it -n kmesh-system kmesh-hvffp -- curl localhost:15200/debug/config_dump/ads, and check routeConfigs: there is not fortio-server

fortio-service.yaml:
apiVersion: v1
kind: Service
metadata:
  labels:
    app: fortio-server
  name: fortio-server
  namespace: default
spec:
  type: ClusterIP
  ports:
    - port: 80
      targetPort: 8080
  selector:
    app: fortio-server

Anything else we need to know?:

Environment:

hzxuzhonghu commented 2 weeks ago

We Investigated, and found the case is

  1. RDS request is send with the inferred route names from parsing listener, and with the nonce from previous rds response.
  2. Istiod has pushed new rds response, but kmesh hasn't yet received.
  3. The rds request sent from 1, then has nonce mismatch with istiod. So istiod will ignore and print Expired nonce received log