search

kmesh-net / kmesh

High Performance ServiceMesh Data Plane Based on Programmable Kernel
https://kmesh.net
Apache License 2.0
419 stars 57 forks source link

Fault Injection yaml file does not take effect #694

Closed noobwei closed 2 weeks ago

noobwei commented 1 month ago

What happened:

I am expanding the presentation of Kmesh L7 capabilities, and when I try Fault Injection, the yaml does not take effect.

After running kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.21/samples/bookinfo/networking/virtual-service-ratings-test-delay.yaml,

The result of kubectl get virtualservice ratings -o yaml is 

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"networking.istio.io/v1alpha3","kind":"VirtualService","metadata":{"annotations":{},"name":"ratings","namespace":"default"},"spec":{"hosts":["ratings"],"http":[{"fault":{"delay":{"fixedDelay":"7s","percentage":{"value":100}}},"match":[{"headers":{"end-user":{"exact":"jason"}}}],"route":[{"destination":{"host":"ratings","subset":"v1"}}]},{"route":[{"destination":{"host":"ratings","subset":"v1"}}]}]}}
  creationTimestamp: "2024-08-06T08:12:14Z"
  generation: 2
  name: ratings
  namespace: default
  resourceVersion: "303812"
  uid: 34e2d70a-46ed-4b08-90d1-4e50d1cd57c0
spec:
  hosts:
  - ratings
  http:
  - fault:
      delay:
        fixedDelay: 7s
        percentage:
          value: 100
    match:
    - headers:
        end-user:
          exact: jason
    route:
    - destination:
        host: ratings
        subset: v1
  - route:
    - destination:
        host: ratings
        subset: v1

The yaml shows I have added fixedDelay of 7s when I log as Jason

But the review of productpage is still there, and no delay added for the website.

image

What you expected to happen:

On the productpage web page, log in as user jason.

I expect the Bookinfo home page to load without errors in approximately 7 seconds. However, there is a problem: the Reviews section displays an error message: Sorry, product reviews are currently unavailable for this book.

How to reproduce it (as minimally and precisely as possible):

  1. Follow the preparation part of https://kmesh.net/en/docs/userguide/try_waypoint/#preparation

  2. kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.21/samples/bookinfo/networking/virtual-service-all-v1.yaml

  3. kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.21/samples/bookinfo/networking/virtual-service-reviews-test-v2.yaml

With the above configuration, this is how requests flow:

  1. kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.21/samples/bookinfo/networking/virtual-service-ratings-test-delay.yaml

Anything else we need to know?:

https://istio.io/latest/docs/tasks/traffic-management/fault-injection/

Environment:

hzxuzhonghu commented 1 month ago

Can you debug it refering https://istio.io/latest/docs/ops/diagnostic-tools/proxy-cmd/

And show the output of istioctl proxy-config routes {waypoint} -ojson

noobwei commented 1 month ago

image

And the result of istioctl proxy-config routes reviews-svc-waypoint-6884756fc5-g9cb2.default -o json is

[
    {
        "name": "inbound-vip|9080|http|reviews.default.svc.cluster.local",
        "virtualHosts": [
            {
                "name": "inbound|http|9080",
                "domains": [
                    "*"
                ],
                "routes": [
                    {
                        "match": {
                            "prefix": "/",
                            "caseSensitive": true,
                            "headers": [
                                {
                                    "name": "end-user",
                                    "stringMatch": {
                                        "exact": "jason"
                                    }
                                }
                            ]
                        },
                        "route": {
                            "cluster": "inbound-vip|9080|http/v2|reviews.default.svc.cluster.local",
                            "timeout": "0s",
                            "retryPolicy": {
                                "retryOn": "connect-failure,refused-stream,unavailable,cancelled,retriable-status-codes",
                                "numRetries": 2,
                                "retryHostPredicate": [
                                    {
                                        "name": "envoy.retry_host_predicates.previous_hosts",
                                        "typedConfig": {
                                            "@type": "type.googleapis.com/envoy.extensions.retry.host.previous_hosts.v3.PreviousHostsPredicate"
                                        }
                                    }
                                ],
                                "hostSelectionRetryMaxAttempts": "5",
                                "retriableStatusCodes": [
                                    503
                                ]
                            },
                            "maxGrpcTimeout": "0s"
                        },
                        "metadata": {
                            "filterMetadata": {
                                "istio": {
                                    "config": "/apis/networking.istio.io/v1alpha3/namespaces/default/virtual-service/reviews"
                                }
                            }
                        },
                        "decorator": {
                            "operation": "reviews.default.svc.cluster.local:9080/*"
                        }
                    },
                    {
                        "match": {
                            "prefix": "/"
                        },
                        "route": {
                            "cluster": "inbound-vip|9080|http/v1|reviews.default.svc.cluster.local",
                            "timeout": "0s",
                            "retryPolicy": {
                                "retryOn": "connect-failure,refused-stream,unavailable,cancelled,retriable-status-codes",
                                "numRetries": 2,
                                "retryHostPredicate": [
                                    {
                                        "name": "envoy.retry_host_predicates.previous_hosts",
                                        "typedConfig": {
                                            "@type": "type.googleapis.com/envoy.extensions.retry.host.previous_hosts.v3.PreviousHostsPredicate"
                                        }
                                    }
                                ],
                                "hostSelectionRetryMaxAttempts": "5",
                                "retriableStatusCodes": [
                                    503
                                ]
                            },
                            "maxGrpcTimeout": "0s"
                        },
                        "metadata": {
                            "filterMetadata": {
                                "istio": {
                                    "config": "/apis/networking.istio.io/v1alpha3/namespaces/default/virtual-service/reviews"
                                }
                            }
                        },
                        "decorator": {
                            "operation": "reviews.default.svc.cluster.local:9080/*"
                        }
                    }
                ]
            }
        ],
        "validateClusters": false
    },
    {
        "name": "default",
        "virtualHosts": [
            {
                "name": "default",
                "domains": [
                    "*"
                ],
                "routes": [
                    {
                        "match": {
                            "prefix": "/"
                        },
                        "route": {
                            "cluster": "main_internal"
                        }
                    }
                ]
            }
        ]
    },
    {
        "virtualHosts": [
            {
                "name": "backend",
                "domains": [
                    "*"
                ],
                "routes": [
                    {
                        "match": {
                            "prefix": "/stats/prometheus"
                        },
                        "route": {
                            "cluster": "prometheus_stats"
                        }
                    }
                ]
            }
        ]
    },
    {
        "virtualHosts": [
            {
                "name": "backend",
                "domains": [
                    "*"
                ],
                "routes": [
                    {
                        "match": {
                            "prefix": "/healthz/ready"
                        },
                        "route": {
                            "cluster": "agent"
                        }
                    }
                ]
            }
        ]
    },
    {
        "name": "default",
        "virtualHosts": [
            {
                "name": "default",
                "domains": [
                    "*"
                ],
                "routes": [
                    {
                        "match": {
                            "connectMatcher": {}
                        },
                        "route": {
                            "cluster": "main_internal",
                            "upgradeConfigs": [
                                {
                                    "upgradeType": "CONNECT",
                                    "connectConfig": {}
                                }
                            ]
                        }
                    }
                ]
            }
        ]
    },
    {
        "name": "encap",
        "virtualHosts": [
            {
                "name": "inbound|http|0",
                "domains": [
                    "*"
                ],
                "routes": [
                    {
                        "name": "default",
                        "match": {
                            "prefix": "/"
                        },
                        "route": {
                            "cluster": "encap",
                            "timeout": "0s",
                            "maxStreamDuration": {
                                "maxStreamDuration": "0s",
                                "grpcTimeoutHeaderMax": "0s"
                            }
                        },
                        "decorator": {
                            "operation": ":0/*"
                        }
                    }
                ]
            }
        ],
        "validateClusters": false
    }
]
hzxuzhonghu commented 1 month ago

I cannot find delay is configurred in the route, need to dig into istio code.

YaoZengzeng commented 1 month ago 
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"networking.istio.io/v1alpha3","kind":"VirtualService","metadata":{"annotations":{},"name":"ratings","namespace":"default"},"spec":{"hosts":["ratings"],"http":[{"fault":{"delay":{"fixedDelay":"7s","percentage":{"value":100}}},"match":[{"headers":{"end-user":{"exact":"jason"}}}],"route":[{"destination":{"host":"ratings","subset":"v1"}}]},{"route":[{"destination":{"host":"ratings","subset":"v1"}}]}]}}
  creationTimestamp: "2024-08-06T08:12:14Z"
  generation: 2
  name: ratings
  namespace: default
  resourceVersion: "303812"
  uid: 34e2d70a-46ed-4b08-90d1-4e50d1cd57c0
spec:
  hosts:
  - ratings
  http:
  - fault:
      delay:
        fixedDelay: 7s
        percentage:
          value: 100
    match:
    - headers:
        end-user:
          exact: jason
    route:
    - destination:
        host: ratings
        subset: v1
  - route:
    - destination:
        host: ratings
        subset: v1

Actually this VirtualService is for ratings service, it will only take effect after we deploy a waypoint for this service.

But there is still a bug in reviews' access to ratings, ref: https://github.com/kmesh-net/kmesh/issues/756 .

Once this bug is fixed, we can continue writing this doc.

noobwei commented 4 weeks ago

Doc Request Timeouts also need to wait till this bug fixed

YaoZengzeng commented 3 weeks ago

@noobwei bug is fixed, you could move on 🚀

hzxuzhonghu commented 3 weeks ago

what's the status of this?

noobwei commented 3 weeks ago

Still under confirmation, there are some problems with my environment at present, I will comment after confirmation

noobwei commented 2 weeks ago

It has been fixed! Thanks. I will close this issue soon.