search

Kong / kong

🦍 The Cloud-Native API Gateway and AI Gateway.
https://konghq.com/install/#kong-community
Apache License 2.0
38.89k stars 4.78k forks source link

Istio Circuit Breaker Not working with Kong #8147

Closed dawo0od closed 2 years ago

dawo0od commented 2 years ago

Is there an existing issue for this?

Kong version ($ kong version)

2.2

Current Behavior

We’re running Istio on Kubernetes and Kong as API gateway and ingress controller for our K8S cluster. We’ve created virtual services and destination rules and applied circuit breaking rules for our service.

I'm using fortio client in order to test circuit breaker functionality. Functionality is working perfectly fine and circuit breaking is happening if fortio is calling the service directly.

The same is not working as expected when fortio is calling Kong and Kong itself is calling the service.

fortio ==> service (Working fine) fortio ==> kong ==> service (Not working)

Could please anyone advise here urgently ?

Expected Behavior

Circuit breaking should happen even if the call is coming through Kong.

Steps To Reproduce

No response

Anything else?

No response

fffonion commented 2 years ago

@dawo0od Make sure Kong is configured properly to forward status, headers to client according your circuit breaker policies.

dawo0od commented 2 years ago

@fffonion I have found that in my Istio-proxy logs for my Kong POD , it is connecting to my service with the POD IP , not the service IP. I believe this is the issue.

Is this the normal behavior for Kong that it is calling the service with the POD IP ?

dawo0od commented 2 years ago

[C117875][S7310483570645225078] cluster 'PassthroughCluster' match for URL '/price-listing-service/v1/config/currency/0' 2021-12-27T11:06:31.643385Z debug envoy upstream transport socket match, socket default selected for host with address 10.50.16.93:8080 2021-12-27T11:06:31.643394Z debug envoy upstream Created host 10.50.16.93:8080. 2021-12-27T11:06:31.643470Z debug envoy upstream addHost() adding 10.50.16.93:8080 2021-12-27T11:06:31.643513Z debug envoy upstream membership update for TLS cluster PassthroughCluster added 1 removed 0 2021-12-27T11:06:31.643543Z debug envoy upstream re-creating local LB for TLS cluster PassthroughCluster 2021-12-27T11:06:31.643572Z debug envoy router [C117875][S7310483570645225078] router decoding headers:

dawo0od commented 2 years ago

@fffonion Any update here please ?

fffonion commented 2 years ago

@dawo0od Sorry for the late reply. Try to use the annotation mentioned in this blogpost: https://kubernetes.io/blog/2020/03/18/kong-ingress-controller-and-istio-service-mesh/ ingress.kubernetes.io/service-upstream: "true".

dawo0od commented 2 years ago

@fffonion Yes I have this annotation configured on the service level already and preserve_host is disabled as well but still does not work.

Actually we are struggling with this since long time and not able to get it fixed. I've also noticed that a lot of people have exactly the same issue as per below link.

https://discuss.konghq.com/t/kong-ingress-controller-is-unable-to-route-request-to-istio-virtual-services/4457/5

mayocream commented 2 years ago

cc @Kong/team-k8s

mayocream commented 2 years ago

@dawo0od Hi, what Kong Kubernetes Ingress Controller version are you using?

mayocream commented 2 years ago

@dawo0od Also, can you provide the Helm Chart version of Ingress Controller and post values.yaml file here?

notice: don't forget to remove sensitive secrets from values.yaml

There may be something wrong with the configuration and these versions of running components may help us investigate and find the key to solving your question.

mayocream commented 2 years ago

We have references for running Kong and Istio in your K8s clusters.

Take a look:

dawo0od commented 2 years ago

@mayocream We are using Kong ingress controller version 1.2 and helm chart 3.

Please also note that I've already tried below link you shared on my minikube cluster but still istio routing rules are not working. They are only working when calling from service to service directly and not through Kong. https://docs.konghq.com/kubernetes-ingress-controller/2.0.x/guides/getting-started-istio/

rainest commented 2 years ago

Do the controller logs indicate it's failing to update configuration? If you set ingressController.env.log_level=debug in your Helm values, do you see a fetching endpoints log for the upstream in question?

The code that handles this is quite simple and is tested; there's not an obvious reason for it to fail. We check for the presence of the annotation at: https://github.com/Kong/kubernetes-ingress-controller/blob/v1.2.0/pkg/annotations/annotations.go#L187-L191 https://github.com/Kong/kubernetes-ingress-controller/blob/v1.2.0/pkg/annotations/annotations_test.go#L274-L309

And then set the service hostname as the sole upstream target at: https://github.com/Kong/kubernetes-ingress-controller/blob/v1.2.0/pkg/parser/parser.go#L385-L391 https://github.com/Kong/kubernetes-ingress-controller/blob/v1.2.0/pkg/parser/parser_test.go#L3424-L3458

If you do see a fetching endpoints log for that upstream, that indicates we did not see the annotation on the Service. Can you provide the Service YAML definition and the admin API output for GET /upstreams/<UPSTREAM>/targets, where <UPSTREAM> is <service name>.<service namespace>.<service port>.svc (e.g. for a Service named foo in namespace default on port 80, foo.default.80.svc).

dawo0od commented 2 years ago

@rainest Thanks a lot for your response. The issue is that I was using Ingress.kubernetes.io/service-upstream: "true" instead of ingress.kubernetes.io/service-upstream: "true" ('I' of ingress was capital ) :(

Last question please , in my case do I need to have Istio side car on Kong POD as well ? or it is not mandatory ?

rainest commented 2 years ago

We indicate that you should use sidecars on the Kong instances in https://docs.konghq.com/kubernetes-ingress-controller/2.0.x/guides/getting-started-istio/#deploy-kong-gateway-and-kubernetes-ingress-controller

I think the more specific answer is that you do not need one for some Istio features, e.g. circuit-breaking should not require anything on the Kong side since it can be handled entirely by the upstream Service's sidecar, whereas other features require sidecars on both the client (Kong) and upstream, e.g. mutual TLS requires that both sides have a sidecar to present the managed certificate. Not positive as I haven't tested it exhaustively, but that's what I would expect.

satnamalhan commented 2 years ago

remove here values.yaml in this sensitive information

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.