Kuma 2.8.1 and kong - client failed to connect to the verification server

[balait4]

What happened?

we are using kong(3.5.0)& KIC(3.0.2) and kuma 2.8.1. I'm exploring option to integrate kong with Kuma as per the doc https://kuma.io/docs/2.8.x/guides/gateway-delegated/ When I inject [kuma.io/sidecar-injection](http://kuma.io/sidecar-injection): enabled for kong namespace, the initcontainer kuma-validation is having below issues.

2024-07-10T19:49:02.708Z    INFO    validator   starting iptables validation
2024-07-10T19:49:02.708Z    INFO    validator   listening on 127.0.0.1:15006
2024-07-10T19:49:02.709Z    INFO    validator   [WARNING] client failed to connect to server: dial tcp 127.0.0.1:0->127.0.0.6:29242: connect: connection refused
2024-07-10T19:49:03.710Z    INFO    validator   [WARNING] client failed to connect to server: dial tcp 127.0.0.1:0->127.0.0.6:29242: connect: connection refused
2024-07-10T19:49:12.721Z    INFO    validator   [WARNING] client failed to connect to server: dial tcp 127.0.0.1:0->127.0.0.6:29242: connect: connection refused
Error: validation failed, client failed to connect to the verification server: dial tcp 127.0.0.1:0->127.0.0.6:29242: connect: connection refused

[lukidzi]

we don't intercept inbounds for delegated gateway, we need to check it

[lukidzi]

@balait4 Hi. I tried to reproduce the issue by installing Kuma with CNI in my k3d cluster following these guides:

• Configure the Kuma CNI
• Gateway Delegated

However, I couldn't reproduce the issue. The validator is only applied when using Kuma with CNI and if the inbound traffic is redirected by iptables. In the case of the delegated gateway, we don't redirect the traffic.

Could you send your installation command and let me know if you applied any changes? Also, could you show your Pod definition where you encountered the issue?

[balait4]

Hi,

This is okd (openshift) cluster, used below command to install kuma.

kumactl install control-plane \ --set "cni.enabled=true" \ | kubectl apply -f -

[michaelbeaumont]

@balait4 Since you're using Openshift 4 or 3.11 can you try with the corresponding instructions here? In particular:

--set "cni.containerSecurityContext.privileged=true" \

[balait4]

Thanks for the reply! Yes we are using version 4.15 Openshift. I used the below command and still we are facing the issue.

kumactl install control-plane \
     --set "cni.enabled=true" \
     --set "cni.containerSecurityContext.privileged=true" \
     | kubectl apply -f -

When enable kubectl label namespace kong kuma.io/sidecar-injection=enabled still getting below error.

2024-07-25T10:17:25.024Z    INFO    validator   starting iptables validation
2024-07-25T10:17:25.024Z    INFO    validator   listening on 127.0.0.1:15006
2024-07-25T10:17:25.025Z    INFO    validator   [WARNING] client failed to connect to server: dial tcp 127.0.0.1:0->127.0.0.6:25278: connect: connection refused
2024-07-25T10:17:26.026Z    INFO    validator   [WARNING] client failed to connect to server: dial tcp 127.0.0.1:0->127.0.0.6:25278: connect: connection refused
2024-07-25T10:17:27.027Z    INFO    validator   [WARNING] client failed to connect to server: dial tcp 127.0.0.1:0->127.0.0.6:25278: connect: connection refused
2024-07-25T10:17:28.029Z    INFO    validator   [WARNING] client failed to connect to server: dial tcp 127.0.0.1:0->127.0.0.6:25278: connect: connection refused
2024-07-25T10:17:29.029Z    INFO    validator   [WARNING] client failed to connect to server: dial tcp 127.0.0.1:0->127.0.0.6:25278: connect: connection refused
2024-07-25T10:17:30.030Z    INFO    validator   [WARNING] client failed to connect to server: dial tcp 127.0.0.1:0->127.0.0.6:25278: connect: connection refused
2024-07-25T10:17:31.032Z    INFO    validator   [WARNING] client failed to connect to server: dial tcp 127.0.0.1:0->127.0.0.6:25278: connect: connection refused
2024-07-25T10:17:32.033Z    INFO    validator   [WARNING] client failed to connect to server: dial tcp 127.0.0.1:0->127.0.0.6:25278: connect: connection refused
2024-07-25T10:17:33.034Z    INFO    validator   [WARNING] client failed to connect to server: dial tcp 127.0.0.1:0->127.0.0.6:25278: connect: connection refused
2024-07-25T10:17:34.035Z    INFO    validator   [WARNING] client failed to connect to server: dial tcp 127.0.0.1:0->127.0.0.6:25278: connect: connection refused
2024-07-25T10:17:35.036Z    INFO    validator   [WARNING] client failed to connect to server: dial tcp 127.0.0.1:0->127.0.0.6:25278: connect: connection refused
Error: validation failed, client failed to connect to the verification server: dial tcp 127.0.0.1:0->127.0.0.6:25278: connect: connection refused

[michaelbeaumont]

@balait4 could you post the Pod yaml where the container is failing?

[balait4]

Please find the attached Pod yaml file. pod.txt

[michaelbeaumont]

Thanks @balait4 can you share how exactly you installed KIC?

Can you try setting the annotation kuma.io/gateway: enabled on your KIC Pods?

[balait4]

We have db-less, standalone installation. We are using KIC for past 3 years. We installed CRD's and prepared yaml and deployed in our cluster. (not using helm or anyother method) Yes I used the kuma.io/gateway: enabled settings without success.

[michaelbeaumont]

Yes I used the kuma.io/gateway: enabled settings without success.

It's important that the annotation is there when the Pod is created, can you confirm you annotated the Pod template in the KIC Deployment?

[balait4]

Thanks after making entry in pod template (annotation), looks validation is good.. the kuma-sidecar pod is giving the below error

2024-07-26T12:43:56.831Z    INFO    system-certificate-selector using OS provided CA certificate    {"certificate path": "/etc/ssl/certs/ca-certificates.crt"}
2024-07-26T12:43:56.908Z    INFO    kuma-dp.run fetched Envoy version   {"version": {"Build":"32113313a357829ba3a5dce0795b6780bf8cbf4d/1.30.4/Modified/RELEASE/BoringSSL","Version":"1.30.4","KumaDpCompatible":true}}
2024-07-26T12:43:56.908Z    INFO    kuma-dp.run generating bootstrap configuration
2024-07-26T12:43:56.908Z    INFO    dataplane   trying to fetch bootstrap configuration from the Control Plane
2024-07-26T12:43:56.913Z    INFO    dataplane   could not fetch bootstrap configuration, make sure you are not trying to connect to global-cp. retrying (this could help only if you're connecting to zone-cp). {"backoff": "3s", "err": "request to bootstrap server failed: Post \"https://kuma-control-plane.kuma-system:5678/bootstrap\": dial tcp: lookup kuma-control-plane.kuma-system on 185.12.64.1:53: no such host"}
2024-07-26T12:43:59.914Z    INFO    dataplane   trying to fetch bootstrap configuration from the Control Plane

THe other demo pod deployment works fine and in gui able to see it online.

[michaelbeaumont]

Hmm that looks like something may not be correctly configured with your DNS, it looks like the Pod is trying to resolve names using the Hetzner nameservers?

[balait4]

Thanks adjusted the DNS policy of the deployment. Looks fine now.