Kubernetes > 1.22

[tawalaya]

Problem

Hi, we are running a recent version of Kubernetes, and for some reason, scrapandre does not pick up any pods. I've looked into it a bit, and it might be related to the kube-sync crate. It is using a fairly old version of k8s-openapi that only supports Kubernetes 1.22. Looking into this a bit more, there may be an issue with the kube-sync in the near future as the k8s-openapi removed some of the used features in favor of the kube library (API clients) see here.

Solution

Thus, it might be better to migrate away from the kube-sync crate to the kube crate to stay up to date with newer Kubernetes releases.

Alternatives

Update the kube-sync to work with newer versions of Kubernetes.

[mmadoo]

I have a cluster on k8s 1.26.8 with scaphandre docker image with tag dev and it works well.

[tawalaya]

We are running v1.29. It could also be a different reason why the pods don't show up. However, the deprecation of the kube-openapi API will be an issue sooner or later.

[tawalaya]

The error i see in the logs is this btw:

scaphandre::exporters: watching kubernetes...
isahc::client: send; method=GET uri=https://10.96.0.1:443/api/v1/namespaces//pods?
isahc::handler: handler;
isahc::handler: handler; id=0
isahc::handler:   Trying 10.96.0.1:443...
isahc::handler: Connected to 10.96.0.1 (10.96.0.1) port 443 (#0)
isahc::handler: ALPN: offers h2,http/1.1
isahc::handler: TLSv1.3 (OUT), TLS handshake, Client hello (1):
isahc::handler: TLSv1.3 (IN), TLS handshake, Server hello (2):
isahc::handler: TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
isahc::handler: TLSv1.3 (IN), TLS handshake, Request CERT (13):
isahc::handler: TLSv1.3 (IN), TLS handshake, Certificate (11):
isahc::handler: TLSv1.3 (IN), TLS handshake, CERT verify (15):
isahc::handler: TLSv1.3 (IN), TLS handshake, Finished (20):
isahc::handler: TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
isahc::handler: TLSv1.3 (OUT), TLS handshake, Certificate (11):
isahc::handler: TLSv1.3 (OUT), TLS handshake, Finished (20):
isahc::handler: SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
isahc::handler: ALPN: server accepted h2
isahc::handler: Server certificate:
isahc::handler:  subject: CN=kube-apiserver
isahc::handler:  start date: Jan  5 14:52:55 2024 GMT
isahc::handler:  expire date: Jan  4 14:57:56 2025 GMT
isahc::handler:  subjectAltName: host "10.96.0.1" matched cert's IP address!
isahc::handler:  issuer: CN=kubernetes
isahc::handler:  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
isahc::handler: using HTTP/2
isahc::handler: h2h3 [:method: GET]
isahc::handler: h2h3 [:path: /api/v1/namespaces//pods?]
isahc::handler: h2h3 [:scheme: https]
isahc::handler: h2h3 [:authority: 10.96.0.1]
isahc::handler: h2h3 [accept: */*]
isahc::handler: h2h3 [accept-encoding: deflate, gzip]
isahc::handler: h2h3 [authorization: Bearer OMITTED]
isahc::handler: h2h3 [user-agent: curl/8.0.1-DEV isahc/1.7.2]
isahc::handler: h2h3 [content-length: 0]
isahc::handler: Using Stream ID: 1 (easy handle 0x563c2768d6d0)
isahc::handler: TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
isahc::handler: Connection #0 to host 10.96.0.1 left intact
scaphandre::exporters: Failed getting pods list, despite client seems ok. Couldn't build http client (isahc)
scaphandre::exporters: First check done on pods.

[josefhandl]

I have the exact same problem on kubernetes v1.28.8. The Kubernetes API call hangs in my case. I think this is caused by the empty namespace in the URL, because hardcoded specific namespace name works fine. So it is not possible to list any pod.

According to the Kubernetes documentation, specifying the namespace in the URL is required. And from what I found, it looks like the methods used from the k8s-sync and k8s-openapi libraries are designed to specify non-empty namespace name.

isahc::client: send; method=GET uri=https://localhost:6443/api/v1/namespaces//pods?
isahc::handler: handler;
isahc::handler: handler; id=0
isahc::handler:   Trying 127.0.0.1:6443...
isahc::handler: Connected to localhost (127.0.0.1) port 6443 (#0)
isahc::handler: ALPN: offers h2,http/1.1
isahc::handler: TLSv1.3 (OUT), TLS handshake, Client hello (1):
isahc::handler: TLSv1.3 (IN), TLS handshake, Server hello (2):
isahc::handler: TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
isahc::handler: TLSv1.3 (IN), TLS handshake, Request CERT (13):
isahc::handler: TLSv1.3 (IN), TLS handshake, Certificate (11):
isahc::handler: TLSv1.3 (IN), TLS handshake, CERT verify (15):
isahc::handler: TLSv1.3 (IN), TLS handshake, Finished (20):
isahc::handler: TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
isahc::handler: TLSv1.3 (OUT), TLS handshake, Certificate (11):
isahc::handler: TLSv1.3 (OUT), TLS handshake, CERT verify (15):
isahc::handler: TLSv1.3 (OUT), TLS handshake, Finished (20):
isahc::handler: SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
isahc::handler: ALPN: server accepted h2
isahc::handler: Server certificate:
isahc::handler:  subject: CN=kube-apiserver
isahc::handler:  start date: Nov  1 17:11:01 2021 GMT
isahc::handler:  expire date: Oct 31 00:59:44 2024 GMT
isahc::handler:  subjectAltName: host "localhost" matched cert's "localhost"
isahc::handler:  issuer: CN=rke2-server-ca@1635786661
isahc::handler:  SSL certificate verify result: self-signed certificate in certificate chain (19), continuing anyway.
isahc::handler: using HTTP/2
isahc::handler: h2h3 [:method: GET]
isahc::handler: h2h3 [:path: /api/v1/namespaces//pods?]
isahc::handler: h2h3 [:scheme: https]
isahc::handler: h2h3 [:authority: localhost:6443]
isahc::handler: h2h3 [accept: */*]
isahc::handler: h2h3 [accept-encoding: deflate, gzip]
isahc::handler: h2h3 [user-agent: curl/8.0.1-DEV isahc/1.7.2]
isahc::handler: h2h3 [content-length: 0]
isahc::handler: Using Stream ID: 1 (easy handle 0x561ea8743f70)
isahc::handler: TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):

[josefhandl]

I was wrong. The empty namespace in the URL is not the problem. My problem was related with freezing of the parsing of the Kubernetes API response by the k8s-sync library. This problem occurred in my case if the response was too large.

I created a patch in the k8s-sync library. Using my patch the parsing works fine.

After finding this, I am not sure if my problem is related with this issue, so apologize for spam. But still, I think that the parsing function in the k8s-sync is not ideal.