[Bug] Network costs crashing on node startup

[elcomtik]

Kubecost Version

2.0.2

Kubernetes Version

1.28

Kubernetes Platform

EKS

Description

I updated from kubecost v1.105.1 to v1.106.7 and later to v2.0.2

Network costs were updated from v0.6.7 to v0.7.2 and problems started when the new k8s node was created.

I use standard AL2 EKS nodes, it worked before the update. It seems that the downgrade of network-costs container fixed the issue temporarily.

Steps to reproduce

When a new k8s node is created Daemonset starts the network-costs pod on the new node and the following error occurs on pod:

ERROR kube_client::client::builder: failed with error error trying to connect: deadline has elapsed
ERROR kube_client::client::builder: failed with error error trying to connect: deadline has elapsed
ERROR kube_client::client::builder: failed with error error trying to connect: deadline has elapsed
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: HyperError(hyper::Error(Connect, Custom { kind: TimedOut, error: Elapsed(()) }))', src/kube/netmap.rs:570:44 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: HyperError(hyper::Error(Connect, Custom { kind: TimedOut, error: Elapsed(()) }))', src/kube/netmap.rs:619:45
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: HyperError(hyper::Error(Connect, Custom { kind: TimedOut, error: Elapsed(()) }))', src/kube/netmap.rs:663:47

The pod is not emitting any metrics and also not restarted automatically.

If I restart it manually, it starts working as expected.

Expected behavior

Network costs start without panicking, or getting restarted by the liveness probe.

Impact

Network cost metrics are not complete.

Screenshots

No response

Logs

ERROR kube_client::client::builder: failed with error error trying to connect: deadline has elapsed
ERROR kube_client::client::builder: failed with error error trying to connect: deadline has elapsed
ERROR kube_client::client::builder: failed with error error trying to connect: deadline has elapsed
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: HyperError(hyper::Error(Connect, Custom { kind: TimedOut, error: Elapsed(()) }))', src/kube/netmap.rs:570:44 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: HyperError(hyper::Error(Connect, Custom { kind: TimedOut, error: Elapsed(()) }))', src/kube/netmap.rs:619:45
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: HyperError(hyper::Error(Connect, Custom { kind: TimedOut, error: Elapsed(()) }))', src/kube/netmap.rs:663:47

Slack discussion

No response

Troubleshooting

• [X] I have read and followed the issue guidelines and this is a bug impacting only the Kubecost application.
• [X] I have searched other issues in this repository and mine is not recorded.

[AjayTripathy]

cc @mbolt35 is this a timeout contacting the k8s apiserver?

[AjayTripathy]

Hi @elcomtik we're pretty sure that this is an intermittent timeout contacting the API server on pod startup, which is why the restart/rollback appears to fix. Could you try again and see if the behavior is the same?

[elcomtik]

What version should I test?

[AjayTripathy]

v0.17.3

[zumic96]

I have added startupProbe and livenessProbe to DaemonSet to go around this issue, strangely I thought they were there by default.

livenessProbe:
  tcpSocket:
    port: 3001
  initialDelaySeconds: 5
  timeoutSeconds: 1
  periodSeconds: 10
  successThreshold: 1
  failureThreshold: 5
startupProbe:
  tcpSocket:
    port: 3001
  initialDelaySeconds: 5
  timeoutSeconds: 1
  periodSeconds: 10
  successThreshold: 1
  failureThreshold: 5

[elcomtik]

just tested v0.17.3, works good. This issue can be closed.

Thanks a lot!

[elcomtik]

I have to admit, I made premature conclusions. Still the same issue.

[elcomtik]

@zumic96 adding probes works for me too

[chipzoller]

Hello, in an effort to consolidate our bug and feature request tracking, we are deprecating using GitHub to track tickets. If this issue is still outstanding and you have not done so already, please raise a request at https://support.kubecost.com/.