kflynn
commented
8 months ago @Nutties93 Happy new year!
So, first, linkerd-policy is a headless service, so it's OK that it doesn't have a ClusterIP -- that's to be expected.
Beyond that, though, could we get a kubectl describe from the linkerd-destination pod, and the logs from its policy container?
DPOD=$(kubectl get pods -n linkerd -l 'linkerd.io/control-plane-component=destination' -o jsonpath='{ .items[0].metadata.name }')
kubectl describe -n linkerd pod $DPOD
kubectl logs -n linkerd $DPOD -c policyThanks!! 🙂
piotrrojek
faizan-planview
yishaihl
dverzolla
What is the issue?
I am facing issues with linkerd-destination and linkerd-proxy-injector. Pods that have linkerd-inject enabled are stuck in podInitializing state. I tried to restart the deployment of the destination and proxy-injector, sometimes the pods managed to be deployed. But after that, subsequent pods will be stuck at podInitializing state. My nodes have sufficient CPU and memory space.
When I run linkerd check, there are no errors as well.
How can it be reproduced?
install linked via cli on EKS.
Logs, error output, etc
Linkerd-destination logs: kubectl -n linkerd logs deploy/linkerd-destination Defaulted container "linkerd-proxy" out of: linkerd-proxy, destination, sp-validator, policy, linkerd-init (init) [ 0.003199s] INFO ThreadId(01) linkerd2_proxy: release 2.210.4 (5a910be) by linkerd on 2023-11-22T17:01:46Z [ 0.004399s] INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime [ 0.005481s] INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191 [ 0.005534s] INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143 [ 0.005539s] INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140 [ 0.005543s] INFO ThreadId(01) linkerd2_proxy: Tap interface on 0.0.0.0:4190 [ 0.005547s] INFO ThreadId(01) linkerd2_proxy: Local identity is linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local [ 0.005552s] INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local) [ 0.005556s] INFO ThreadId(01) linkerd2_proxy: Destinations resolved via localhost:8086 [ 0.007430s] WARN ThreadId(01) watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)] [ 0.036092s] INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity id=linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local [ 0.112753s] WARN ThreadId(01) watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)] [ 0.326477s] WARN ThreadId(01) watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)] [ 0.741192s] WARN ThreadId(01) watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)] [ 1.242244s] WARN ThreadId(01) watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)] [ 1.743246s] WARN ThreadId(01) watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
Linkerd-proxy-injector logs: Found 2 pods, using pod/linkerd-proxy-injector-757d76768f-h9bw2 Defaulted container "linkerd-proxy" out of: linkerd-proxy, proxy-injector, linkerd-init (init) [ 0.002738s] INFO ThreadId(01) linkerd2_proxy: release 2.210.4 (5a910be) by linkerd on 2023-11-22T17:01:46Z [ 0.003994s] INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime [ 0.004994s] INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191 [ 0.005012s] INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143 [ 0.005016s] INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140 [ 0.005020s] INFO ThreadId(01) linkerd2_proxy: Tap interface on 0.0.0.0:4190 [ 0.005023s] INFO ThreadId(01) linkerd2_proxy: Local identity is linkerd-proxy-injector.linkerd.serviceaccount.identity.linkerd.cluster.local [ 0.005026s] INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local) [ 0.005029s] INFO ThreadId(01) linkerd2_proxy: Destinations resolved via linkerd-dst-headless.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local) [ 0.022461s] INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity id=linkerd-proxy-injector.linkerd.serviceaccount.identity.linkerd.cluster.local [ 14.050531s] WARN ThreadId(01) watch{port=4191}: linkerd_app_inbound::policy::api: Unexpected policy controller response; retrying with a backoff grpc.status=The operation completed successfully grpc.message="stream ended" [ 14.050577s] WARN ThreadId(01) watch{port=8443}: linkerd_app_inbound::policy::api: Unexpected policy controller response; retrying with a backoff grpc.status=The operation completed successfully grpc.message="stream ended" [ 14.050593s] WARN ThreadId(01) watch{port=9995}: linkerd_app_inbound::policy::api: Unexpected policy controller response; retrying with a backoff grpc.status=The operation completed successfully grpc.message="stream ended" [ 14.050622s] WARN ThreadId(01) watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=172.25.1.130:8090}: linkerd_reconnect: Service failed error=endpoint 172.25.1.130:8090: channel closed error.sources=[channel closed] [ 14.158073s] WARN ThreadId(01) watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=172.25.1.130:8090}: linkerd_reconnect: Failed to connect error=endpoint 172.25.1.130:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)] [ 14.363576s] WARN ThreadId(01) watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=172.25.1.130:8090}: linkerd_reconnect: Failed to connect error=endpoint 172.25.1.130:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)] [ 14.784335s] WARN ThreadId(01) watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=172.25.1.130:8090}: linkerd_reconnect: Failed to connect error=endpoint 172.25.1.130:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)] [ 749.680863s] WARN ThreadId(01) watch{port=8443}: linkerd_app_inbound::policy::api: Unexpected policy controller response; retrying with a backoff grpc.status=The operation completed successfully grpc.message="stream ended" [ 749.680916s] WARN ThreadId(01) watch{port=4191}: linkerd_app_inbound::policy::api: Unexpected policy controller response; retrying with a backoff grpc.status=The operation completed successfully grpc.message="stream ended" [ 749.680934s] WARN ThreadId(01) watch{port=9995}: linkerd_app_inbound::policy::api: Unexpected policy controller response; retrying with a backoff grpc.status=The operation completed successfully grpc.message="stream ended" [ 750.683751s] WARN ThreadId(01) watch{port=8443}: linkerd_app_inbound::policy::api: Unexpected policy controller response; retrying with a backoff grpc.status=The service is currently unavailable grpc.message="client 172.25.1.64:34186: server: 172.25.1.82:8090: server 172.25.1.82:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service in fail-fast" [ 750.683879s] WARN ThreadId(01) watch{port=4191}: linkerd_app_inbound::policy::api: Unexpected policy controller response; retrying with a backoff grpc.status=The service is currently unavailable grpc.message="client 172.25.1.64:34186: server: 172.25.1.82:8090: server 172.25.1.82:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service in fail-fast" [ 750.683981s] WARN ThreadId(01) watch{port=9995}: linkerd_app_inbound::policy::api: Unexpected policy controller response; retrying with a backoff grpc.status=The service is currently unavailable grpc.message="client 172.25.1.64:34186: server: 172.25.1.82:8090: server 172.25.1.82:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service in fail-fast" [ 750.791438s] WARN ThreadId(01) watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=172.25.1.82:8090}: linkerd_reconnect: Service failed error=endpoint 172.25.1.82:8090: channel closed error.sources=[channel closed] [ 751.905842s] WARN ThreadId(01) watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=172.25.1.82:8090}: linkerd_reconnect: Failed to connect error=endpoint 172.25.1.82:8090: connect timed out after 1s error.sources=[connect timed out after 1s] [ 753.115467s] WARN ThreadId(01) watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=172.25.1.82:8090}: linkerd_reconnect: Failed to connect error=endpoint 172.25.1.82:8090: connect timed out after 1s error.sources=[connect timed out after 1s] [ 754.546487s] WARN ThreadId(01) watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=172.25.1.82:8090}: linkerd_reconnect: Failed to connect error=endpoint 172.25.1.82:8090: connect timed out after 1s error.sources=[connect timed out after 1s]
Control-plane metrics logs: #
POD linkerd-proxy-injector-77b4777668-k5z8b (13 of 13)
ERROR Get "http://localhost:46023/metrics": EOF
output of
linkerd check -o shortlinkerd-version
‼ can determine the latest version Get "https://versioncheck.linkerd.io/version.json?version=stable-2.14.5&uuid=89015292-fa49-4de5-90b4-280126337b83&source=cli": net/http: TLS handshake timeout see https://linkerd.io/2.14/checks/#l5d-version-latest for hints ‼ cli is up-to-date unsupported version channel: stable-2.14.5 see https://linkerd.io/2.14/checks/#l5d-version-cli for hints
control-plane-version
‼ control plane is up-to-date unsupported version channel: stable-2.14.5 see https://linkerd.io/2.14/checks/#l5d-version-control for hints
linkerd-control-plane-proxy
\ pod "linkerd-destination-765b97b794-zwn27" status is Running
Environment
EKS 1.27 Client version: stable-2.14.5 Server version: stable-2.14.5
Installation via CLI commands.
Possible solution
My suspicion is that maybe the service is not assigned local clusterIP address.
When I run k describe svc linkerd-policy -n linkerd,
Name: linkerd-policy
Namespace: linkerd
Labels: linkerd.io/control-plane-component=destination
linkerd.io/control-plane-ns=linkerd
Annotations: linkerd.io/created-by: linkerd/cli stable-2.14.5
Selector: linkerd.io/control-plane-component=destination
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: None
IPs: None
Port: grpc 8090/TCP
TargetPort: 8090/TCP
Endpoints: 172.25.1.195:8090
Session Affinity: None
Events:
Somehow the linkerd-policy endpoints are not using clusterIP but rather the CIDR of the subnet.
Additional context
Currently, i have to keep restarting the deploy of destination and proxy-injector, somehow after X tries , my pods can run. Has anybody successfully installed linkerd on EKS?
Would you like to work on fixing this bug?
no