mateiidavid
commented
2 weeks ago Hey @wamak9,
Going to walk through the errors one-by-one to explain what they mean.
1. linkerd enabled pod logs:
2024-06-06T14:03:42.566024937Z [ 61835.932270s] INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=direct connections must be mutually authenticated error.sources=[direct connections must be mutually authenticated] client.addr=10.109.192.187:34180 server.addr=10.109.194.85:4143
2024-06-06T14:03:43.072400780Z [ 61836.438591s] INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=direct connections must be mutually authenticated error.sources=[direct connections must be mutually authenticated] client.addr=10.109.194.225:43040 server.addr=10.109.194.85:4143This indicates that the client of this service (10.109.192.187:34180) is not mTLS'd but is trying to connect directly to the proxy's inbound port 4143. Only another mTLS participant can initiate connections to 4143. Typically, we might see this if the server is marked as opaque or is a multicluster gateway. If it opaque, then the client is either not injected with a proxy (in which case it should send directly to the proxy's port anyway) or misconfigured. It's hard to tell without more information.
2. linkerd destination logs:
2024-06-06T13:02:43.467367168Z time="2024-06-06T13:02:43Z" level=error msg="failed to find LINKERD2_PROXY_INBOUND_LISTEN_ADDR environment variable in any container for given pod spec" addr=":8086" component=endpoint-profile-translator context-ns=retail context-pod=tapi-api-7c47bcf658-bcpqb remote="10.109.194.212:47462An endpoint is marked as injected (probably has a control plane label) but it is not. Something went wrong where either the pod received the proxy with incomplete configuration, or the pod has been improperly annotated. This is not a typical occurence. Are you using any other features such as native sidecars?
3. linkerd proxy logs: these logs are benign. They're from the destination service's proxy and it simply signals it cannot establish a connection since the socket has been bound yet. The proxy starts first before the destination container warms up, these only happen within the first 10-20s of the proxy's lifetime.
4. policy controller logs: seems like the API server's conn is a bit wonky? I'm not sure this is necessarily related to what you're seeing. Is that a common occurence or is it just in a limited time interval when this happens.
I think it's helpful to isolate the logs that are relevant here, and that's probably the first set of logs. errno 111 simply means a connection cannot be established. Typically, it's because the socket's not listening. Sometimes you might see these out in the wild but if it's not a recurrence, or if it doesn't directly impact your traffic and services, it's safe to ignore.
From your original description:
IPs seen in the logs are releated to self-hosted Prometheus.
Where exactly? The one not being mutually authenticated, or...
For the connection refused in your linkerd enabled pod, is the server listening. Can you confirm that? Would be useful what is supposed to happen in that pod.
annotations: config.linkerd.io/default-inbound-policy: all-unauthenticated config.linkerd.io/image-pull-policy: Always config.linkerd.io/proxy-outbound-connect-timeout: "5"
On your self-hosted prometheus? Is it injected or not injected? If it's not, then the annotation won't have anything to configure. Can you do a kubectl get pods so we can see how your set-up looks like?
wamak9
What is the issue?
I keep seeing
os error 111issue on linkerd and have no idea how to fix it.More logs below. IPs seen in the logs are releated to self-hosted Prometheus. There are two sets on Prometheus running, one is from Linkerd and one is self hosted by us. Tried to add bunch of annotations
How can it be reproduced?
Current version running is stable-2.14.9. Enable linkerd on one of the namespace and then deploy. Deploy prometheus on different namepace with no linkeerd annotation.
Logs, error output, etc
Linkerd enabled POD logs.
linkerd-destination logs
linkerd Proxy logs
Linkerd Policy Logs
output of
linkerd check -o shortEnvironment
Possible solution
N/A
Additional context
So, I tried to add linkerd to Monitoring namespace which is running Prometheus. But that did not help and I am seeing way too many error logs in Prometheus on the same OS error.
I did add an authorization policy, so our self installed Prometheus can scrape metrics.
Would you like to work on fixing this bug?
maybe