ricsanfre
commented
2 years ago Root cause is that longhorn-driver-deployer is not able to communicate with longhorn-manager when checking the MointPropagation option. The log error printed by the pod is misleading.
This error in the connectivity can be shown checking linkerd-proxy sidecar logs:
oss@node1:~$ kubectl logs longhorn-manager-x7h2t linkerd-proxy -n longhorn-system
[ 0.002680s] INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime
[ 0.005036s] INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
[ 0.005103s] INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[ 0.005116s] INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[ 0.005125s] INFO ThreadId(01) linkerd2_proxy: Tap DISABLED
[ 0.005135s] INFO ThreadId(01) linkerd2_proxy: Local identity is longhorn-service-account.longhorn-system.serviceaccount.identity.linkerd.cluster.local
[ 0.005147s] INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
[ 0.005158s] INFO ThreadId(01) linkerd2_proxy: Destinations resolved via linkerd-dst-headless.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local)
[ 0.068813s] INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity: longhorn-service-account.longhorn-system.serviceaccount.identity.linkerd.cluster.local
[ 74.392084s] INFO ThreadId(01) inbound:server{port=9500}:rescue{client.addr=10.42.3.5:37924}: linkerd_app_core::errors::respond: Request failed error=error trying to connect: Connection refused (os error 111)
[ 76.427399s] INFO ThreadId(01) inbound:server{port=9500}:rescue{client.addr=10.42.3.5:38066}: linkerd_app_core::errors::respond: Request failed error=error trying to connect: Connection refused (os error 111)
[ 78.465339s] INFO ThreadId(01) inbound:server{port=9500}:rescue{client.addr=10.42.3.5:38086}: linkerd_app_core::errors::respond: Request failed error=error trying to connect: Connection refused (os error 111)
[ 80.505249s] INFO ThreadId(01) inbound:server{port=9500}:rescue{client.addr=10.42.3.5:38094}: linkerd_app_core::errors::respond: Request failed error=error trying to connect: Connection refused (os error 111)Last log lines shows that inbound connections to longhorn-manager (tcp port 9500) are being refused at OS level. Linkerd is trying to route the traffic coming from longhorn-driver-deployer but the connection is refused by longhorn-manager process.
The problem is that linkerd-proxy when forwarding the incoming traffic uses as source IP of the connection localhost but longhorn-manager process is only listening at POD IP address, not at localhost and so the connection is refused.
This commands show longhorn-manager container opening Port 9500 only at the POD IP (10.42.3.6).
oss@node1:~$ kubectl exec -it longhorn-manager-x7h2t -c longhorn-manager -n longhorn-system -- /bin/bash
root@longhorn-manager-x7h2t:/# ss -tunlp
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
tcp LISTEN 0 4096 127.0.0.1:6060 0.0.0.0:* users:(("longhorn-manage",pid=1,fd=39))
tcp LISTEN 0 128 127.0.0.1:4140 0.0.0.0:*
tcp LISTEN 0 128 0.0.0.0:4143 0.0.0.0:*
tcp LISTEN 0 4096 10.42.3.6:9500 0.0.0.0:* users:(("longhorn-manage",pid=1,fd=41))
tcp LISTEN 0 128 0.0.0.0:4191 0.0.0.0:*
Issue Description
When installing Longhorn in a namespace annotated to inject automatically linkerd-proxy (implicit annotation), Longhorn installation hangs do not deploying CSI plugin components.
longhorn-driver-deployerpod keep son restarting and crashing every time with the same error "Got an error when checking MountProgapgation with node status, Node XX is not support mount propagation" and "Error deploying driver: CSI cannot be deployed because MountPropagation is not set: Node XX is not support mount propagation"longhorn-dirver-deployer logs:
longhorn-manager logs does not show any informative log