[BUG] Manager listens only to one address in dual-stack cluster

[poksiala]

Describe the bug In dual-stack cluster pod status might have multiple IPs

$ kubectl describe pod longhorn-manager-wg867
Name:                 longhorn-manager-wg867
...
Status:               Running
IP:                   10.1.197.205
IPs:
  IP:           10.1.197.205
  IP:           fd01::11e9:d415:68f4:c5ca

But longhorn-manager only listens to one of them as per

    Environment:
      POD_IP:                 (v1:status.podIP)

For this reason other pieces of the puzzle fail to connect to the pod as they receive IPv6 address from DNS.

To Reproduce Steps to reproduce the behavior:

1. Set up dual-stack kubernetes cluster with service preference to IPv6 over IPv4
2. Deploy longhorn 1.1.0 helm chart
3. Observe wait-manager tasks waiting for ever.
4. Observe ui not able to make contact with API. Nginx logs are filled with timeouts to manager ipv6 address

Expected behavior Manager should listen to all pod IPs and the system should start

Log

time="2021-04-07T17:02:20Z" level=info msg="Listening on 10.1.197.205:9500"

Environment:

• Longhorn version: 1.1.0
• Installation method (e.g. Rancher Catalog App/Helm/Kubectl): Helm
• Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: 3 node HA microk8s
• Node config
  • OS type and version: Ubuntu Server 21.04
  • CPU per node: 4
  • Memory per node: 16
  • Disk type(e.g. SSD/NVMe): NVMe and SSD
  • Network bandwidth between the nodes: 2Gbps
• Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): Baremetal
• Number of Longhorn volumes in the cluster: 0 thus far

Additional context Proper fix would most likely be to alter manager to listen on all IPs.

env:
- name: POD_IPS
  valueFrom:
    fieldRef:
      fieldPath: status.podIPs

I fixed this for my own cluster by defining services as IPv4 single-stack. See https://github.com/longhorn/longhorn/pull/2451

[joshimoo]

related ipv6 support #2259 #1723

[jenting]

Thank you for the PR, I left some comments.

[poksiala]

Just to reaffirm: The PR made by me is not a proper fix to the issue, but more like a workaround that works in dual-stack clusters.

[jenting]

Waits for other team members to review.

[innobead]

@poksiala just wants to confirm your use case: do you only want to use the IPv6 address to access the UI service? because besides UI, there are still some things we need to figure out first for fully supporting IPv6 in the backend components, not just UI/Manager.

I would prefer we have a complete story for the dual-stack supported.