Worker nodes behind NAT, unable to access lvmd 1736 port

[rafi]

My worker nodes are on the "edge", behind a NAT, and unreachable. However, my kube-apiserver is in the cloud, and has a public ip. The issue is communication to the lvmd port:

CLOUD (control-plane)       |        EDGE (workers)
-------                     |        -----------
                            |
 controller(csi-plugin) ----X----->  :1736
                            |
                            |

By the way, I'm also using apiserver-network-proxy (konnectivity) - is there a way to route the lvmd requests through kube-apiserver?

Is there any different solution maybe?

This is my current setup:

CLOUD:

• control-plane: api-server, scheduler, controller-manager, etc.
• open-local-scheduler-extender: (1 container)
  • scheduler --port=23000 --scheduler-strategy=spread --kubeconfig=/etc/kubernetes/kubeconfig.conf
• open-local-controller: (6 containers) (env EXTENDER_SVC_IP=open-local-scheduler-extender)
  • csi-plugin: csi --endpoint=$(CSI_ENDPOINT) --nodeID=$(KUBE_NODE_NAME) --driver=local.csi.aliyun.com --driver-mode=controller --kubeconfig=/etc/kubernetes/kubeconfig.conf
  • controller: controller --initconfig=open-local --feature-gates=UpdateNLS=true --kubeconfig=/etc/kubernetes/kubeconfig.conf
  • csi-provisioner: --csi-address=$(ADDRESS) --volume-name-prefix=local --feature-gates=Topology=True --strict-topology=True --extra-create-metadata=true --timeout=10m --kubeconfig=/etc/kubernetes/kubeconfig.conf
  • csi-resizer: --csi-address=$(ADDRESS) --kubeconfig=/etc/kubernetes/kubeconfig.conf
  • csi-snapshotter: --csi-address=$(ADDRESS) --snapshot-name-prefix=snap --kubeconfig=/etc/kubernetes/kubeconfig.conf
  • snapshot-controller: --kubeconfig=/etc/kubernetes/kubeconfig.conf

EDGE:

• open-local-agent: (3 containers)
  • csi --endpoint=$(CSI_ENDPOINT) --nodeID=$(KUBE_NODE_NAME) --driver=local.csi.aliyun.com --driver-mode=node
  • agent --nodename=$(KUBE_NODE_NAME) --path.sysfs=/host_sys --path.mount=/mnt/open-local/ --lvname=local
  • csi-node-driver-registrar: --v=5 --csi-address=/csi/csi.sock --kubelet-registration-path=/var/lib/kubelet/plugins/local.csi.aliyun.com/csi.sock

Thanks!

[peter-wangxu]

This issue partally addressed by https://github.com/alibaba/open-local/pull/207 in version v0.7.1 which edge solution are you using?

[rafi]

Is it possible to integrate konnectivity-client into open-local GRPC client?

[peter-wangxu]

Is it possible to integrate konnectivity-client into open-local GRPC client?

I am happy to see that happens,do you have anymore information?

[rafi]

Thanks for the offer, I successfully patched the grpc connection to use a different Dial that proxies the request with konnectivity-client. Would you be interested in such PR?

[peter-wangxu]

Sure, please go ahead

[rafi]

@peter-wangxu https://github.com/alibaba/open-local/pull/229