open-local seemed not working properly with konnectivity

[peter-wangxu]

Ⅰ. Issue Description

Set up konnectivity server and agent, so that open-local should work well with them

Ⅱ. Describe what happened

when creating volume, the open-local controller keep failing with error as following:

I0902 14:21:53.141958       1 konnectivity.go:51] using konnectivity UDS dialer
I0902 14:21:53.141113       1 konnectivity.go:51] using konnectivity UDS dialer
E0902 14:21:53.188822       1 client.go:351] "Connection not recognized" connectionID=19 packetType="DATA"
I0902 14:21:53.220049       1 controllerserver.go:177] CreateVolume: scheduler arch of pvc(default/html-nginx-lvm-1) is extender
E0902 14:21:53.220598       1 client.go:261] "stream read failure" err="rpc error: code = Canceled desc = context canceled"
I0902 14:21:53.225078       1 controllerserver.go:195] CreateVolume: schedule LVM local-890a0606-d1be-4808-8322-faa0a6c98033 with centos9-2, open-local-pool-0
E0902 14:21:53.228094       1 grpc.go:195] Get Lvm with error: rpc error: code = Unavailable desc = connection error: desc = "transport: failed to write client preface: EOF"
E0902 14:21:53.228381       1 driver.go:128] GRPC error: rpc error: code = Internal desc = CreateVolume: fail to get lv local-890a0606-d1be-4808-8322-faa0a6c98033 from node centos9-2: rpc error: code = Unavailable desc = connection error: desc = "transport: failed to write client preface: EOF"
W0902 14:21:56.142774       1 grpc.go:139] Connection to 192.168.25.131:1736 timed out
I0902 14:21:56.142875       1 controllerserver.go:177] CreateVolume: scheduler arch of pvc(default/html-nginx-lvm-2) is extender
I0902 14:21:56.147791       1 controllerserver.go:195] CreateVolume: schedule LVM local-8efff465-ff8f-45c4-82e4-a281c4fa3c62 with centos9-2, open-local-pool-0
E0902 14:21:56.151195       1 grpc.go:195] Get Lvm with error: rpc error: code = Unavailable desc = connection error: desc = "transport: failed to write client preface: EOF"
E0902 14:21:56.152317       1 driver.go:128] GRPC error: rpc error: code = Internal desc = CreateVolume: fail to get lv local-8efff465-ff8f-45c4-82e4-a281c4fa3c62 from node centos9-2: rpc error: code = Unavailable desc = connection error: desc = "transport: failed to write client preface: EOF"

Ⅲ. Describe what you expected to happen

as my konnectivity is working well with k8s, as i can kubectl exec and log , the open-local could use the proxy as well

Ⅳ. How to reproduce it (as minimally and precisely as possible)

1. setup konnectivity server and agent via UDS and grpc
2. the k8s is working well both via kubectl exec and logs
3. create open-local lvm with stateful set workload
4. error occured as above

Ⅴ. Anything else we need to know?

Ⅵ. Environment:

• Open-Local version: lastest version
• OS (e.g. from /etc/os-release): centos 9 stream
• Kernel (e.g. uname -a): 5.14.0-354.el9.x86_64
• Install tools: helm
• Others:

[peter-wangxu]

@rafi what's your testing result around anp and open-local? can you shield some light on this？ thanks

here is my open-local config:

...
      - args:
        - csi
        - --endpoint=$(CSI_ENDPOINT)
        - --nodeID=$(KUBE_NODE_NAME)
        - --driver=local.csi.aliyun.com
        - --driver-mode=controller
        - --konnectivity-uds=/etc/kubernetes/konnectivity-server/konnectivity-server.socket
        - --konnectivity-proxy-mode=grpc
...
      - hostPath:
          path: /etc/kubernetes/konnectivity-server
          type: Directory
        name: konn-dir

[rafi]

Sorry for late response, I see you've found the culprit. Thanks!