`csi-rbdplugin` and `csi-cephfsplugin` panic on new nodes because `ControllerGetCapabilities` is called

[yuaanlin]

Describe the bug

I have a 2-node Kubernetes cluster on Huawei Cloud using rook-ceph and rbdplugin as the CSI driver, which has been running stably for several months. Recently, I added a new node to the cluster and found that both csi-rbdplugin and csi-cephfsplugin crashed and failed to start on the new node.

Environment details

• Image/version of Ceph CSI driver: v3.11.0
• Helm chart version: Not using Helm chart, deploy from yaml files directly
• Kernel version: 5.10.0-60.18.0.50.r1083_58.hce2.x86_64
• Mounter used for mounting PVC: rbd-nbd
• Kubernetes cluster version: v1.28.6
• Ceph cluster version: v1.14.4

Steps to reproduce

1. Setup details: 2-node Kubernetes cluster on Huawei Cloud using rook-ceph and rbdplugin as the CSI driver.
2. Add a new node to the cluster.
3. Observe the crash and failure to start of csi-rbdplugin and csi-cephfsplugin on the new node.

Actual results

The csi-rbdplugin and csi-cephfsplugin containers crash and fail to start on the newly added node.

Expected behavior

The csi-rbdplugin and csi-cephfsplugin containers should start and run stably on the new node, as they do on the existing nodes.

Logs

Logs from the driver-registrar container of csi-rbdplugin:

driver-registrar 8:00 I0814 12:47:51.440405       1 main.go:135] Version: v2.10.1
driver-registrar 8:00 I0814 12:47:51.440486       1 main.go:136] Running node-driver-registrar in mode=
driver-registrar 8:00 I0814 12:47:51.442707       1 node_register.go:55] Starting Registration Server at: /registration/rook-ceph.cephfs.csi.ceph.com-reg.sock
driver-registrar 8:00 I0814 12:47:51.442814       1 node_register.go:64] Registration Server started at: /registration/rook-ceph.cephfs.csi.ceph.com-reg.sock
driver-registrar 8:00 I0814 12:47:51.442928       1 node_register.go:88] Skipping HTTP server because endpoint is set to: ""
driver-registrar 8:00 I0814 12:47:52.639825       1 main.go:90] Received GetInfo call: &InfoRequest{}
driver-registrar 8:00 I0814 12:47:52.641597       1 main.go:101] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:false,Error:RegisterPlugin error -- plugin registration failed with err: failed to get controller capabilities from driver rook-ceph.cephfs.csi.ceph.com: rpc error: code = Internal desc = panic runtime error: invalid memory address or nil pointer dereference,}
driver-registrar 8:00 E0814 12:47:52.641635       1 main.go:103] Registration process failed with error: RegisterPlugin error -- plugin registration failed with err: failed to get controller capabilities from driver rook-ceph.cephfs.csi.ceph.com: rpc error: code = Internal desc = panic runtime error: invalid memory address or nil pointer dereference, restarting registration container.
driver-registrar Stream closed: EOF

Logs from the csi-cephfsplugin container:

csi-cephfsplugin 8:00 E0814 12:47:08.620469       1 utils.go:220] panic occurred: runtime error: invalid memory address or nil pointer dereference
csi-cephfsplugin 8:00 goroutine 117 [running]:
csi-cephfsplugin 8:00 runtime/debug.Stack()
csi-cephfsplugin 8:00   /usr/local/go/src/runtime/debug/stack.go:24 +0x5e
csi-cephfsplugin 8:00 runtime/debug.PrintStack()
csi-cephfsplugin 8:00   /usr/local/go/src/runtime/debug/stack.go:16 +0x13
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.panicHandler.func1()
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:221 +0xca
csi-cephfsplugin 8:00 panic({0x1ea86e0?, 0x3a72420?})
csi-cephfsplugin 8:00   /usr/local/go/src/runtime/panic.go:914 +0x21f
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/cephfs.(*ControllerServer).ControllerGetCapabilities(0x0?, {0x27c0450?, 0xc0008d9620?}, 0xc0008a3710?)
csi-cephfsplugin 8:00   <autogenerated>:1 +0x21
csi-cephfsplugin 8:00 github.com/container-storage-interface/spec/lib/go/csi._Controller_ControllerGetCapabilities_Handler.func1({0x27c0450, 0xc0008d9620}, {0x2055620?, 0xc000489e00})
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:6713 +0x72
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.panicHandler({0x27c0450?, 0xc0008d9620?}, {0x2055620?, 0xc000489e00?}, 0xc000599038?, 0xc000000101?)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:226 +0x77
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.NewMiddlewareServerOption.ChainUnaryServer.func2.1({0x27c0450?, 0xc0008d9620?}, {0x2055620?, 0xc000489e00?})
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:48 +0x45
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.logGRPC({0x27c0450, 0xc0008d9620}, {0x2055620?, 0xc000489e00?}, 0x1d765e0?, 0xc000729180)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:201 +0x183
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.NewMiddlewareServerOption.ChainUnaryServer.func2.1({0x27c0450?, 0xc0008d9620?}, {0x2055620?, 0xc000489e00?})
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:48 +0x45
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.contextIDInjector({0x27c0450, 0xc0008d94a0}, {0x2055620, 0xc000489e00}, 0xc0004fc858?, 0xc0007291c0)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:189 +0x132
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.NewMiddlewareServerOption.ChainUnaryServer.func2({0x27c0450, 0xc0008d94a0}, {0x2055620, 0xc000489e00}, 0xc000489e20, 0x1ea94c0?)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:53 +0x135
csi-cephfsplugin 8:00 github.com/container-storage-interface/spec/lib/go/csi._Controller_ControllerGetCapabilities_Handler({0x216f9a0?, 0x0}, {0x27c0450, 0xc0008d94a0}, 0xc000730c00, 0xc0008d8c00)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:6715 +0x135
csi-cephfsplugin 8:00 google.golang.org/grpc.(*Server).processUnaryRPC(0xc0001ed400, {0x27c0450, 0xc0008d9410}, {0x27c8d40, 0xc000503a00}, 0xc0007f30e0, 0xc0008d8de0, 0x3a89a48, 0x0)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1386 +0xe23

Additional context

I tried adding debug prints to the source code and found that the panic was caused by calling the ControllerGetCapabilities RPC in the csi-cephfsplugin.

I am not very familiar with the CSI principles, but I believe this method should only be used in the controller, not in the nodeserver.

Interestingly, I added logs to the ControllerGetCapabilities method and found that this method is only used on the new node, while it is not used on the two existing nodes.

To temporarily fix the issue, I crudely modified the _Controller_ControllerGetCapabilities_Handler method in vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:

func _Controller_ControllerGetCapabilities_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
    in := new(ControllerGetCapabilitiesRequest)
    if err := dec(in); err != nil {
        return nil, err
    }
    if interceptor == nil {
        return srv.(ControllerServer).ControllerGetCapabilities(ctx, in)
    }
    info := &grpc.UnaryServerInfo{
        Server:     srv,
        FullMethod: "/csi.v1.Controller/ControllerGetCapabilities",
    }
    handler := func(ctx context.Context, req interface{}) (interface{}, error) {
        return nil, nil
        // return srv.(ControllerServer).ControllerGetCapabilities(ctx, req.(*ControllerGetCapabilitiesRequest))
    }
    return interceptor(ctx, in, info, handler)
}

I replaced the original return srv.(ControllerServer).ControllerGetCapabilities(ctx, req.(*ControllerGetCapabilitiesRequest)) (which directly caused the panic) with return nil, nil, and the container started successfully.

driver-registrar 8:00 I0814 15:12:42.611713  116281 main.go:135] Version: v2.10.1
driver-registrar 8:00 I0814 15:12:42.611762  116281 main.go:136] Running node-driver-registrar in mode=
driver-registrar 8:00 I0814 15:12:44.640847  116281 node_register.go:55] Starting Registration Server at: /registration/rook-ceph.rbd.csi.ceph.com-reg.sock
driver-registrar 8:00 I0814 15:12:44.640987  116281 node_register.go:64] Registration Server started at: /registration/rook-ceph.rbd.csi.ceph.com-reg.sock
driver-registrar 8:00 I0814 15:12:44.641059  116281 node_register.go:88] Skipping HTTP server because endpoint is set to: ""
driver-registrar 8:00 I0814 15:12:44.671057  116281 main.go:90] Received GetInfo call: &InfoRequest{}
driver-registrar 8:00 I0814 15:12:44.673127  116281 main.go:101] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:true,Error:,}
driver-registrar 8:00 I0814 15:14:00.190150  116281 main.go:90] Received GetInfo call: &InfoRequest{}
driver-registrar 8:00 I0814 15:14:00.219751  116281 main.go:101] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:true,Error:,}

However, this is clearly not the correct way to fix the issue. I am opening this issue in the hope of getting help and assisting others who may encounter a similar problem.

I believe the solution may lie in investigating why the plugin on the new node is calling the ControllerGetCapabilities method, while it is not being called on the original two nodes.

[nixpanic]

@yuaanlin, can you share a little more of the logs of the csi-cephfsplugin container, from before the panic occurs? That ideally give a little more hints on what was sent/received.

ControllerGetCapabilities is a standard call that is used by the container platform to check what features the CSI-driver supports. It is expected to be called when the CSI-driver has just been started.

[Madhu-1]

@yuaanlin also please provide the yaml of the pod which is having the problem

[yuaanlin]

@nixpanic

can you share a little more of the logs of the csi-cephfsplugin container, from before the panic occurs? That ideally give a little more hints on what was sent/received.

There is no logs before csi-cephfsplugin container panic.

driver-registrar 8:00 I0814 12:47:51.440405       1 main.go:135] Version: v2.10.1
driver-registrar 8:00 I0814 12:47:51.440486       1 main.go:136] Running node-driver-registrar in mode=
driver-registrar 8:00 I0814 12:47:51.442707       1 node_register.go:55] Starting Registration Server at: /registration/rook-ceph.cephfs.csi.ceph.com-reg.sock
driver-registrar 8:00 I0814 12:47:51.442814       1 node_register.go:64] Registration Server started at: /registration/rook-ceph.cephfs.csi.ceph.com-reg.sock
driver-registrar 8:00 I0814 12:47:51.442928       1 node_register.go:88] Skipping HTTP server because endpoint is set to: ""
driver-registrar 8:00 I0814 12:47:52.639825       1 main.go:90] Received GetInfo call: &InfoRequest{}
driver-registrar 8:00 I0814 12:47:52.641597       1 main.go:101] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:false,Error:RegisterPlugin error -- plugin registration failed with err: failed to get controller capabilities from driver rook-ceph.cephfs.csi.ceph.com: rpc error: code = Internal desc = panic runtime error: invalid memory address or nil pointer dereference,}
driver-registrar 8:00 E0814 12:47:52.641635       1 main.go:103] Registration process failed with error: RegisterPlugin error -- plugin registration failed with err: failed to get controller capabilities from driver rook-ceph.cephfs.csi.ceph.com: rpc error: code = Internal desc = panic runtime error: invalid memory address or nil pointer dereference, restarting registration container.
driver-registrar Stream closed

csi-cephfsplugin 8:00 E0814 12:47:08.620469       1 utils.go:220] panic occurred: runtime error: invalid memory address or nil pointer dereference
csi-cephfsplugin 8:00 goroutine 117 [running]:
csi-cephfsplugin 8:00 runtime/debug.Stack()
csi-cephfsplugin 8:00   /usr/local/go/src/runtime/debug/stack.go:24 +0x5e
csi-cephfsplugin 8:00 runtime/debug.PrintStack()
csi-cephfsplugin 8:00   /usr/local/go/src/runtime/debug/stack.go:16 +0x13
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.panicHandler.func1()
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:221 +0xca
csi-cephfsplugin 8:00 panic({0x1ea86e0?, 0x3a72420?})
csi-cephfsplugin 8:00   /usr/local/go/src/runtime/panic.go:914 +0x21f
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/cephfs.(*ControllerServer).ControllerGetCapabilities(0x0?, {0x27c0450?, 0xc0008d9620?}, 0xc0008a3710?)
csi-cephfsplugin 8:00   <autogenerated>:1 +0x21
csi-cephfsplugin 8:00 github.com/container-storage-interface/spec/lib/go/csi._Controller_ControllerGetCapabilities_Handler.func1({0x27c0450, 0xc0008d9620}, {0x2055620?, 0xc000489e00})
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:6713 +0x72
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.panicHandler({0x27c0450?, 0xc0008d9620?}, {0x2055620?, 0xc000489e00?}, 0xc000599038?, 0xc000000101?)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:226 +0x77
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.NewMiddlewareServerOption.ChainUnaryServer.func2.1({0x27c0450?, 0xc0008d9620?}, {0x2055620?, 0xc000489e00?})
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:48 +0x45
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.logGRPC({0x27c0450, 0xc0008d9620}, {0x2055620?, 0xc000489e00?}, 0x1d765e0?, 0xc000729180)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:201 +0x183
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.NewMiddlewareServerOption.ChainUnaryServer.func2.1({0x27c0450?, 0xc0008d9620?}, {0x2055620?, 0xc000489e00?})
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:48 +0x45
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.contextIDInjector({0x27c0450, 0xc0008d94a0}, {0x2055620, 0xc000489e00}, 0xc0004fc858?, 0xc0007291c0)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:189 +0x132
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.NewMiddlewareServerOption.ChainUnaryServer.func2({0x27c0450, 0xc0008d94a0}, {0x2055620, 0xc000489e00}, 0xc000489e20, 0x1ea94c0?)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:53 +0x135
csi-cephfsplugin 8:00 github.com/container-storage-interface/spec/lib/go/csi._Controller_ControllerGetCapabilities_Handler({0x216f9a0?, 0x0}, {0x27c0450, 0xc0008d94a0}, 0xc000730c00, 0xc0008d8c00)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:6715 +0x135
csi-cephfsplugin 8:00 google.golang.org/grpc.(*Server).processUnaryRPC(0xc0001ed400, {0x27c0450, 0xc0008d9410}, {0x27c8d40, 0xc000503a00}, 0xc0007f30e0, 0xc0008d8de0, 0x3a89a48, 0x0)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1386 +0xe23
csi-cephfsplugin 8:00 google.golang.org/grpc.(*Server).handleStream(0xc0001ed400, {0x27c8d40, 0xc000503a00}, 0xc0007f30e0)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1797 +0x100c
csi-cephfsplugin 8:00 google.golang.org/grpc.(*Server).serveStreams.func2.1()
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1027 +0x8b
csi-cephfsplugin 8:00 created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 106
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1038 +0x135
csi-cephfsplugin 8:00 E0814 12:47:08.620699       1 utils.go:203] ID: 3 GRPC error: rpc error: code = Internal desc = panic runtime error: invalid memory address or nil pointer dereference
csi-cephfsplugin 8:00 E0814 12:47:24.627410       1 utils.go:220] panic occurred: runtime error: invalid memory address or nil pointer dereference
csi-cephfsplugin 8:00 goroutine 24 [running]:
csi-cephfsplugin 8:00 runtime/debug.Stack()
csi-cephfsplugin 8:00   /usr/local/go/src/runtime/debug/stack.go:24 +0x5e
csi-cephfsplugin 8:00 runtime/debug.PrintStack()
csi-cephfsplugin 8:00   /usr/local/go/src/runtime/debug/stack.go:16 +0x13
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.panicHandler.func1()
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:221 +0xca
csi-cephfsplugin 8:00 panic({0x1ea86e0?, 0x3a72420?})
csi-cephfsplugin 8:00   /usr/local/go/src/runtime/panic.go:914 +0x21f
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/cephfs.(*ControllerServer).ControllerGetCapabilities(0x0?, {0x27c0450?, 0xc000a10600?}, 0xc0005d1710?)
csi-cephfsplugin 8:00   <autogenerated>:1 +0x21
csi-cephfsplugin 8:00 github.com/container-storage-interface/spec/lib/go/csi._Controller_ControllerGetCapabilities_Handler.func1({0x27c0450, 0xc000a10600}, {0x2055620?, 0xc0000a4340})
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:6713 +0x72
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.panicHandler({0x27c0450?, 0xc000a10600?}, {0x2055620?, 0xc0000a4340?}, 0xc000118138?, 0xc000000101?)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:226 +0x77
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.NewMiddlewareServerOption.ChainUnaryServer.func2.1({0x27c0450?, 0xc000a10600?}, {0x2055620?, 0xc0000a4340?})
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:48 +0x45
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.logGRPC({0x27c0450, 0xc000a10600}, {0x2055620?, 0xc0000a4340?}, 0x1d765e0?, 0xc000a16240)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:201 +0x183
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.NewMiddlewareServerOption.ChainUnaryServer.func2.1({0x27c0450?, 0xc000a10600?}, {0x2055620?, 0xc0000a4340?})
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:48 +0x45
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.contextIDInjector({0x27c0450, 0xc000a105d0}, {0x2055620, 0xc0000a4340}, 0xc000012fd8?, 0xc000a16280)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:189 +0x132
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.NewMiddlewareServerOption.ChainUnaryServer.func2({0x27c0450, 0xc000a105d0}, {0x2055620, 0xc0000a4340}, 0xc0000a4360, 0x1ea94c0?)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:53 +0x135
csi-cephfsplugin 8:00 github.com/container-storage-interface/spec/lib/go/csi._Controller_ControllerGetCapabilities_Handler({0x216f9a0?, 0x0}, {0x27c0450, 0xc000a105d0}, 0xc000a2c100, 0xc0008d8c00)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:6715 +0x135
csi-cephfsplugin 8:00 google.golang.org/grpc.(*Server).processUnaryRPC(0xc0001ed400, {0x27c0450, 0xc000a10540}, {0x27c8d40, 0xc000102d00}, 0xc000595e60, 0xc0008d8de0, 0x3a89a48, 0x0)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1386 +0xe23
csi-cephfsplugin 8:00 google.golang.org/grpc.(*Server).handleStream(0xc0001ed400, {0x27c8d40, 0xc000102d00}, 0xc000595e60)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1797 +0x100c
csi-cephfsplugin 8:00 google.golang.org/grpc.(*Server).serveStreams.func2.1()
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1027 +0x8b
csi-cephfsplugin 8:00 created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 110
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1038 +0x135
csi-cephfsplugin 8:00 E0814 12:47:24.627540       1 utils.go:203] ID: 6 GRPC error: rpc error: code = Internal desc = panic runtime error: invalid memory address or nil pointer dereference
csi-cephfsplugin 8:00 E0814 12:47:52.641126       1 utils.go:220] panic occurred: runtime error: invalid memory address or nil pointer dereference
csi-cephfsplugin 8:00 goroutine 131 [running]:
csi-cephfsplugin 8:00 runtime/debug.Stack()
csi-cephfsplugin 8:00   /usr/local/go/src/runtime/debug/stack.go:24 +0x5e
csi-cephfsplugin 8:00 runtime/debug.PrintStack()
csi-cephfsplugin 8:00   /usr/local/go/src/runtime/debug/stack.go:16 +0x13
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.panicHandler.func1()
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:221 +0xca
csi-cephfsplugin 8:00 panic({0x1ea86e0?, 0x3a72420?})
csi-cephfsplugin 8:00   /usr/local/go/src/runtime/panic.go:914 +0x21f
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/cephfs.(*ControllerServer).ControllerGetCapabilities(0x0?, {0x27c0450?, 0xc0009166c0?}, 0xc0005c5710?)
csi-cephfsplugin 8:00   <autogenerated>:1 +0x21
csi-cephfsplugin 8:00 github.com/container-storage-interface/spec/lib/go/csi._Controller_ControllerGetCapabilities_Handler.func1({0x27c0450, 0xc0009166c0}, {0x2055620?, 0xc00059ba00})
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:6713 +0x72
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.panicHandler({0x27c0450?, 0xc0009166c0?}, {0x2055620?, 0xc00059ba00?}, 0xc0007aeb88?, 0xc000000101?)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:226 +0x77
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.NewMiddlewareServerOption.ChainUnaryServer.func2.1({0x27c0450?, 0xc0009166c0?}, {0x2055620?, 0xc00059ba00?})
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:48 +0x45
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.logGRPC({0x27c0450, 0xc0009166c0}, {0x2055620?, 0xc00059ba00?}, 0x1d765e0?, 0xc00069d740)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:201 +0x183
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.NewMiddlewareServerOption.ChainUnaryServer.func2.1({0x27c0450?, 0xc0009166c0?}, {0x2055620?, 0xc00059ba00?})
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:48 +0x45
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.contextIDInjector({0x27c0450, 0xc000916690}, {0x2055620, 0xc00059ba00}, 0xc000530d80?, 0xc00069d780)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:189 +0x132
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.NewMiddlewareServerOption.ChainUnaryServer.func2({0x27c0450, 0xc000916690}, {0x2055620, 0xc00059ba00}, 0xc00059ba20, 0x1ea94c0?)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:53 +0x135
csi-cephfsplugin 8:00 github.com/container-storage-interface/spec/lib/go/csi._Controller_ControllerGetCapabilities_Handler({0x216f9a0?, 0x0}, {0x27c0450, 0xc000916690}, 0xc00065af80, 0xc0008d8c00)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:6715 +0x135
csi-cephfsplugin 8:00 google.golang.org/grpc.(*Server).processUnaryRPC(0xc0001ed400, {0x27c0450, 0xc000916600}, {0x27c8d40, 0xc000103380}, 0xc000635320, 0xc0008d8de0, 0x3a89a48, 0x0)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1386 +0xe23
csi-cephfsplugin 8:00 google.golang.org/grpc.(*Server).handleStream(0xc0001ed400, {0x27c8d40, 0xc000103380}, 0xc000635320)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1797 +0x100c
csi-cephfsplugin 8:00 google.golang.org/grpc.(*Server).serveStreams.func2.1()
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1027 +0x8b
csi-cephfsplugin 8:00 created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 130
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1038 +0x135
csi-cephfsplugin 8:00 E0814 12:47:52.641277       1 utils.go:203] ID: 9 GRPC error: rpc error: code = Internal desc = panic runtime error: invalid memory address or nil pointer dereference
csi-cephfsplugin 8:00 E0814 12:48:43.667474       1 utils.go:220] panic occurred: runtime error: invalid memory address or nil pointer dereference
csi-cephfsplugin 8:00 goroutine 149 [running]:
csi-cephfsplugin 8:00 runtime/debug.Stack()
csi-cephfsplugin 8:00   /usr/local/go/src/runtime/debug/stack.go:24 +0x5e
csi-cephfsplugin 8:00 runtime/debug.PrintStack()
csi-cephfsplugin 8:00   /usr/local/go/src/runtime/debug/stack.go:16 +0x13
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.panicHandler.func1()
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:221 +0xca
csi-cephfsplugin 8:00 panic({0x1ea86e0?, 0x3a72420?})
csi-cephfsplugin 8:00   /usr/local/go/src/runtime/panic.go:914 +0x21f
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/cephfs.(*ControllerServer).ControllerGetCapabilities(0x0?, {0x27c0450?, 0xc00083f7a0?}, 0xc0005d5710?)
csi-cephfsplugin 8:00   <autogenerated>:1 +0x21
csi-cephfsplugin 8:00 github.com/container-storage-interface/spec/lib/go/csi._Controller_ControllerGetCapabilities_Handler.func1({0x27c0450, 0xc00083f7a0}, {0x2055620?, 0xc00079d460})
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:6713 +0x72
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.panicHandler({0x27c0450?, 0xc00083f7a0?}, {0x2055620?, 0xc00079d460?}, 0xc000872048?, 0xc000000101?)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:226 +0x77
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.NewMiddlewareServerOption.ChainUnaryServer.func2.1({0x27c0450?, 0xc00083f7a0?}, {0x2055620?, 0xc00079d460?})
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:48 +0x45
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.logGRPC({0x27c0450, 0xc00083f7a0}, {0x2055620?, 0xc00079d460?}, 0x1d765e0?, 0xc0008502c0)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:201 +0x183
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.NewMiddlewareServerOption.ChainUnaryServer.func2.1({0x27c0450?, 0xc00083f7a0?}, {0x2055620?, 0xc00079d460?})
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:48 +0x45
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.contextIDInjector({0x27c0450, 0xc00083f770}, {0x2055620, 0xc00079d460}, 0xc0007dc678?, 0xc000850300)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:189 +0x132
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.NewMiddlewareServerOption.ChainUnaryServer.func2({0x27c0450, 0xc00083f770}, {0x2055620, 0xc00079d460}, 0xc00079d480, 0x1ea94c0?)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:53 +0x135
csi-cephfsplugin 8:00 github.com/container-storage-interface/spec/lib/go/csi._Controller_ControllerGetCapabilities_Handler({0x216f9a0?, 0x0}, {0x27c0450, 0xc00083f770}, 0xc000835d00, 0xc0008d8c00)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:6715 +0x135
csi-cephfsplugin 8:00 google.golang.org/grpc.(*Server).processUnaryRPC(0xc0001ed400, {0x27c0450, 0xc00083f6e0}, {0x27c8d40, 0xc000753860}, 0xc0006deea0, 0xc0008d8de0, 0x3a89a48, 0x0)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1386 +0xe23
csi-cephfsplugin 8:00 google.golang.org/grpc.(*Server).handleStream(0xc0001ed400, {0x27c8d40, 0xc000753860}, 0xc0006deea0)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1797 +0x100c
csi-cephfsplugin 8:00 google.golang.org/grpc.(*Server).serveStreams.func2.1()
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1027 +0x8b
csi-cephfsplugin 8:00 created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 148
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1038 +0x135
csi-cephfsplugin 8:00 E0814 12:48:43.667606       1 utils.go:203] ID: 13 GRPC error: rpc error: code = Internal desc = panic runtime error: invalid memory address or nil pointer dereference

However, with my debug prints, I can confirm that csi-cephfsplugin is crashed because ControllerGetCapabilities RPC is called.

func _Controller_ControllerGetCapabilities_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
    in := new(ControllerGetCapabilitiesRequest)
    if err := dec(in); err != nil {
        return nil, err
    }
    if interceptor == nil {
        return srv.(ControllerServer).ControllerGetCapabilities(ctx, in)
    }
    info := &grpc.UnaryServerInfo{
        Server:     srv,
        FullMethod: "/csi.v1.Controller/ControllerGetCapabilities",
    }
    handler := func(ctx context.Context, req interface{}) (interface{}, error) {
             return srv.(ControllerServer).ControllerGetCapabilities(ctx, req.(*ControllerGetCapabilitiesRequest))
    }
    return interceptor(ctx, in, info, handler)
}

When this is called, handler will return this:

return srv.(ControllerServer).ControllerGetCapabilities(ctx, req.(*ControllerGetCapabilitiesRequest))

But srv is nil, so container panic.

ControllerGetCapabilities is a standard call that is used by the container platform to check what features the CSI-driver supports. It is expected to be called when the CSI-driver has just been started.

If ControllerGetCapabilities should be called when CSI plugin started, I think the question is:

1. Why pods on the old two nodes not being called ControllerGetCapabilities method, so they are not panic?
2. Why srv in _Controller_ControllerGetCapabilities_Handler is nil?

[yuaanlin]

@Madhu-1

also please provide the yaml of the pod which is having the problem

apiVersion: apps/v1
kind: DaemonSet
metadata:
  annotations:
    deprecated.daemonset.template.generation: "24"
  creationTimestamp: 2024-05-17T16:14:50Z
  generation: 24
  name: csi-cephfsplugin
  namespace: rook-ceph
  ownerReferences:
    - apiVersion: apps/v1
      blockOwnerDeletion: false
      controller: true
      kind: Deployment
      name: rook-ceph-operator
      uid: d5c6b5eb-7f27-4157-a75d-bdc2f59b6824
  resourceVersion: "36983923"
  uid: a0032c45-2ad4-4cc7-a002-7a5c9bfced3d
spec:
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: csi-cephfsplugin
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: csi-cephfsplugin
        contains: csi-cephfsplugin-metrics
    spec:
      affinity:
        nodeAffinity: {}
      containers:
        - args:
            - --v=0
            - --csi-address=/csi/csi.sock
            - --kubelet-registration-path=/mnt/paas/kubernetes/kubelet/plugins/rook-ceph.cephfs.csi.ceph.com/csi.sock
          env:
            - name: KUBE_NODE_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: spec.nodeName
          image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.10.1
          imagePullPolicy: IfNotPresent
          name: driver-registrar
          resources: {}
          securityContext:
            capabilities:
              drop:
                - ALL
            privileged: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /csi
              name: plugin-dir
            - mountPath: /registration
              name: registration-dir
        - args:
            - --nodeid=$(NODE_ID)
            - --type=cephfs
            - --endpoint=$(CSI_ENDPOINT)
            - --v=0
            - --nodeserver=true
            - --drivername=rook-ceph.cephfs.csi.ceph.com
            - --pidlimit=-1
            - --forcecephkernelclient=true
          env:
            - name: POD_IP
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: status.podIP
            - name: NODE_ID
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: spec.nodeName
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
            - name: CSI_ENDPOINT
              value: unix:///csi/csi.sock
          image: quay.io/cephcsi/cephcsi:v3.11.0
          imagePullPolicy: IfNotPresent
          name: csi-cephfsplugin
          resources: {}
          securityContext:
            allowPrivilegeEscalation: true
            capabilities:
              add:
                - SYS_ADMIN
              drop:
                - ALL
            privileged: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /csi
              name: plugin-dir
            - mountPath: /csi/mountinfo
              name: ceph-csi-mountinfo
            - mountPath: /mnt/paas/kubernetes/kubelet/plugins
              mountPropagation: Bidirectional
              name: csi-plugins-dir
            - mountPath: /mnt/paas/kubernetes/kubelet/pods
              mountPropagation: Bidirectional
              name: pods-mount-dir
            - mountPath: /sys
              name: host-sys
            - mountPath: /lib/modules
              name: lib-modules
              readOnly: true
            - mountPath: /dev
              name: host-dev
            - mountPath: /etc/ceph-csi-config/
              name: ceph-csi-config
            - mountPath: /tmp/csi/keys
              name: keys-tmp-dir
            - mountPath: /run/mount
              name: host-run-mount
      dnsPolicy: ClusterFirstWithHostNet
      hostNetwork: true
      priorityClassName: system-node-critical
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: rook-csi-cephfs-plugin-sa
      serviceAccountName: rook-csi-cephfs-plugin-sa
      terminationGracePeriodSeconds: 30
      volumes:
        - hostPath:
            path: /mnt/paas/kubernetes/kubelet/plugins/rook-ceph.cephfs.csi.ceph.com/mountinfo
            type: DirectoryOrCreate
          name: ceph-csi-mountinfo
        - hostPath:
            path: /mnt/paas/kubernetes/kubelet/plugins/rook-ceph.cephfs.csi.ceph.com/
            type: DirectoryOrCreate
          name: plugin-dir
        - hostPath:
            path: /mnt/paas/kubernetes/kubelet/plugins
            type: Directory
          name: csi-plugins-dir
        - hostPath:
            path: /mnt/paas/kubernetes/kubelet/plugins_registry/
            type: Directory
          name: registration-dir
        - hostPath:
            path: /mnt/paas/kubernetes/kubelet/pods
            type: Directory
          name: pods-mount-dir
        - hostPath:
            path: /sys
            type: ""
          name: host-sys
        - hostPath:
            path: /lib/modules
            type: ""
          name: lib-modules
        - hostPath:
            path: /dev
            type: ""
          name: host-dev
        - configMap:
            defaultMode: 420
            items:
              - key: csi-cluster-config-json
                path: config.json
            name: rook-ceph-csi-config
          name: ceph-csi-config
        - emptyDir:
            medium: Memory
          name: keys-tmp-dir
        - hostPath:
            path: /run/mount
            type: ""
          name: host-run-mount
  updateStrategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
status:
  currentNumberScheduled: 2
  desiredNumberScheduled: 2
  numberAvailable: 2
  numberMisscheduled: 0
  numberReady: 2
  observedGeneration: 24
  updatedNumberScheduled: 2

[Madhu-1]

@yuaanlin as per the above yaml, there are 2 pods available and both are running. How is this new pod coming up, If it was part of this daemonset I expect the count to match 3, not 2. am i missing anything?

[yuaanlin]

@Madhu-1 The yaml is copy and modify from current status, so the number is wrong.

When the problem occurred, count should be 3 pods: 2 running, 1 crashLoopBackOff.

[nixpanic]

@yuaanlin , can you increase the logging --v=0 parameter for the csi-cephfsplugin container and see if that results in more logs?

[yuaanlin]

@nixpanic Sure!

driver-registrar 8:00 I0816 14:38:39.869828       1 main.go:135] Version: v2.10.1
driver-registrar 8:00 I0816 14:38:39.869883       1 main.go:136] Running node-driver-registrar in mode=
driver-registrar 8:00 I0816 14:38:39.873016       1 node_register.go:55] Starting Registration Server at: /registration/rook-ceph.cephfs.csi.ceph.com-reg.sock
driver-registrar 8:00 I0816 14:38:39.873361       1 node_register.go:64] Registration Server started at: /registration/rook-ceph.cephfs.csi.ceph.com-reg.sock
driver-registrar 8:00 I0816 14:38:39.873438       1 node_register.go:88] Skipping HTTP server because endpoint is set to: ""
driver-registrar 8:00 I0816 14:38:39.993070       1 main.go:90] Received GetInfo call: &InfoRequest{}
driver-registrar 8:00 I0816 14:38:39.996096       1 main.go:101] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:false,Error:RegisterPlugin error -- plugin registration failed with err: failed to get controller capabilities from driver rook-ceph.cephfs.csi.ceph.com: rpc error: code = Internal desc = panic runtime error: invalid memory address or nil pointer dereference,}
driver-registrar 8:00 E0816 14:38:39.996187       1 main.go:103] Registration process failed with error: RegisterPlugin error -- plugin registration failed with err: failed to get controller capabilities from driver rook-ceph.cephfs.csi.ceph.com: rpc error: code = Internal desc = panic runtime error: invalid memory address or nil pointer dereference, restarting registration container.

csi-cephfsplugin 8:00 I0816 14:37:52.881875       1 cephcsi.go:191] Driver version: v3.11.0 and Git version: bc24b5eca87626d690a29effa9d7420cc0154a7a
csi-cephfsplugin 8:00 I0816 14:37:52.888193       1 cephcsi.go:268] Initial PID limit is set to -1
csi-cephfsplugin 8:00 I0816 14:37:52.888259       1 cephcsi.go:274] Reconfigured PID limit to -1 (max)
csi-cephfsplugin 8:00 I0816 14:37:52.888813       1 cephcsi.go:223] Starting driver type: cephfs with name: rook-ceph.cephfs.csi.ceph.com
csi-cephfsplugin 8:00 I0816 14:37:52.918696       1 volumemounter.go:79] loaded mounter: kernel
csi-cephfsplugin 8:00 I0816 14:37:52.942156       1 volumemounter.go:90] loaded mounter: fuse
csi-cephfsplugin 8:00 I0816 14:37:52.990417       1 mount_linux.go:282] Detected umount with safe 'not mounted' behavior
csi-cephfsplugin 8:00 I0816 14:37:52.990828       1 server.go:114] listening for CSI-Addons requests on address: &net.UnixAddr{Name:"/tmp/csi-addons.sock", Net:"unix"}
csi-cephfsplugin 8:00 I0816 14:37:52.991208       1 server.go:117] Listening for connections on address: &net.UnixAddr{Name:"//csi/csi.sock", Net:"unix"}
csi-cephfsplugin 8:00 I0816 14:37:53.094143       1 utils.go:198] ID: 1 GRPC call: /csi.v1.Identity/GetPluginInfo
csi-cephfsplugin 8:00 I0816 14:37:53.095973       1 utils.go:199] ID: 1 GRPC request: {}
csi-cephfsplugin 8:00 I0816 14:37:53.095993       1 identityserver-default.go:40] ID: 1 Using default GetPluginInfo
csi-cephfsplugin 8:00 I0816 14:37:53.096049       1 utils.go:205] ID: 1 GRPC response: {"name":"rook-ceph.cephfs.csi.ceph.com","vendor_version":"v3.11.0"}
csi-cephfsplugin 8:00 I0816 14:37:53.343814       1 utils.go:198] ID: 2 GRPC call: /csi.v1.Node/NodeGetInfo
csi-cephfsplugin 8:00 I0816 14:37:53.343937       1 utils.go:199] ID: 2 GRPC request: {}
csi-cephfsplugin 8:00 I0816 14:37:53.343951       1 nodeserver-default.go:45] ID: 2 Using default NodeGetInfo
csi-cephfsplugin 8:00 I0816 14:37:53.344032       1 utils.go:205] ID: 2 GRPC response: {"accessible_topology":{},"node_id":"172.31.25.27"}
csi-cephfsplugin 8:00 I0816 14:37:53.922562       1 utils.go:198] ID: 3 GRPC call: /csi.v1.Identity/GetPluginInfo
csi-cephfsplugin 8:00 I0816 14:37:53.922591       1 utils.go:199] ID: 3 GRPC request: {}
csi-cephfsplugin 8:00 I0816 14:37:53.922597       1 identityserver-default.go:40] ID: 3 Using default GetPluginInfo
csi-cephfsplugin 8:00 I0816 14:37:53.922623       1 utils.go:205] ID: 3 GRPC response: {"name":"rook-ceph.cephfs.csi.ceph.com","vendor_version":"v3.11.0"}
csi-cephfsplugin 8:00 I0816 14:37:53.925839       1 utils.go:198] ID: 4 GRPC call: /csi.v1.Controller/ControllerGetCapabilities
csi-cephfsplugin 8:00 I0816 14:37:53.925903       1 utils.go:199] ID: 4 GRPC request: {}
csi-cephfsplugin 8:00 E0816 14:37:53.926654       1 utils.go:220] panic occurred: runtime error: invalid memory address or nil pointer dereference
csi-cephfsplugin 8:00 goroutine 58 [running]:
csi-cephfsplugin 8:00 runtime/debug.Stack()
csi-cephfsplugin 8:00   /usr/local/go/src/runtime/debug/stack.go:24 +0x5e
csi-cephfsplugin 8:00 runtime/debug.PrintStack()
csi-cephfsplugin 8:00   /usr/local/go/src/runtime/debug/stack.go:16 +0x13
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.panicHandler.func1()
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:221 +0xca
csi-cephfsplugin 8:00 panic({0x1ea86e0?, 0x3a72420?})
csi-cephfsplugin 8:00   /usr/local/go/src/runtime/panic.go:914 +0x21f
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/cephfs.(*ControllerServer).ControllerGetCapabilities(0x1?, {0x27c0450?, 0xc0008ab950?}, 0x7f5cc84b05b0?)
csi-cephfsplugin 8:00   <autogenerated>:1 +0x21
csi-cephfsplugin 8:00 github.com/container-storage-interface/spec/lib/go/csi._Controller_ControllerGetCapabilities_Handler.func1({0x27c0450, 0xc0008ab950}, {0x2055620?, 0xc0005c3ca0})
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:6713 +0x72
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.panicHandler({0x27c0450?, 0xc0008ab950?}, {0x2055620?, 0xc0005c3ca0?}, 0xc00049eba0?, 0xc0006bafc0?)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:226 +0x77
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.NewMiddlewareServerOption.ChainUnaryServer.func2.1({0x27c0450?, 0xc0008ab950?}, {0x2055620?, 0xc0005c3ca0?})
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:48 +0x45
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.logGRPC({0x27c0450, 0xc0008ab950}, {0x2055620?, 0xc0005c3ca0?}, 0x1d765e0?, 0xc00056e5c0)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:201 +0x183
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.NewMiddlewareServerOption.ChainUnaryServer.func2.1({0x27c0450?, 0xc0008ab950?}, {0x2055620?, 0xc0005c3ca0?})
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:48 +0x45
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.contextIDInjector({0x27c0450, 0xc0008ab7d0}, {0x2055620, 0xc0005c3ca0}, 0xc0005eed80?, 0xc00056e600)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:189 +0x132
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.NewMiddlewareServerOption.ChainUnaryServer.func2({0x27c0450, 0xc0008ab7d0}, {0x2055620, 0xc0005c3ca0}, 0xc0005c3cc0, 0x1ea94c0?)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:53 +0x135
csi-cephfsplugin 8:00 github.com/container-storage-interface/spec/lib/go/csi._Controller_ControllerGetCapabilities_Handler({0x216f9a0?, 0x0}, {0x27c0450, 0xc0008ab7d0}, 0xc00088e500, 0xc000797200)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:6715 +0x135
csi-cephfsplugin 8:00 google.golang.org/grpc.(*Server).processUnaryRPC(0xc000812200, {0x27c0450, 0xc0008ab740}, {0x27c8d40, 0xc0006824e0}, 0xc0007a0d80, 0xc0007973e0, 0x3a89a48, 0x0)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1386 +0xe23
csi-cephfsplugin 8:00 google.golang.org/grpc.(*Server).handleStream(0xc000812200, {0x27c8d40, 0xc0006824e0}, 0xc0007a0d80)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1797 +0x100c
csi-cephfsplugin 8:00 google.golang.org/grpc.(*Server).serveStreams.func2.1()
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1027 +0x8b
csi-cephfsplugin 8:00 created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 100
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1038 +0x135
csi-cephfsplugin 8:00 E0816 14:37:53.927809       1 utils.go:203] ID: 4 GRPC error: rpc error: code = Internal desc = panic runtime error: invalid memory address or nil pointer dereference
csi-cephfsplugin 8:00 I0816 14:37:54.740871       1 utils.go:198] ID: 5 GRPC call: /csi.v1.Identity/GetPluginInfo
csi-cephfsplugin 8:00 I0816 14:37:54.740904       1 utils.go:199] ID: 5 GRPC request: {}
csi-cephfsplugin 8:00 I0816 14:37:54.740913       1 identityserver-default.go:40] ID: 5 Using default GetPluginInfo
csi-cephfsplugin 8:00 I0816 14:37:54.740951       1 utils.go:205] ID: 5 GRPC response: {"name":"rook-ceph.cephfs.csi.ceph.com","vendor_version":"v3.11.0"}
csi-cephfsplugin 8:00 I0816 14:37:55.902700       1 utils.go:198] ID: 6 GRPC call: /csi.v1.Identity/GetPluginInfo
csi-cephfsplugin 8:00 I0816 14:37:55.902751       1 utils.go:199] ID: 6 GRPC request: {}
csi-cephfsplugin 8:00 I0816 14:37:55.902762       1 identityserver-default.go:40] ID: 6 Using default GetPluginInfo
csi-cephfsplugin 8:00 I0816 14:37:55.902792       1 utils.go:205] ID: 6 GRPC response: {"name":"rook-ceph.cephfs.csi.ceph.com","vendor_version":"v3.11.0"}
csi-cephfsplugin 8:00 I0816 14:37:55.903050       1 utils.go:198] ID: 7 GRPC call: /csi.v1.Controller/ControllerGetCapabilities
csi-cephfsplugin 8:00 I0816 14:37:55.903072       1 utils.go:199] ID: 7 GRPC request: {}
csi-cephfsplugin 8:00 E0816 14:37:55.903097       1 utils.go:220] panic occurred: runtime error: invalid memory address or nil pointer dereference
csi-cephfsplugin 8:00 goroutine 28 [running]:
csi-cephfsplugin 8:00 runtime/debug.Stack()
csi-cephfsplugin 8:00   /usr/local/go/src/runtime/debug/stack.go:24 +0x5e
csi-cephfsplugin 8:00 runtime/debug.PrintStack()
csi-cephfsplugin 8:00   /usr/local/go/src/runtime/debug/stack.go:16 +0x13
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.panicHandler.func1()
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:221 +0xca
csi-cephfsplugin 8:00 panic({0x1ea86e0?, 0x3a72420?})
csi-cephfsplugin 8:00   /usr/local/go/src/runtime/panic.go:914 +0x21f
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/cephfs.(*ControllerServer).ControllerGetCapabilities(0x1?, {0x27c0450?, 0xc000a02930?}, 0x7f5cc83f0268?)
csi-cephfsplugin 8:00   <autogenerated>:1 +0x21
csi-cephfsplugin 8:00 github.com/container-storage-interface/spec/lib/go/csi._Controller_ControllerGetCapabilities_Handler.func1({0x27c0450, 0xc000a02930}, {0x2055620?, 0xc000a18140})
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:6713 +0x72
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.panicHandler({0x27c0450?, 0xc000a02930?}, {0x2055620?, 0xc000a18140?}, 0xc00005c5e8?, 0xc000a045a0?)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:226 +0x77
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.NewMiddlewareServerOption.ChainUnaryServer.func2.1({0x27c0450?, 0xc000a02930?}, {0x2055620?, 0xc000a18140?})
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:48 +0x45
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.logGRPC({0x27c0450, 0xc000a02930}, {0x2055620?, 0xc000a18140?}, 0x1d765e0?, 0xc000a0c3c0)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:201 +0x183
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.NewMiddlewareServerOption.ChainUnaryServer.func2.1({0x27c0450?, 0xc000a02930?}, {0x2055620?, 0xc000a18140?})
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:48 +0x45
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.contextIDInjector({0x27c0450, 0xc000a02900}, {0x2055620, 0xc000a18140}, 0xc000a00390?, 0xc000a0c400)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:189 +0x132
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.NewMiddlewareServerOption.ChainUnaryServer.func2({0x27c0450, 0xc000a02900}, {0x2055620, 0xc000a18140}, 0xc000a18160, 0x1ea94c0?)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:53 +0x135
csi-cephfsplugin 8:00 github.com/container-storage-interface/spec/lib/go/csi._Controller_ControllerGetCapabilities_Handler({0x216f9a0?, 0x0}, {0x27c0450, 0xc000a02900}, 0xc000a0e300, 0xc000797200)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:6715 +0x135
csi-cephfsplugin 8:00 google.golang.org/grpc.(*Server).processUnaryRPC(0xc000812200, {0x27c0450, 0xc000a02870}, {0x27c8d40, 0xc000702340}, 0xc00001fe60, 0xc0007973e0, 0x3a89a48, 0x0)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1386 +0xe23
csi-cephfsplugin 8:00 google.golang.org/grpc.(*Server).handleStream(0xc000812200, {0x27c8d40, 0xc000702340}, 0xc00001fe60)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1797 +0x100c
csi-cephfsplugin 8:00 google.golang.org/grpc.(*Server).serveStreams.func2.1()
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1027 +0x8b
csi-cephfsplugin 8:00 created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 26
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1038 +0x135
csi-cephfsplugin 8:00 E0816 14:37:55.903274       1 utils.go:203] ID: 7 GRPC error: rpc error: code = Internal desc = panic runtime error: invalid memory address or nil pointer dereference
csi-cephfsplugin 8:00 I0816 14:38:11.857512       1 utils.go:198] ID: 8 GRPC call: /csi.v1.Identity/GetPluginInfo
csi-cephfsplugin 8:00 I0816 14:38:11.857549       1 utils.go:199] ID: 8 GRPC request: {}
csi-cephfsplugin 8:00 I0816 14:38:11.857558       1 identityserver-default.go:40] ID: 8 Using default GetPluginInfo
csi-cephfsplugin 8:00 I0816 14:38:11.857586       1 utils.go:205] ID: 8 GRPC response: {"name":"rook-ceph.cephfs.csi.ceph.com","vendor_version":"v3.11.0"}
csi-cephfsplugin 8:00 I0816 14:38:11.981289       1 utils.go:198] ID: 9 GRPC call: /csi.v1.Identity/GetPluginInfo
csi-cephfsplugin 8:00 I0816 14:38:11.981324       1 utils.go:199] ID: 9 GRPC request: {}
csi-cephfsplugin 8:00 I0816 14:38:11.981347       1 identityserver-default.go:40] ID: 9 Using default GetPluginInfo
csi-cephfsplugin 8:00 I0816 14:38:11.981378       1 utils.go:205] ID: 9 GRPC response: {"name":"rook-ceph.cephfs.csi.ceph.com","vendor_version":"v3.11.0"}
csi-cephfsplugin 8:00 I0816 14:38:11.981710       1 utils.go:198] ID: 10 GRPC call: /csi.v1.Controller/ControllerGetCapabilities
csi-cephfsplugin 8:00 I0816 14:38:11.981734       1 utils.go:199] ID: 10 GRPC request: {}
csi-cephfsplugin 8:00 E0816 14:38:11.981752       1 utils.go:220] panic occurred: runtime error: invalid memory address or nil pointer dereference
csi-cephfsplugin 8:00 goroutine 65 [running]:
csi-cephfsplugin 8:00 runtime/debug.Stack()
csi-cephfsplugin 8:00   /usr/local/go/src/runtime/debug/stack.go:24 +0x5e
csi-cephfsplugin 8:00 runtime/debug.PrintStack()
csi-cephfsplugin 8:00   /usr/local/go/src/runtime/debug/stack.go:16 +0x13
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.panicHandler.func1()
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:221 +0xca
csi-cephfsplugin 8:00 panic({0x1ea86e0?, 0x3a72420?})
csi-cephfsplugin 8:00   /usr/local/go/src/runtime/panic.go:914 +0x21f
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/cephfs.(*ControllerServer).ControllerGetCapabilities(0x1?, {0x27c0450?, 0xc0008e87b0?}, 0x7f5cc846f548?)
csi-cephfsplugin 8:00   <autogenerated>:1 +0x21
csi-cephfsplugin 8:00 github.com/container-storage-interface/spec/lib/go/csi._Controller_ControllerGetCapabilities_Handler.func1({0x27c0450, 0xc0008e87b0}, {0x2055620?, 0xc0005c3f00})
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:6713 +0x72
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.panicHandler({0x27c0450?, 0xc0008e87b0?}, {0x2055620?, 0xc0005c3f00?}, 0xc00049eff0?, 0xc0008f6030?)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:226 +0x77
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.NewMiddlewareServerOption.ChainUnaryServer.func2.1({0x27c0450?, 0xc0008e87b0?}, {0x2055620?, 0xc0005c3f00?})
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:48 +0x45
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.logGRPC({0x27c0450, 0xc0008e87b0}, {0x2055620?, 0xc0005c3f00?}, 0x1d765e0?, 0xc00056f040)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:201 +0x183
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.NewMiddlewareServerOption.ChainUnaryServer.func2.1({0x27c0450?, 0xc0008e87b0?}, {0x2055620?, 0xc0005c3f00?})
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:48 +0x45
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.contextIDInjector({0x27c0450, 0xc0008e8780}, {0x2055620, 0xc0005c3f00}, 0xc0005ef350?, 0xc00056f080)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:189 +0x132
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.NewMiddlewareServerOption.ChainUnaryServer.func2({0x27c0450, 0xc0008e8780}, {0x2055620, 0xc0005c3f00}, 0xc0005c3f20, 0x1ea94c0?)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:53 +0x135
csi-cephfsplugin 8:00 github.com/container-storage-interface/spec/lib/go/csi._Controller_ControllerGetCapabilities_Handler({0x216f9a0?, 0x0}, {0x27c0450, 0xc0008e8780}, 0xc00088ea80, 0xc000797200)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:6715 +0x135
csi-cephfsplugin 8:00 google.golang.org/grpc.(*Server).processUnaryRPC(0xc000812200, {0x27c0450, 0xc0008e86f0}, {0x27c8d40, 0xc000682b60}, 0xc0007a1320, 0xc0007973e0, 0x3a89a48, 0x0)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1386 +0xe23
csi-cephfsplugin 8:00 google.golang.org/grpc.(*Server).handleStream(0xc000812200, {0x27c8d40, 0xc000682b60}, 0xc0007a1320)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1797 +0x100c
csi-cephfsplugin 8:00 google.golang.org/grpc.(*Server).serveStreams.func2.1()
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1027 +0x8b
csi-cephfsplugin 8:00 created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 104
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1038 +0x135
csi-cephfsplugin 8:00 E0816 14:38:11.981906       1 utils.go:203] ID: 10 GRPC error: rpc error: code = Internal desc = panic runtime error: invalid memory address or nil pointer dereference
csi-cephfsplugin 8:00 I0816 14:38:39.871322       1 utils.go:198] ID: 11 GRPC call: /csi.v1.Identity/GetPluginInfo
csi-cephfsplugin 8:00 I0816 14:38:39.871360       1 utils.go:199] ID: 11 GRPC request: {}
csi-cephfsplugin 8:00 I0816 14:38:39.871369       1 identityserver-default.go:40] ID: 11 Using default GetPluginInfo
csi-cephfsplugin 8:00 I0816 14:38:39.871397       1 utils.go:205] ID: 11 GRPC response: {"name":"rook-ceph.cephfs.csi.ceph.com","vendor_version":"v3.11.0"}
csi-cephfsplugin 8:00 I0816 14:38:39.994425       1 utils.go:198] ID: 12 GRPC call: /csi.v1.Identity/GetPluginInfo
csi-cephfsplugin 8:00 I0816 14:38:39.994510       1 utils.go:199] ID: 12 GRPC request: {}
csi-cephfsplugin 8:00 I0816 14:38:39.994576       1 identityserver-default.go:40] ID: 12 Using default GetPluginInfo
csi-cephfsplugin 8:00 I0816 14:38:39.994605       1 utils.go:205] ID: 12 GRPC response: {"name":"rook-ceph.cephfs.csi.ceph.com","vendor_version":"v3.11.0"}
csi-cephfsplugin 8:00 I0816 14:38:39.995225       1 utils.go:198] ID: 13 GRPC call: /csi.v1.Controller/ControllerGetCapabilities
csi-cephfsplugin 8:00 I0816 14:38:39.995295       1 utils.go:199] ID: 13 GRPC request: {}
csi-cephfsplugin 8:00 E0816 14:38:39.995323       1 utils.go:220] panic occurred: runtime error: invalid memory address or nil pointer dereference
csi-cephfsplugin 8:00 goroutine 146 [running]:
csi-cephfsplugin 8:00 runtime/debug.Stack()
csi-cephfsplugin 8:00   /usr/local/go/src/runtime/debug/stack.go:24 +0x5e
csi-cephfsplugin 8:00 runtime/debug.PrintStack()
csi-cephfsplugin 8:00   /usr/local/go/src/runtime/debug/stack.go:16 +0x13
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.panicHandler.func1()
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:221 +0xca
csi-cephfsplugin 8:00 panic({0x1ea86e0?, 0x3a72420?})
csi-cephfsplugin 8:00   /usr/local/go/src/runtime/panic.go:914 +0x21f
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/cephfs.(*ControllerServer).ControllerGetCapabilities(0x1?, {0x27c0450?, 0xc000a94270?}, 0x7f5cc8471df8?)
csi-cephfsplugin 8:00   <autogenerated>:1 +0x21
csi-cephfsplugin 8:00 github.com/container-storage-interface/spec/lib/go/csi._Controller_ControllerGetCapabilities_Handler.func1({0x27c0450, 0xc000a94270}, {0x2055620?, 0xc000aa6040})
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:6713 +0x72
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.panicHandler({0x27c0450?, 0xc000a94270?}, {0x2055620?, 0xc000aa6040?}, 0xc000a90078?, 0xc000a96180?)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:226 +0x77
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.NewMiddlewareServerOption.ChainUnaryServer.func2.1({0x27c0450?, 0xc000a94270?}, {0x2055620?, 0xc000aa6040?})
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:48 +0x45
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.logGRPC({0x27c0450, 0xc000a94270}, {0x2055620?, 0xc000aa6040?}, 0x1d765e0?, 0xc000a980c0)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:201 +0x183
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.NewMiddlewareServerOption.ChainUnaryServer.func2.1({0x27c0450?, 0xc000a94270?}, {0x2055620?, 0xc000aa6040?})
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:48 +0x45
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.contextIDInjector({0x27c0450, 0xc000a94240}, {0x2055620, 0xc000aa6040}, 0xc000a92108?, 0xc000a98100)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:189 +0x132
csi-cephfsplugin 8:00 github.com/ceph/ceph-csi/internal/csi-common.NewMiddlewareServerOption.ChainUnaryServer.func2({0x27c0450, 0xc000a94240}, {0x2055620, 0xc000aa6040}, 0xc000aa6060, 0x1ea94c0?)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:53 +0x135
csi-cephfsplugin 8:00 github.com/container-storage-interface/spec/lib/go/csi._Controller_ControllerGetCapabilities_Handler({0x216f9a0?, 0x0}, {0x27c0450, 0xc000a94240}, 0xc000ab6000, 0xc000797200)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:6715 +0x135
csi-cephfsplugin 8:00 google.golang.org/grpc.(*Server).processUnaryRPC(0xc000812200, {0x27c0450, 0xc000a941b0}, {0x27c8d40, 0xc0007024e0}, 0xc0001da900, 0xc0007973e0, 0x3a89a48, 0x0)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1386 +0xe23
csi-cephfsplugin 8:00 google.golang.org/grpc.(*Server).handleStream(0xc000812200, {0x27c8d40, 0xc0007024e0}, 0xc0001da900)
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1797 +0x100c
csi-cephfsplugin 8:00 google.golang.org/grpc.(*Server).serveStreams.func2.1()
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1027 +0x8b
csi-cephfsplugin 8:00 created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 130
csi-cephfsplugin 8:00   /go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1038 +0x135
csi-cephfsplugin 8:00 E0816 14:38:39.995539       1 utils.go:203] ID: 13 GRPC error: rpc error: code = Internal desc = panic runtime error: invalid memory address or nil pointer dereference

[nixpanic]

It seems a few calls work correctly:

• /csi.v1.Identity/GetPluginInfo
• /csi.v1.Node/NodeGetInfo
• /csi.v1.Identity/GetPluginInfo

The next one that fails is:

• /csi.v1.Controller/ControllerGetCapabilities

Interestingly, this is called against the Ceph-CSI instance that functions as a Node-plugin, not a Controller (provisioner).

fs.cs is the ControllerServer handler, but it is not set in the case of a NodeServer:

https://github.com/ceph/ceph-csi/blob/0eabe32b76340a838603c91178ce17e949fa473b/internal/cephfs/driver.go#L159-L187

At the moment I am unsure how this used to work, and not always caused a panic...