search

ceph / ceph-csi

CSI driver for Ceph
Apache License 2.0
1.2k stars 528 forks source link

Incorrect Error Message when Volume Mount Fails #4562

Open cbears opened 2 months ago

cbears commented 2 months ago

Describe the bug

Null variable backtrace when attempting to mount a volume as opposed to an error describing the issue.

Environment details

Steps to reproduce

Steps to reproduce the behavior:

  1. Cause a reason for the mount to fail, for instance, not being able to connect to ceph
  2. Observe bad error message.

Actual results

Expected behavior

The snippet below is from internal/rbd/nodeserver.go. When an error occurs variable rv is NULL, and thus the rv.Destroy() call can't possibly work. The code should check if rv is Null first and then do an rv.Destroy(). 

    rv, err = GenVolFromVolID(ctx, volID, cr, req.GetSecrets())
    if err != nil {
      rv.Destroy()
      log.ErrorLog(ctx, "error generating volume %s: %v", volID, err)

      return nil, status.Errorf(codes.Internal, "error generating volume %s: %v", volID, err)
    }

Logs

4f89650cde07543a9529ff93415f5eb006fce9f211fd63251e2ced5dcf5b9323/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":6}},"volume_id":"nginx-pv"}
2024-04-16T20:50:39.579157107Z stderr F E0416 20:50:39.579119 3190592 utils.go:220] panic occurred: runtime error: invalid memory address or nil pointer dereference
2024-04-16T20:50:39.579345709Z stderr F goroutine 15477 [running]:
2024-04-16T20:50:39.579350155Z stderr F runtime/debug.Stack()
2024-04-16T20:50:39.579352493Z stderr F     /usr/local/go/src/runtime/debug/stack.go:24 +0x5e
2024-04-16T20:50:39.579354524Z stderr F runtime/debug.PrintStack()
2024-04-16T20:50:39.579359937Z stderr F     /usr/local/go/src/runtime/debug/stack.go:16 +0x13
2024-04-16T20:50:39.579362102Z stderr F github.com/ceph/ceph-csi/internal/csi-common.panicHandler.func1()
2024-04-16T20:50:39.57936392Z stderr F  /go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:221 +0xca
2024-04-16T20:50:39.579365844Z stderr F panic({0x1ea8620?, 0x3a72420?})
2024-04-16T20:50:39.579367951Z stderr F     /usr/local/go/src/runtime/panic.go:914 +0x21f
2024-04-16T20:50:39.579370465Z stderr F github.com/ceph/ceph-csi/internal/rbd.(*NodeServer).populateRbdVol(0x27c0370?, {0x27c0370, 0xc00070f1d0}, 0xc0006b0300, 0xc?)
2024-04-16T20:50:39.579372381Z stderr F     /go/src/github.com/ceph/ceph-csi/internal/rbd/nodeserver.go:201 +0xe10
2024-04-16T20:50:39.579374428Z stderr F github.com/ceph/ceph-csi/internal/rbd.(*NodeServer).NodeStageVolume(0xc00005e9d0, {0x27c0370?, 0xc00070f1d0}, 0xc0006b0300)
2024-04-16T20:50:39.579376247Z stderr F     /go/src/github.com/ceph/ceph-csi/internal/rbd/nodeserver.go:344 +0x42d
2024-04-16T20:50:39.579378354Z stderr F github.com/container-storage-interface/spec/lib/go/csi._Node_NodeStageVolume_Handler.func1({0x27c0370, 0xc00070f1d0}, {0x20c3220?, 0xc0006b0300})
2024-04-16T20:50:39.579380296Z stderr F     /go/src/github.com/ceph/ceph-csi/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:7223 +0x72
2024-04-16T20:50:39.579382162Z stderr F github.com/ceph/ceph-csi/internal/csi-common.panicHandler({0x27c0370?, 0xc00070f1d0?}, {0x20c3220?, 0xc0006b0300?}, 0xc000148c60?, 0xc000a8a140?)
2024-04-16T20:50:39.579388353Z stderr F     /go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:226 +0x77
2024-04-16T20:50:39.579390295Z stderr F github.com/ceph/ceph-csi/internal/csi-common.NewMiddlewareServerOption.ChainUnaryServer.func2.1({0x27c0370?, 0xc00070f1d0?}, {0x20c3220?, 0xc0006b0300?})
2024-04-16T20:50:39.579392562Z stderr F     /go/src/github.com/ceph/ceph-csi/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:48 +0x45
2024-04-16T20:50:39.579394677Z stderr F github.com/ceph/ceph-csi/internal/csi-common.logGRPC({0x27c0370, 0xc00070f1d0}, {0x20c3220?, 0xc0006b0300?}, 0x1d76520?, 0xc0000bcc00)
2024-04-16T20:50:39.57939655Z stderr F  /go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:201 +0x183
2024-04-16T20:50:39.579398543Z stderr F github.com/ceph/ceph-csi/internal/csi-common.NewMiddlewareServerOption.ChainUnaryServer.func2.1({0x27c0370?, 0xc00070f1d0?}, {0x20c3220?, 0xc0006b0300?})
2024-04-16T20:50:39.579400417Z stderr F     /go/src/github.com/ceph/ceph-csi/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:48 +0x45
2024-04-16T20:50:39.579407469Z stderr F github.com/ceph/ceph-csi/internal/csi-common.contextIDInjector({0x27c0370, 0xc00070f110}, {0x20c3220, 0xc0006b0300}, 0xc000a065e8?, 0xc0000bcc40)
2024-04-16T20:50:39.579409518Z stderr F     /go/src/github.com/ceph/ceph-csi/internal/csi-common/utils.go:189 +0x132
2024-04-16T20:50:39.579419808Z stderr F github.com/ceph/ceph-csi/internal/csi-common.NewMiddlewareServerOption.ChainUnaryServer.func2({0x27c0370, 0xc00070f110}, {0x20c3220, 0xc0006b0300}, 0xc0004666c0, 0x1ea9400?)
2024-04-16T20:50:39.579421812Z stderr F     /go/src/github.com/ceph/ceph-csi/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:53 +0x135
2024-04-16T20:50:39.579423803Z stderr F github.com/container-storage-interface/spec/lib/go/csi._Node_NodeStageVolume_Handler({0x2125660?, 0xc00005e9d0}, {0x27c0370, 0xc00070f110}, 0xc00068e380, 0xc000293b90)
2024-04-16T20:50:39.579427137Z stderr F     /go/src/github.com/ceph/ceph-csi/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:7225 +0x135
2024-04-16T20:50:39.57942903Z stderr F google.golang.org/grpc.(*Server).processUnaryRPC(0xc00043d000, {0x27c0370, 0xc00070f080}, {0x27c8c60, 0xc0002b96c0}, 0xc00088d200, 0xc000293e00, 0x3a88c60, 0x0)
2024-04-16T20:50:39.579431282Z stderr F     /go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1386 +0xe23
2024-04-16T20:50:39.579433132Z stderr F google.golang.org/grpc.(*Server).handleStream(0xc00043d000, {0x27c8c60, 0xc0002b96c0}, 0xc00088d200)
2024-04-16T20:50:39.579434887Z stderr F     /go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1797 +0x100c
2024-04-16T20:50:39.579436944Z stderr F google.golang.org/grpc.(*Server).serveStreams.func2.1()
2024-04-16T20:50:39.57943887Z stderr F  /go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1027 +0x8b
2024-04-16T20:50:39.579440744Z stderr F created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 15476
2024-04-16T20:50:39.579442583Z stderr F     /go/src/github.com/ceph/ceph-csi/vendor/google.golang.org/grpc/server.go:1038 +0x135
2024-04-16T20:50:39.579489141Z stderr F E0416 20:50:39.579456 3190592 utils.go:203] ID: 4176 Req-ID: nginx-pv GRPC error: rpc error: code = Internal desc = panic runtime error: invalid memory address or nil pointer dereference

Additional context

Add any other context about the problem here.

For example:

Any existing bug report which describe about the similar issue/behavior

cbears commented 2 months ago

Error msg:

MountVolume.MountDevice failed for volume "nginx-pv" : rpc error: code = Internal desc = panic runtime error: invalid memory address or nil pointer dereference