search

seaweedfs / seaweedfs-csi-driver

SeaweedFS CSI Driver https://github.com/seaweedfs/seaweedfs
Apache License 2.0
210 stars 50 forks source link

When setting concurrentWriters=0 A fatal exception occurs - SIGSEGV #163

Open spastorclovr opened 5 months ago

spastorclovr commented 5 months ago

I have been trying to set concurrentWriters to 0 to make sure no limits are being put in the number of writers trying to write on the mounted volume.

I got it working at some point and it did improve a lot the writing thoughput for our service. But, I noticied that the filerUI did not display the same file listing as the one I could see through the mounted volume. So, I tried from scratch (shutting down all pods and up again) , and I have now the error below :point_down: (Adjusting -cacheCapacityMB did not make any difference)

Seaweedfs version: 3.63 Seaweedfs-csi helm version: 0.2.1

I0408 08:06:59.211052 nodeserver.go:103 node publish volume pvc-6c3e3e3e-68d1-44cb-bfdd-da9ba5073c30 to /var/lib/kubelet/pods/138db1a7-7bda-48d0-844c-c4735a7220c1/volumes/kubernetes.io~csi/pvc-6c3e3e3e-68d1-44cb-bfdd-da9ba5073c30/mount
I0408 08:06:59.211383 nodeserver.go:137 volume pvc-80c9113b-5473-42ed-8041-c54a02cc28b7 successfully published to /var/lib/kubelet/pods/138db1a7-7bda-48d0-844c-c4735a7220c1/volumes/kubernetes.io~csi/pvc-80c9113b-5473-42ed-8041-c54a02cc28b7/mount
I0408 08:06:59.211409 utils.go:68 GRPC /csi.v1.Node/NodePublishVolume response
I0408 08:06:59.212475 nodeserver.go:137 volume pvc-6c3e3e3e-68d1-44cb-bfdd-da9ba5073c30 successfully published to /var/lib/kubelet/pods/138db1a7-7bda-48d0-844c-c4735a7220c1/volumes/kubernetes.io~csi/pvc-6c3e3e3e-68d1-44cb-bfdd-da9ba5073c30/mount
I0408 08:06:59.212492 utils.go:68 GRPC /csi.v1.Node/NodePublishVolume response
I0408 08:07:01.241693 meta_cache_init.go:44 ReadDirAllEntries /buckets/pvc-bc59ef4b-5919-48f4-8067-d90e0efd1624 ...
I0408 08:07:01.241748 filer_client.go:120 read directory: directory:"/buckets/pvc-bc59ef4b-5919-48f4-8067-d90e0efd1624" limit:10001
I0408 08:07:01.243415 weedfs_dir_lookup.go:51 dir Lookup cache hit /buckets/pvc-bc59ef4b-5919-48f4-8067-d90e0efd1624/bitcoin.conf
I0408 08:07:01.363138 weedfs_dir_lookup.go:51 dir Lookup cache hit /buckets/pvc-bc59ef4b-5919-48f4-8067-d90e0efd1624/settings.json
ActiveLock 1 Read locked 0 type=0 with waiters 0 active r0 w0.
I0408 08:07:01.368452 reader_at.go:179 zero2 [4,4096) of file size 4 bytes
I0408 08:07:01.368480 page_writer.go:54 ReadDirtyDataAt 0 [0, 4096)
ActiveLock 1 Read unlocked 0 type=0 with waiters 0 active r0 w0.
I0408 08:07:01.368543 weedfs_file_sync.go:98 doFlush /buckets/pvc-bc59ef4b-5919-48f4-8067-d90e0efd1624/settings.json fh 0
ActiveLock 2 doFlush wait for 0 type=1 with waiters 0 active r0 w0.
ActiveLock 2 doFlush locked 0 type=1 with waiters 0 active r0 w0.
ActiveLock 2 doFlush unlocked 0 type=1 with waiters 0 active r0 w0.
ActiveLock 3 ReleaseHandle wait for 0 type=1 with waiters 0 active r0 w0.
ActiveLock 3 ReleaseHandle locked 0 type=1 with waiters 0 active r0 w0.
ActiveLock 3 ReleaseHandle unlocked 0 type=1 with waiters 0 active r0 w0.
I0408 08:07:01.368718 weedfs_dir_lookup.go:51 dir Lookup cache hit /buckets/pvc-bc59ef4b-5919-48f4-8067-d90e0efd1624/settings.json.tmp
I0408 08:07:01.368866 weedfs_attr.go:53 /buckets/pvc-bc59ef4b-5919-48f4-8067-d90e0efd1624/settings.json.tmp setattr set size=0 chunks=0
ActiveLock 4 Write wait for 1 type=1 with waiters 0 active r0 w0.
ActiveLock 4 Write locked 1 type=1 with waiters 0 active r0 w0.
I0408 08:07:01.368977 page_writer.go:34 1 AddPage [0, 4)
I0408 08:07:01.368988 dirty_pages_chunked.go:44 1 memory AddPage [0, 4)
ActiveLock 4 Write unlocked 1 type=1 with waiters 0 active r0 w0.
I0408 08:07:01.369174 weedfs_file_sync.go:98 doFlush /buckets/pvc-bc59ef4b-5919-48f4-8067-d90e0efd1624/settings.json.tmp fh 1
I0408 08:07:01.369184 upload_pipeline.go:171  uploaderCount 0 ++> 1
fatal error: sync: unlock of unlocked mutex
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0xcf2b85]

goroutine 117 [running]:
sync.fatal({0x23d8342?, 0x44?})
    /usr/local/go/src/runtime/panic.go:1061 +0x18
sync.(*Mutex).unlockSlow(0xc0005bc828, 0xffffffff)
    /usr/local/go/src/sync/mutex.go:229 +0x35
sync.(*Mutex).Unlock(0x0?)
    /usr/local/go/src/sync/mutex.go:223 +0x25
panic({0x1fbc240?, 0x3d121f0?})
    /usr/local/go/src/runtime/panic.go:914 +0x21f
github.com/seaweedfs/seaweedfs/weed/util.(*LimitedConcurrentExecutor).Execute(0x0, 0xc00070c520)
    /go/src/github.com/seaweedfs/seaweedfs/weed/util/limited_executor.go:32 +0x25
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).moveToSealed(0xc0005bc7e0, {0x2a57d90?, 0xc0006cfd80}, 0x0)
    /go/src/github.com/seaweedfs/seaweedfs/weed/mount/page_writer/upload_pipeline.go:185 +0x325
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).flushChunks(0xc0005bc7e0)
    /go/src/github.com/seaweedfs/seaweedfs/weed/mount/page_writer/upload_pipeline.go:159 +0xe5
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).FlushAll(0x0?)
    /go/src/github.com/seaweedfs/seaweedfs/weed/mount/page_writer/upload_pipeline.go:150 +0x18
github.com/seaweedfs/seaweedfs/weed/mount.(*ChunkedDirtyPages).FlushData(0xc0006d16e0)
    /go/src/github.com/seaweedfs/seaweedfs/weed/mount/dirty_pages_chunked.go:54 +0x29
github.com/seaweedfs/seaweedfs/weed/mount.(*PageWriter).FlushData(...)
    /go/src/github.com/seaweedfs/seaweedfs/weed/mount/page_writer.go:50
github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).doFlush(0xc0006ec840, 0xc0005fa180, 0x3e8, 0x3e8)
    /go/src/github.com/seaweedfs/seaweedfs/weed/mount/weedfs_file_sync.go:101 +0x169
github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).Flush(0xc0006ec840, 0x7f31eebc1f18?, 0xc0007123d8)
    /go/src/github.com/seaweedfs/seaweedfs/weed/mount/weedfs_file_sync.go:59 +0x48
github.com/hanwen/go-fuse/v2/fuse.doFlush(0xc000712240?, 0xc000712240)
    /go/pkg/mod/github.com/hanwen/go-fuse/v2@v2.4.0/fuse/opcode.go:402 +0x2c
github.com/hanwen/go-fuse/v2/fuse.(*Server).handleRequest(0xc0002d0f20, 0xc000712240)
    /go/pkg/mod/github.com/hanwen/go-fuse/v2@v2.4.0/fuse/server.go:526 +0x267
github.com/hanwen/go-fuse/v2/fuse.(*Server).loop(0xc0002d0f20, 0x0?)
    /go/pkg/mod/github.com/hanwen/go-fuse/v2@v2.4.0/fuse/server.go:499 +0x110
created by github.com/hanwen/go-fuse/v2/fuse.(*Server).readRequest in goroutine 1
    /go/pkg/mod/github.com/hanwen/go-fuse/v2@v2.4.0/fuse/server.go:366 +0x53f

goroutine 1 [syscall]:
syscall.Syscall(0x4800a5?, 0xc0002d0fb8?, 0x47ffea?, 0x1e6ea60?)
    /usr/local/go/src/syscall/syscall_linux.go:69 +0x25
syscall.read(0xc0002d0fb8?, {0xc0019881b0?, 0xc0007e29f0?, 0xc00095d828?})
    /usr/local/go/src/syscall/zsyscall_linux_amd64.go:721 +0x38
syscall.Read(...)
    /usr/local/go/src/syscall/syscall_unix.go:181
github.com/hanwen/go-fuse/v2/fuse.(*Server).readRequest.func1(...)
    /go/pkg/mod/github.com/hanwen/go-fuse/v2@v2.4.0/fuse/server.go:334
github.com/hanwen/go-fuse/v2/fuse.handleEINTR(...)
    /go/pkg/mod/github.com/hanwen/go-fuse/v2@v2.4.0/fuse/server.go:309
github.com/hanwen/go-fuse/v2/fuse.(*Server).readRequest(0xc0002d0f20, 0xc0?)
    /go/pkg/mod/github.com/hanwen/go-fuse/v2@v2.4.0/fuse/server.go:332 +0x1ac
github.com/hanwen/go-fuse/v2/fuse.(*Server).loop(0xc0002d0f20, 0x38?)
    /go/pkg/mod/github.com/hanwen/go-fuse/v2@v2.4.0/fuse/server.go:477 +0x7b
github.com/hanwen/go-fuse/v2/fuse.(*Server).Serve(0xc0002d0f20)
    /go/pkg/mod/github.com/hanwen/go-fuse/v2@v2.4.0/fuse/server.go:421 +0x29
github.com/seaweedfs/seaweedfs/weed/command.RunMount(0x3d4c4e0, 0x0)
    /go/src/github.com/seaweedfs/seaweedfs/weed/command/mount_std.go:271 +0x1712
github.com/seaweedfs/seaweedfs/weed/command.runMount(0x3d2fd78?, {0xc000138260?, 0x0, 0xa?})
    /go/src/github.com/seaweedfs/seaweedfs/weed/command/mount_std.go:55 +0x154
main.main()
    /go/src/github.com/seaweedfs/seaweedfs/weed/weed.go:80 +0x309

goroutine 19 [chan receive]:
github.com/seaweedfs/seaweedfs/weed/glog.(*loggingT).flushDaemon(0x0?)
    /go/src/github.com/seaweedfs/seaweedfs/weed/glog/glog.go:884 +0x66
created by github.com/seaweedfs/seaweedfs/weed/glog.init.0 in goroutine 1
    /go/src/github.com/seaweedfs/seaweedfs/weed/glog/glog.go:409 +0x1ae

goroutine 24 [syscall]:
os/signal.signal_recv()
    /usr/local/go/src/runtime/sigqueue.go:152 +0x29
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:23 +0x13
created by os/signal.Notify.func1.1 in goroutine 1
    /usr/local/go/src/os/signal/signal.go:151 +0x1f

goroutine 50 [chan receive]:
github.com/seaweedfs/seaweedfs/weed/util/grace.init.0.func1()
    /go/src/github.com/seaweedfs/seaweedfs/weed/util/grace/signal_handling.go:38 +0x45
created by github.com/seaweedfs/seaweedfs/weed/util/grace.init.0 in goroutine 1
    /go/src/github.com/seaweedfs/seaweedfs/weed/util/grace/signal_handling.go:37 +0x113

goroutine 51 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc0001cab00)
    /go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:292 +0x9f
created by go.opencensus.io/stats/view.init.0 in goroutine 1
    /go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x8d

goroutine 82 [select]:
github.com/seaweedfs/seaweedfs/weed/stats.(*ServerStats).Start(0xc0000614c0)
    /go/src/github.com/seaweedfs/seaweedfs/weed/stats/stats.go:92 +0x1c5
created by github.com/seaweedfs/seaweedfs/weed/server.init.0 in goroutine 1
    /go/src/github.com/seaweedfs/seaweedfs/weed/server/common.go:41 +0x7b

goroutine 83 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc0002cd9c0, {0x2a4d518, 0xc0007cde00})
    /go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:76 +0x115
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1
    /go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:52 +0x129

goroutine 84 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc0002cd9f0, {0x2a4d518, 0xc0007cde50})
    /go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:76 +0x115
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1
    /go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:52 +0x129

goroutine 85 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc0002cda20, {0x2a4d518, 0xc0007cdea0})
    /go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:76 +0x115
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1
    /go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:52 +0x129

goroutine 42 [IO wait]:
internal/poll.runtime_pollWait(0x7f31a7fa8dc0, 0x72)
    /usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc0007c2000?, 0xc000282000?, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0007c2000, {0xc000282000, 0x8000, 0x8000})
    /usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc0007c2000, {0xc000282000?, 0x1060100000000?, 0x8?})
    /usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc0005be0f0, {0xc000282000?, 0x18?, 0xc0004c3c80?})
    /usr/local/go/src/net/net.go:179 +0x45
bufio.(*Reader).Read(0xc0000b6660, {0xc0006b8120, 0x9, 0x7f31eebc1f18?})
    /usr/local/go/src/bufio/bufio.go:244 +0x197
io.ReadAtLeast({0x2a2dfe0, 0xc0000b6660}, {0xc0006b8120, 0x9, 0x9}, 0x9)
    /usr/local/go/src/io/io.go:335 +0x90
io.ReadFull(...)
    /usr/local/go/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0xc0006b8120, 0x9, 0x3d4b520?}, {0x2a2dfe0?, 0xc0000b6660?})
    /go/pkg/mod/golang.org/x/net@v0.20.0/http2/frame.go:237 +0x65
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0006b80e0)
    /go/pkg/mod/golang.org/x/net@v0.20.0/http2/frame.go:498 +0x85
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0007e4480, 0x0?)
    /go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:1587 +0x245
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 86
    /go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:398 +0x1ebe

goroutine 41 [select]:
google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc0007e4480)
    /go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:1661 +0x151
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 86
    /go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:388 +0x1e4b

goroutine 43 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0006e6370, 0x1)
    /go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/controlbuf.go:418 +0x113
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0000361c0)
    /go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/controlbuf.go:552 +0x86
google.golang.org/grpc/internal/transport.newHTTP2Client.func6()
    /go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:452 +0x85
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 86
    /go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:450 +0x246b

goroutine 101 [select]:
github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc00092e1c0)
    /go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190318030020-c3a204f8e965/leveldb/util/buffer_pool.go:206 +0xb6
created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool in goroutine 1
    /go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190318030020-c3a204f8e965/leveldb/util/buffer_pool.go:237 +0x196

goroutine 102 [select]:
github.com/syndtr/goleveldb/leveldb.(*session).refLoop(0xc00017c780)
    /go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190318030020-c3a204f8e965/leveldb/session_util.go:189 +0x59e
created by github.com/syndtr/goleveldb/leveldb.newSession in goroutine 1
    /go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190318030020-c3a204f8e965/leveldb/session.go:93 +0x2a6

goroutine 103 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc0002dfba0)
    /go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190318030020-c3a204f8e965/leveldb/db_compaction.go:90 +0x12a
created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 1
    /go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190318030020-c3a204f8e965/leveldb/db.go:142 +0x456

goroutine 104 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc0002dfba0)
    /go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190318030020-c3a204f8e965/leveldb/db_state.go:101 +0x9c
created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 1
    /go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190318030020-c3a204f8e965/leveldb/db.go:143 +0x496

goroutine 105 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc0002dfba0)
    /go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190318030020-c3a204f8e965/leveldb/db_compaction.go:825 +0x698
created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 1
    /go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190318030020-c3a204f8e965/leveldb/db.go:149 +0x505

goroutine 106 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc0002dfba0)
    /go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190318030020-c3a204f8e965/leveldb/db_compaction.go:762 +0x105
created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 1
    /go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190318030020-c3a204f8e965/leveldb/db.go:150 +0x545

goroutine 114 [IO wait]:
internal/poll.runtime_pollWait(0x7f31a7fa8eb8, 0x72)
    /usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc0001cb600?, 0x20?, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc0001cb600)
    /usr/local/go/src/internal/poll/fd_unix.go:611 +0x2ac
net.(*netFD).accept(0xc0001cb600)
    /usr/local/go/src/net/fd_unix.go:172 +0x29
net.(*UnixListener).accept(0x448e60?)
    /usr/local/go/src/net/unixsock_posix.go:172 +0x16
net.(*UnixListener).Accept(0xc000982d50)
    /usr/local/go/src/net/unixsock.go:260 +0x30
google.golang.org/grpc.(*Server).Serve(0xc0004d0800, {0x2a465e0?, 0xc000982d50})
    /go/pkg/mod/google.golang.org/grpc@v1.60.1/server.go:871 +0x462
created by github.com/seaweedfs/seaweedfs/weed/command.RunMount in goroutine 1
    /go/src/github.com/seaweedfs/seaweedfs/weed/command/mount_std.go:264 +0x1545

goroutine 115 [select]:
google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc0007de120)
    /go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/transport.go:329 +0x7c
google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
    /go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/transport.go:344
google.golang.org/grpc.(*csAttempt).recvMsg(0xc0006cb040, {0x21c50a0?, 0xc0005c84b0}, 0x41aae8?)
    /go/pkg/mod/google.golang.org/grpc@v1.60.1/stream.go:1070 +0xc9
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x52?)
    /go/pkg/mod/google.golang.org/grpc@v1.60.1/stream.go:927 +0x1f
google.golang.org/grpc.(*clientStream).withRetry(0xc000706fc0, 0xc000763850, 0xc000763840)
    /go/pkg/mod/google.golang.org/grpc@v1.60.1/stream.go:776 +0x13a
google.golang.org/grpc.(*clientStream).RecvMsg(0xc000706fc0, {0x21c50a0?, 0xc0005c84b0?})
    /go/pkg/mod/google.golang.org/grpc@v1.60.1/stream.go:926 +0x125
github.com/seaweedfs/seaweedfs/weed/pb/filer_pb.(*seaweedFilerSubscribeMetadataClient).Recv(0xc0006765c0)
    /go/src/github.com/seaweedfs/seaweedfs/weed/pb/filer_pb/filer_grpc.pb.go:294 +0x46
github.com/seaweedfs/seaweedfs/weed/pb.WithFilerClientFollowMetadata.makeSubscribeMetadataFunc.func1({0x2a699b8, 0xc000984ad0})
    /go/src/github.com/seaweedfs/seaweedfs/weed/pb/filer_pb_tail.go:78 +0x23f
github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).WithFilerClient.func1.1(0xc00065c380)
    /go/src/github.com/seaweedfs/seaweedfs/weed/mount/wfs_filer_client.go:27 +0x68
github.com/seaweedfs/seaweedfs/weed/pb.WithGrpcClient(0xaf?, 0x5bfc52d2?, 0xc000763ce8, {0xc000129290, 0x26}, 0x0?, {0xc000094cd8?, 0x1?, 0x1?})
    /go/src/github.com/seaweedfs/seaweedfs/weed/pb/grpc_client_server.go:155 +0x348
github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).WithFilerClient.func1()
    /go/src/github.com/seaweedfs/seaweedfs/weed/mount/wfs_filer_client.go:25 +0x15e
github.com/seaweedfs/seaweedfs/weed/util.Retry({0x23a1371, 0xa}, 0xc000763df8)
    /go/src/github.com/seaweedfs/seaweedfs/weed/util/retry.go:16 +0xa7
github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).WithFilerClient(0x0?, 0x0?, 0x0?)
    /go/src/github.com/seaweedfs/seaweedfs/weed/mount/wfs_filer_client.go:18 +0x65
github.com/seaweedfs/seaweedfs/weed/pb.WithFilerClientFollowMetadata({0x2a45f20, 0xc0006ec840}, 0xc0007c3980, 0xc0007e2660)
    /go/src/github.com/seaweedfs/seaweedfs/weed/pb/filer_pb_tail.go:50 +0x87
github.com/seaweedfs/seaweedfs/weed/mount/meta_cache.SubscribeMetaEvents.func2()
    /go/src/github.com/seaweedfs/seaweedfs/weed/mount/meta_cache/meta_cache_subscribe.go:74 +0x26
github.com/seaweedfs/seaweedfs/weed/util.RetryUntil({0x23b086f, 0x11}, 0xc000763f60, 0x24e8368)
    /go/src/github.com/seaweedfs/seaweedfs/weed/util/retry.go:64 +0x4e
github.com/seaweedfs/seaweedfs/weed/mount/meta_cache.SubscribeMetaEvents(0xc0007fafc0, 0x5bfc52d2, {0x2a45f20?, 0xc0006ec840}, {0x7ffcab9b563e, 0x31}, 0x17c43f89c0ca0d72)
    /go/src/github.com/seaweedfs/seaweedfs/weed/mount/meta_cache/meta_cache_subscribe.go:72 +0x185
created by github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).StartBackgroundTasks in goroutine 1
    /go/src/github.com/seaweedfs/seaweedfs/weed/mount/weedfs.go:123 +0x152

goroutine 116 [sleep]:
time.Sleep(0xe33e22200)
    /usr/local/go/src/runtime/time.go:195 +0x125
github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).loopCheckQuota(0xc0006ec840)
    /go/src/github.com/seaweedfs/seaweedfs/weed/mount/weedfs_quota.go:15 +0x31
created by github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).StartBackgroundTasks in goroutine 1
    /go/src/github.com/seaweedfs/seaweedfs/weed/mount/weedfs.go:124 +0x18e

goroutine 70 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc0009847b0, {0x2a4d518, 0xc0000c13b0})
    /go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:76 +0x115
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 115
    /go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:52 +0x129

goroutine 71 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc0009847e0, {0x2a4d518, 0xc0000c1450})
    /go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:76 +0x115
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 115
    /go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:52 +0x129

goroutine 72 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc000984810, {0x2a4d518, 0xc0000c14a0})
    /go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:76 +0x115
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 115
    /go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:52 +0x129

goroutine 91 [select]:
google.golang.org/grpc.newClientStreamWithParams.func4()
    /go/pkg/mod/google.golang.org/grpc@v1.60.1/stream.go:391 +0x8c
created by google.golang.org/grpc.newClientStreamWithParams in goroutine 115
    /go/pkg/mod/google.golang.org/grpc@v1.60.1/stream.go:390 +0xe9c

goroutine 88 [select]:
google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc0004d66c0)
    /go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:1661 +0x151
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 73
    /go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:388 +0x1e4b

goroutine 89 [IO wait]:
internal/poll.runtime_pollWait(0x7f31a7fa8cc8, 0x72)
    /usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc0005ba000?, 0xc000786000?, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0005ba000, {0xc000786000, 0x8000, 0x8000})
    /usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc0005ba000, {0xc000786000?, 0x60100000000?, 0x8?})
    /usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc000782000, {0xc000786000?, 0x40b3fe?, 0x4726b2?})
    /usr/local/go/src/net/net.go:179 +0x45
bufio.(*Reader).Read(0xc0007faba0, {0xc00092e040, 0x9, 0xc0?})
    /usr/local/go/src/bufio/bufio.go:244 +0x197
io.ReadAtLeast({0x2a2dfe0, 0xc0007faba0}, {0xc00092e040, 0x9, 0x9}, 0x9)
    /usr/local/go/src/io/io.go:335 +0x90
io.ReadFull(...)
    /usr/local/go/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0xc00092e040, 0x9, 0x20b6300?}, {0x2a2dfe0?, 0xc0007faba0?})
    /go/pkg/mod/golang.org/x/net@v0.20.0/http2/frame.go:237 +0x65
golang.org/x/net/http2.(*Framer).ReadFrame(0xc00092e000)
    /go/pkg/mod/golang.org/x/net@v0.20.0/http2/frame.go:498 +0x85
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0004d66c0, 0x19320b2801200218?)
    /go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:1587 +0x245
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 73
    /go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:398 +0x1ebe

goroutine 90 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0002f1e50, 0x1)
    /go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/controlbuf.go:418 +0x113
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000812000)
    /go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/controlbuf.go:552 +0x86
google.golang.org/grpc/internal/transport.newHTTP2Client.func6()
    /go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:452 +0x85
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 73
    /go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:450 +0x246b

goroutine 78 [IO wait]:
internal/poll.runtime_pollWait(0x7f31a7fa8bd0, 0x72)
    /usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc0005fa100?, 0xc0005fe000?, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0005fa100, {0xc0005fe000, 0x1000, 0x1000})
    /usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc0005fa100, {0xc0005fe000?, 0x751120?, 0xc0005cc000?})
    /usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc0007c6108, {0xc0005fe000?, 0x0?, 0xc0005036c0?})
    /usr/local/go/src/net/net.go:179 +0x45
net/http.(*persistConn).Read(0xc000594480, {0xc0005fe000?, 0xc0001036e0?, 0xc000099d38?})
    /usr/local/go/src/net/http/transport.go:1954 +0x4a
bufio.(*Reader).fill(0xc0006d15c0)
    /usr/local/go/src/bufio/bufio.go:113 +0x103
bufio.(*Reader).Peek(0xc0006d15c0, 0x1)
    /usr/local/go/src/bufio/bufio.go:151 +0x53
net/http.(*persistConn).readLoop(0xc000594480)
    /usr/local/go/src/net/http/transport.go:2118 +0x1b9
created by net/http.(*Transport).dialConn in goroutine 119
    /usr/local/go/src/net/http/transport.go:1776 +0x169f

goroutine 79 [select]:
net/http.(*persistConn).writeLoop(0xc000594480)
    /usr/local/go/src/net/http/transport.go:2421 +0xe5
created by net/http.(*Transport).dialConn in goroutine 119
    /usr/local/go/src/net/http/transport.go:1777 +0x16f1
E0408 08:07:01.375783 mounter.go:75 weed mount exit, pid: 23, path: /var/lib/kubelet/plugins/kubernetes.io/csi/seaweedfs-csi-driver/3fbaadb3372904bcd46022cd9d47f5f22451cea3793cf1b8c028f2d721a0287b/globalmount, error: exit status 2
I0408 08:07:16.502410 utils.go:63 GRPC /csi.v1.Identity/Probe request