search

juicedata / juicefs

JuiceFS is a distributed POSIX file system built on top of Redis and S3.
https://juicefs.com
Apache License 2.0
10.63k stars 928 forks source link

s3 gateway is not working #4975

Closed Mangoyap closed 2 months ago

Mangoyap commented 3 months ago

What happened: s3 gateway is not working,when i use the command , and use lsof -i:2090, nothing here. but i use version 1.1.2, it's all ok . What you expected to happen: should bind 0.0.0.0:2090 correct How to reproduce it (as minimally and precisely as possible): just run the command: juicefs gateway redis://:xxxx@localchost:6379/1 localhost:2090 --cache-size 0 --backup-meta 0 --buffer-size 4096 --read-only Anything else we need to know? 1.1.2 is ok ,1.2.0+2024-06-18.873c47b is not Environment:

zhijian-pro commented 3 months ago

Please provide the startup log of juicefs gateway

Mangoyap commented 3 months ago

juicefs gateway redis://:xxxxxxxxx@10.0.0.10:19736/1 localhost:2090 --cache-size 0 --backup-meta 0 --buffer-size 4096 --read-only

2024/07/01 11:49:03.977248 juicefs[471323] : Meta address: redis://:****@10.0.0.10:19736/1 [interface.go:504] 2024/07/01 11:49:03.978683 juicefs[471323] : Ping redis latency: 78.091µs [redis.go:3516] 2024/07/01 11:49:03.979511 juicefs[471323] : Data use minio://10.0.0.2:9000/juicefs/minio-s1/ [gateway.go:226] 2024/07/01 11:49:03.979563 juicefs[471323] : cache-size is 0, writeback and prefetch will be disabled [cached_store.go:563] 2024/07/01 11:49:03.979917 juicefs[471323] : Create read-only session OK with version: 1.2.0+2024-06-18.873c47b [base.go:379] 2024/07/01 11:49:03.980372 juicefs[471323] : Prometheus metrics listening on 127.0.0.1:9567 [mount.go:135]

runtime: goroutine stack exceeds 1000000000-byte limit runtime: sp=0xc02a580360 stack=[0xc02a580000, 0xc04a580000] fatal error: stack overflow

runtime stack: runtime.throw({0x30af764?, 0x7fa437bb39a8?}) /usr/local/go/src/runtime/panic.go:1077 +0x5c fp=0x7fa437bb3890 sp=0x7fa437bb3860 pc=0x43cd9c runtime.newstack() /usr/local/go/src/runtime/stack.go:1107 +0x5ac fp=0x7fa437bb3a40 sp=0x7fa437bb3890 pc=0x45784c runtime.morestack() /usr/local/go/src/runtime/asm_amd64.s:593 +0x8f fp=0x7fa437bb3a48 sp=0x7fa437bb3a40 pc=0x471daf

goroutine 1 [running]: github.com/redis/go-redis/v9/internal/pool.(ConnPool).Get(0xc000e87360, {0x7fa437bc0898, 0xc000e2d140}) /go/pkg/mod/github.com/redis/go-redis/v9@v9.0.2/internal/pool/pool.go:240 +0x1bf fp=0xc02a580370 sp=0xc02a580368 pc=0x993ddf github.com/redis/go-redis/v9.(baseClient)._getConn(0xc000b156c0, {0x7fa437bc0898, 0xc000e2d140}) /go/pkg/mod/github.com/redis/go-redis/v9@v9.0.2/redis.go:248 +0x32 fp=0xc02a5803c8 sp=0xc02a580370 pc=0x9e17b2 github.com/redis/go-redis/v9.(baseClient).getConn(0xc000b156c0, {0x7fa437bc0898?, 0xc000e2d140?}) /go/pkg/mod/github.com/redis/go-redis/v9@v9.0.2/redis.go:236 +0x65 fp=0xc02a580400 sp=0xc02a5803c8 pc=0x9e16e5 github.com/redis/go-redis/v9.(baseClient).withConn(0xc000b156c0, {0x7fa437bc0898?, 0xc000e2d140}, 0xc02a5804d0) /go/pkg/mod/github.com/redis/go-redis/v9@v9.0.2/redis.go:340 +0x53 fp=0xc02a580490 sp=0xc02a580400 pc=0x9e1ef3 github.com/redis/go-redis/v9.(baseClient)._process(0xc000b156c0, {0x7fa437bc0898, 0xc000e2d140}, {0x3defd58?, 0xc009531aa0}, 0x411c85?) /go/pkg/mod/github.com/redis/go-redis/v9@v9.0.2/redis.go:382 +0xee fp=0xc02a580508 sp=0xc02a580490 pc=0x9e222e github.com/redis/go-redis/v9.(baseClient).process(0xc000b156c0, {0x7fa437bc0898, 0xc000e2d140}, {0x3defd58, 0xc009531aa0}) /go/pkg/mod/github.com/redis/go-redis/v9@v9.0.2/redis.go:364 +0x73 fp=0xc02a580550 sp=0xc02a580508 pc=0x9e20d3 github.com/redis/go-redis/v9.(*baseClient).process-fm({0x7fa437bc0898?, 0xc000e2d140?}, {0x3defd58?, 0xc009531aa0?})

:1 +0x45 fp=0xc02a580588 sp=0xc02a580550 pc=0xa91085 github.com/redis/go-redis/v9.(*hooksMixin).processHook(...) /go/pkg/mod/github.com/redis/go-redis/v9@v9.0.2/redis.go:173 github.com/redis/go-redis/v9.(*Client).Process(0xc02a5805f8?, {0x7fa437bc0898?, 0xc000e2d140?}, {0x3defd58?, 0xc009531aa0}) /go/pkg/mod/github.com/redis/go-redis/v9@v9.0.2/redis.go:639 +0x3d fp=0xc02a5805d0 sp=0xc02a580588 pc=0x9e3d7d github.com/redis/go-redis/v9.(*Client).Process-fm({0x7fa437bc0898?, 0xc000e2d140?}, {0x3defd58?, 0xc009531aa0?}) :1 +0x45 fp=0xc02a580608 sp=0xc02a5805d0 pc=0xa91a45 github.com/redis/go-redis/v9.cmdable.HGet(0xc000e37380, {0x7fa437bc0898?, 0xc000e2d140}, {0xc009532780, 0x2}, {0xc009532761, 0xa}) /go/pkg/mod/github.com/redis/go-redis/v9@v9.0.2/commands.go:1289 +0x15e fp=0xc02a580650 sp=0xc02a580608 pc=0x9bbc7e github.com/redis/go-redis/v9.(*Client).HGet(0x0?, {0x7fa437bc0898?, 0xc000e2d140?}, {0xc009532780?, 0x0?}, {0xc009532761?, 0x0?}) :1 +0x34 fp=0xc02a580698 sp=0xc02a580650 pc=0xa25494 github.com/juicedata/juicefs/pkg/meta.(*redisMeta).doLookup(0xc000c12f50, {0x3df3768?, 0xc000e2d140}, 0xc00006ca00?, {0xc009532761, 0xa}, 0xc009532778, 0xc00dabe540) /go/src/github.com/juicedata/juicefs/pkg/meta/redis.go:784 +0x3c2 fp=0xc02a5807e8 sp=0xc02a580698 pc=0x138dba2 github.com/juicedata/juicefs/pkg/meta.(*baseMeta).Lookup(0xc00007cf00, {0x3df3768, 0xc000e2d140}, 0x1, {0xc009532761, 0xa}, 0xc009532778, 0xc00dabe540, 0x0) /go/src/github.com/juicedata/juicefs/pkg/meta/base.go:729 +0x463 fp=0xc02a5808e8 sp=0xc02a5807e8 pc=0x1368ee3 github.com/juicedata/juicefs/pkg/meta.(*redisMeta).Lookup(0x52050e?, {0x3df3768?, 0xc000e2d140?}, 0x2a45f2f?, {0xc009532761?, 0x7fa436a729c8?}, 0x7fa47eda9a68?, 0x20?, 0x0?) :1 +0x32 fp=0xc02a580940 sp=0xc02a5808e8 pc=0x143b932 github.com/juicedata/juicefs/pkg/fs.(*FileSystem).lookup(0xc000e87a40, {0x3df3768, 0xc000e2d140}, 0x1?, {0xc009532761, 0xa}, 0xc009532778, 0xc00dabe540) /go/src/github.com/juicedata/juicefs/pkg/fs/fs.go:709 +0x4fb fp=0xc02a580a30 sp=0xc02a580940 pc=0x1f2b01b github.com/juicedata/juicefs/pkg/fs.(*FileSystem).doResolve(0xc000e87a40, {0x3df3768, 0xc000e2d140}, {0xc009532760, 0xb}, 0x1, 0x45951e?) /go/src/github.com/juicedata/juicefs/pkg/fs/fs.go:783 +0x40d fp=0xc02a580ba8 sp=0xc02a580a30 pc=0x1f2b84d github.com/juicedata/juicefs/pkg/fs.(*FileSystem).resolve(...) /go/src/github.com/juicedata/juicefs/pkg/fs/fs.go:733 github.com/juicedata/juicefs/pkg/fs.(*FileSystem).Stat(0xc000e87a40, {0x3df3768?, 0xc000e2d140}, {0xc009532760, 0xb}) /go/src/github.com/juicedata/juicefs/pkg/fs/fs.go:417 +0x287 fp=0xc02a580d00 sp=0xc02a580ba8 pc=0x1f23c47 github.com/juicedata/juicefs/pkg/gateway.(*jfsObjects).checkBucket(0xc000fd5d10, {0x3ddd860, 0x566a400}, {0x30a36dc, 0xa}) /go/src/github.com/juicedata/juicefs/pkg/gateway/gateway.go:360 +0xb4 fp=0xc02a580d58 sp=0xc02a580d00 pc=0x25e8014 github.com/juicedata/juicefs/pkg/gateway.(*jfsObjects).PutObject(_, {_, _}, {_, _}, {_, _}, _, {{0x0, 0x0}, ...}) /go/src/github.com/juicedata/juicefs/pkg/gateway/gateway.go:780 +0x9b fp=0xc02a580fb8 sp=0xc02a580d58 pc=0x25ede7b github.com/minio/minio/cmd.saveConfig({0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}, {0xc0112b0a00, 0x20}, {0xc00952f4a0, 0x12f, 0x154}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/config-common.go:72 +0x1d9 fp=0xc02a581498 sp=0xc02a580fb8 pc=0x2373cb9 github.com/minio/minio/cmd.(*BucketMetadata).Save(0xc02a5816f0, {0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/bucket-metadata.go:368 +0x185 fp=0xc02a581538 sp=0xc02a581498 pc=0x23591a5 github.com/juicedata/juicefs/pkg/gateway.(*jfsObjects).MakeBucketWithLocation(0xc000fd5d10, {0x3ddd860, 0x566a400}, {0x30a36dc, 0xa}, {{0x0?, 0x0?}, 0x0?, 0x0?}) /go/src/github.com/juicedata/juicefs/pkg/gateway/gateway.go:222 +0x19f fp=0xc02a581890 sp=0xc02a581538 pc=0x25e707f github.com/minio/minio/cmd.saveConfig({0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}, {0xc0112b0940, 0x20}, {0xc00952f340, 0x12f, 0x154}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/config-common.go:74 +0x2a7 fp=0xc02a581d70 sp=0xc02a581890 pc=0x2373d87 github.com/minio/minio/cmd.(*BucketMetadata).Save(0xc02a581fc8, {0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/bucket-metadata.go:368 +0x185 fp=0xc02a581e10 sp=0xc02a581d70 pc=0x23591a5 github.com/juicedata/juicefs/pkg/gateway.(*jfsObjects).MakeBucketWithLocation(0xc000fd5d10, {0x3ddd860, 0x566a400}, {0x30a36dc, 0xa}, {{0x0?, 0x0?}, 0x0?, 0x0?}) /go/src/github.com/juicedata/juicefs/pkg/gateway/gateway.go:222 +0x19f fp=0xc02a582168 sp=0xc02a581e10 pc=0x25e707f github.com/minio/minio/cmd.saveConfig({0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}, {0xc0112b0840, 0x20}, {0xc00952f1e0, 0x12f, 0x154}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/config-common.go:74 +0x2a7 fp=0xc02a582648 sp=0xc02a582168 pc=0x2373d87 github.com/minio/minio/cmd.(*BucketMetadata).Save(0xc02a5828a0, {0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/bucket-metadata.go:368 +0x185 fp=0xc02a5826e8 sp=0xc02a582648 pc=0x23591a5 github.com/juicedata/juicefs/pkg/gateway.(*jfsObjects).MakeBucketWithLocation(0xc000fd5d10, {0x3ddd860, 0x566a400}, {0x30a36dc, 0xa}, {{0x0?, 0x0?}, 0x0?, 0x0?}) /go/src/github.com/juicedata/juicefs/pkg/gateway/gateway.go:222 +0x19f fp=0xc02a582a40 sp=0xc02a5826e8 pc=0x25e707f github.com/minio/minio/cmd.saveConfig({0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}, {0xc0112b0780, 0x20}, {0xc00952f080, 0x12f, 0x154}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/config-common.go:74 +0x2a7 fp=0xc02a582f20 sp=0xc02a582a40 pc=0x2373d87 github.com/minio/minio/cmd.(*BucketMetadata).Save(0xc02a583178, {0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/bucket-metadata.go:368 +0x185 fp=0xc02a582fc0 sp=0xc02a582f20 pc=0x23591a5 github.com/juicedata/juicefs/pkg/gateway.(*jfsObjects).MakeBucketWithLocation(0xc000fd5d10, {0x3ddd860, 0x566a400}, {0x30a36dc, 0xa}, {{0x0?, 0x0?}, 0x0?, 0x0?}) /go/src/github.com/juicedata/juicefs/pkg/gateway/gateway.go:222 +0x19f fp=0xc02a583318 sp=0xc02a582fc0 pc=0x25e707f github.com/minio/minio/cmd.saveConfig({0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}, {0xc0112b06a0, 0x20}, {0xc00952ef20, 0x12f, 0x154}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/config-common.go:74 +0x2a7 fp=0xc02a5837f8 sp=0xc02a583318 pc=0x2373d87 github.com/minio/minio/cmd.(*BucketMetadata).Save(0xc02a583a50, {0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/bucket-metadata.go:368 +0x185 fp=0xc02a583898 sp=0xc02a5837f8 pc=0x23591a5 github.com/juicedata/juicefs/pkg/gateway.(*jfsObjects).MakeBucketWithLocation(0xc000fd5d10, {0x3ddd860, 0x566a400}, {0x30a36dc, 0xa}, {{0x0?, 0x0?}, 0x0?, 0x0?}) /go/src/github.com/juicedata/juicefs/pkg/gateway/gateway.go:222 +0x19f fp=0xc02a583bf0 sp=0xc02a583898 pc=0x25e707f github.com/minio/minio/cmd.saveConfig({0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}, {0xc0112b05e0, 0x20}, {0xc00952edc0, 0x12f, 0x154}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/config-common.go:74 +0x2a7 fp=0xc02a5840d0 sp=0xc02a583bf0 pc=0x2373d87 github.com/minio/minio/cmd.(*BucketMetadata).Save(0xc02a584328, {0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/bucket-metadata.go:368 +0x185 fp=0xc02a584170 sp=0xc02a5840d0 pc=0x23591a5 github.com/juicedata/juicefs/pkg/gateway.(*jfsObjects).MakeBucketWithLocation(0xc000fd5d10, {0x3ddd860, 0x566a400}, {0x30a36dc, 0xa}, {{0x0?, 0x0?}, 0x0?, 0x0?}) /go/src/github.com/juicedata/juicefs/pkg/gateway/gateway.go:222 +0x19f fp=0xc02a5844c8 sp=0xc02a584170 pc=0x25e707f github.com/minio/minio/cmd.saveConfig({0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}, {0xc0112b04e0, 0x20}, {0xc00952ec60, 0x12f, 0x154}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/config-common.go:74 +0x2a7 fp=0xc02a5849a8 sp=0xc02a5844c8 pc=0x2373d87 github.com/minio/minio/cmd.(*BucketMetadata).Save(0xc02a584c00, {0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/bucket-metadata.go:368 +0x185 fp=0xc02a584a48 sp=0xc02a5849a8 pc=0x23591a5 github.com/juicedata/juicefs/pkg/gateway.(*jfsObjects).MakeBucketWithLocation(0xc000fd5d10, {0x3ddd860, 0x566a400}, {0x30a36dc, 0xa}, {{0x0?, 0x0?}, 0x0?, 0x0?}) /go/src/github.com/juicedata/juicefs/pkg/gateway/gateway.go:222 +0x19f fp=0xc02a584da0 sp=0xc02a584a48 pc=0x25e707f github.com/minio/minio/cmd.saveConfig({0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}, {0xc0112b0420, 0x20}, {0xc00952eb00, 0x12f, 0x154}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/config-common.go:74 +0x2a7 fp=0xc02a585280 sp=0xc02a584da0 pc=0x2373d87 github.com/minio/minio/cmd.(*BucketMetadata).Save(0xc02a5854d8, {0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/bucket-metadata.go:368 +0x185 fp=0xc02a585320 sp=0xc02a585280 pc=0x23591a5 github.com/juicedata/juicefs/pkg/gateway.(*jfsObjects).MakeBucketWithLocation(0xc000fd5d10, {0x3ddd860, 0x566a400}, {0x30a36dc, 0xa}, {{0x0?, 0x0?}, 0x0?, 0x0?}) /go/src/github.com/juicedata/juicefs/pkg/gateway/gateway.go:222 +0x19f fp=0xc02a585678 sp=0xc02a585320 pc=0x25e707f github.com/minio/minio/cmd.saveConfig({0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}, {0xc0112b0340, 0x20}, {0xc00952e9a0, 0x12f, 0x154}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/config-common.go:74 +0x2a7 fp=0xc02a585b58 sp=0xc02a585678 pc=0x2373d87 github.com/minio/minio/cmd.(*BucketMetadata).Save(0xc02a585db0, {0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/bucket-metadata.go:368 +0x185 fp=0xc02a585bf8 sp=0xc02a585b58 pc=0x23591a5 github.com/juicedata/juicefs/pkg/gateway.(*jfsObjects).MakeBucketWithLocation(0xc000fd5d10, {0x3ddd860, 0x566a400}, {0x30a36dc, 0xa}, {{0x0?, 0x0?}, 0x0?, 0x0?}) /go/src/github.com/juicedata/juicefs/pkg/gateway/gateway.go:222 +0x19f fp=0xc02a585f50 sp=0xc02a585bf8 pc=0x25e707f github.com/minio/minio/cmd.saveConfig({0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}, {0xc0112b0280, 0x20}, {0xc00952e840, 0x12f, 0x154}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/config-common.go:74 +0x2a7 fp=0xc02a586430 sp=0xc02a585f50 pc=0x2373d87 github.com/minio/minio/cmd.(*BucketMetadata).Save(0xc02a586688, {0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/bucket-metadata.go:368 +0x185 fp=0xc02a5864d0 sp=0xc02a586430 pc=0x23591a5 ...711322 frames elided... github.com/minio/minio/cmd.saveConfig({0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}, {0xc0016924c0, 0x20}, {0xc0016af080, 0x12f, 0x154}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/config-common.go:74 +0x2a7 fp=0xc04a578010 sp=0xc04a577b30 pc=0x2373d87 github.com/minio/minio/cmd.(*BucketMetadata).Save(0xc00153a268, {0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/bucket-metadata.go:368 +0x185 fp=0xc04a5780b0 sp=0xc04a578010 pc=0x23591a5 github.com/juicedata/juicefs/pkg/gateway.(*jfsObjects).MakeBucketWithLocation(0xc000fd5d10, {0x3ddd860, 0x566a400}, {0x30a36dc, 0xa}, {{0x0?, 0x0?}, 0x0?, 0x0?}) /go/src/github.com/juicedata/juicefs/pkg/gateway/gateway.go:222 +0x19f fp=0xc04a578408 sp=0xc04a5780b0 pc=0x25e707f github.com/minio/minio/cmd.saveConfig({0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}, {0xc001692460, 0x20}, {0xc0016aef20, 0x12f, 0x154}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/config-common.go:74 +0x2a7 fp=0xc04a5788e8 sp=0xc04a578408 pc=0x2373d87 github.com/minio/minio/cmd.(*BucketMetadata).Save(0xc00153ab40, {0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/bucket-metadata.go:368 +0x185 fp=0xc04a578988 sp=0xc04a5788e8 pc=0x23591a5 github.com/juicedata/juicefs/pkg/gateway.(*jfsObjects).MakeBucketWithLocation(0xc000fd5d10, {0x3ddd860, 0x566a400}, {0x30a36dc, 0xa}, {{0x0?, 0x0?}, 0x0?, 0x0?}) /go/src/github.com/juicedata/juicefs/pkg/gateway/gateway.go:222 +0x19f fp=0xc04a578ce0 sp=0xc04a578988 pc=0x25e707f github.com/minio/minio/cmd.saveConfig({0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}, {0xc001692400, 0x20}, {0xc0016aedc0, 0x12f, 0x154}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/config-common.go:74 +0x2a7 fp=0xc04a5791c0 sp=0xc04a578ce0 pc=0x2373d87 github.com/minio/minio/cmd.(*BucketMetadata).Save(0xc00153b418, {0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/bucket-metadata.go:368 +0x185 fp=0xc04a579260 sp=0xc04a5791c0 pc=0x23591a5 github.com/juicedata/juicefs/pkg/gateway.(*jfsObjects).MakeBucketWithLocation(0xc000fd5d10, {0x3ddd860, 0x566a400}, {0x30a36dc, 0xa}, {{0x0?, 0x0?}, 0x0?, 0x0?}) /go/src/github.com/juicedata/juicefs/pkg/gateway/gateway.go:222 +0x19f fp=0xc04a5795b8 sp=0xc04a579260 pc=0x25e707f github.com/minio/minio/cmd.saveConfig({0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}, {0xc0016923a0, 0x20}, {0xc0016aec60, 0x12f, 0x154}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/config-common.go:74 +0x2a7 fp=0xc04a579a98 sp=0xc04a5795b8 pc=0x2373d87 github.com/minio/minio/cmd.(*BucketMetadata).Save(0xc00153bcf0, {0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/bucket-metadata.go:368 +0x185 fp=0xc04a579b38 sp=0xc04a579a98 pc=0x23591a5 github.com/juicedata/juicefs/pkg/gateway.(*jfsObjects).MakeBucketWithLocation(0xc000fd5d10, {0x3ddd860, 0x566a400}, {0x30a36dc, 0xa}, {{0x0?, 0x0?}, 0x0?, 0x0?}) /go/src/github.com/juicedata/juicefs/pkg/gateway/gateway.go:222 +0x19f fp=0xc04a579e90 sp=0xc04a579b38 pc=0x25e707f github.com/minio/minio/cmd.saveConfig({0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}, {0xc001692340, 0x20}, {0xc0016aeb00, 0x12f, 0x154}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/config-common.go:74 +0x2a7 fp=0xc04a57a370 sp=0xc04a579e90 pc=0x2373d87 github.com/minio/minio/cmd.(*BucketMetadata).Save(0xc00153c5c8, {0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/bucket-metadata.go:368 +0x185 fp=0xc04a57a410 sp=0xc04a57a370 pc=0x23591a5 github.com/juicedata/juicefs/pkg/gateway.(*jfsObjects).MakeBucketWithLocation(0xc000fd5d10, {0x3ddd860, 0x566a400}, {0x30a36dc, 0xa}, {{0x0?, 0x0?}, 0x0?, 0x0?}) /go/src/github.com/juicedata/juicefs/pkg/gateway/gateway.go:222 +0x19f fp=0xc04a57a768 sp=0xc04a57a410 pc=0x25e707f github.com/minio/minio/cmd.saveConfig({0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}, {0xc0016922e0, 0x20}, {0xc0016ae9a0, 0x12f, 0x154}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/config-common.go:74 +0x2a7 fp=0xc04a57ac48 sp=0xc04a57a768 pc=0x2373d87 github.com/minio/minio/cmd.(*BucketMetadata).Save(0xc00153cea0, {0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/bucket-metadata.go:368 +0x185 fp=0xc04a57ace8 sp=0xc04a57ac48 pc=0x23591a5 github.com/juicedata/juicefs/pkg/gateway.(*jfsObjects).MakeBucketWithLocation(0xc000fd5d10, {0x3ddd860, 0x566a400}, {0x30a36dc, 0xa}, {{0x0?, 0x0?}, 0x0?, 0x0?}) /go/src/github.com/juicedata/juicefs/pkg/gateway/gateway.go:222 +0x19f fp=0xc04a57b040 sp=0xc04a57ace8 pc=0x25e707f github.com/minio/minio/cmd.saveConfig({0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}, {0xc001692280, 0x20}, {0xc0016ae840, 0x12f, 0x154}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/config-common.go:74 +0x2a7 fp=0xc04a57b520 sp=0xc04a57b040 pc=0x2373d87 github.com/minio/minio/cmd.(*BucketMetadata).Save(0xc00153d778, {0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/bucket-metadata.go:368 +0x185 fp=0xc04a57b5c0 sp=0xc04a57b520 pc=0x23591a5 github.com/juicedata/juicefs/pkg/gateway.(*jfsObjects).MakeBucketWithLocation(0xc000fd5d10, {0x3ddd860, 0x566a400}, {0x30a36dc, 0xa}, {{0x0?, 0x0?}, 0x0?, 0x0?}) /go/src/github.com/juicedata/juicefs/pkg/gateway/gateway.go:222 +0x19f fp=0xc04a57b918 sp=0xc04a57b5c0 pc=0x25e707f github.com/minio/minio/cmd.saveConfig({0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}, {0xc001692220, 0x20}, {0xc0016ae6e0, 0x12f, 0x154}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/config-common.go:74 +0x2a7 fp=0xc04a57bdf8 sp=0xc04a57b918 pc=0x2373d87 github.com/minio/minio/cmd.(*BucketMetadata).Save(0xc00153e050, {0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/bucket-metadata.go:368 +0x185 fp=0xc04a57be98 sp=0xc04a57bdf8 pc=0x23591a5 github.com/juicedata/juicefs/pkg/gateway.(*jfsObjects).MakeBucketWithLocation(0xc000fd5d10, {0x3ddd860, 0x566a400}, {0x30a36dc, 0xa}, {{0x0?, 0x0?}, 0x0?, 0x0?}) /go/src/github.com/juicedata/juicefs/pkg/gateway/gateway.go:222 +0x19f fp=0xc04a57c1f0 sp=0xc04a57be98 pc=0x25e707f github.com/minio/minio/cmd.saveConfig({0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}, {0xc0016921c0, 0x20}, {0xc0016ae580, 0x12f, 0x154}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/config-common.go:74 +0x2a7 fp=0xc04a57c6d0 sp=0xc04a57c1f0 pc=0x2373d87 github.com/minio/minio/cmd.(*BucketMetadata).Save(0xc00153e928, {0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/bucket-metadata.go:368 +0x185 fp=0xc04a57c770 sp=0xc04a57c6d0 pc=0x23591a5 github.com/juicedata/juicefs/pkg/gateway.(*jfsObjects).MakeBucketWithLocation(0xc000fd5d10, {0x3ddd860, 0x566a400}, {0x30a36dc, 0xa}, {{0x0?, 0x0?}, 0x0?, 0x0?}) /go/src/github.com/juicedata/juicefs/pkg/gateway/gateway.go:222 +0x19f fp=0xc04a57cac8 sp=0xc04a57c770 pc=0x25e707f github.com/minio/minio/cmd.saveConfig({0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}, {0xc001692160, 0x20}, {0xc0016ae420, 0x12f, 0x154}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/config-common.go:74 +0x2a7 fp=0xc04a57cfa8 sp=0xc04a57cac8 pc=0x2373d87 github.com/minio/minio/cmd.(*BucketMetadata).Save(0xc00153f200, {0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/bucket-metadata.go:368 +0x185 fp=0xc04a57d048 sp=0xc04a57cfa8 pc=0x23591a5 github.com/juicedata/juicefs/pkg/gateway.(*jfsObjects).MakeBucketWithLocation(0xc000fd5d10, {0x3ddd860, 0x566a400}, {0x30a36dc, 0xa}, {{0x0?, 0x0?}, 0x0?, 0x0?}) /go/src/github.com/juicedata/juicefs/pkg/gateway/gateway.go:222 +0x19f fp=0xc04a57d3a0 sp=0xc04a57d048 pc=0x25e707f github.com/minio/minio/cmd.saveConfig({0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}, {0xc001692100, 0x20}, {0xc0016ae2c0, 0x12f, 0x154}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/config-common.go:74 +0x2a7 fp=0xc04a57d880 sp=0xc04a57d3a0 pc=0x2373d87 github.com/minio/minio/cmd.(*BucketMetadata).Save(0xc00153fad8, {0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/bucket-metadata.go:368 +0x185 fp=0xc04a57d920 sp=0xc04a57d880 pc=0x23591a5 github.com/juicedata/juicefs/pkg/gateway.(*jfsObjects).MakeBucketWithLocation(0xc000fd5d10, {0x3ddd860, 0x566a400}, {0x30a36dc, 0xa}, {{0x0?, 0x0?}, 0x0?, 0x0?}) /go/src/github.com/juicedata/juicefs/pkg/gateway/gateway.go:222 +0x19f fp=0xc04a57dc78 sp=0xc04a57d920 pc=0x25e707f github.com/minio/minio/cmd.saveConfig({0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}, {0xc0016920a0, 0x20}, {0xc0016ae160, 0x12f, 0x154}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/config-common.go:74 +0x2a7 fp=0xc04a57e158 sp=0xc04a57dc78 pc=0x2373d87 github.com/minio/minio/cmd.(*BucketMetadata).Save(0xc0015403b0, {0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/bucket-metadata.go:368 +0x185 fp=0xc04a57e1f8 sp=0xc04a57e158 pc=0x23591a5 github.com/juicedata/juicefs/pkg/gateway.(*jfsObjects).MakeBucketWithLocation(0xc000fd5d10, {0x3ddd860, 0x566a400}, {0x30a36dc, 0xa}, {{0x0?, 0x0?}, 0x0?, 0x0?}) /go/src/github.com/juicedata/juicefs/pkg/gateway/gateway.go:222 +0x19f fp=0xc04a57e550 sp=0xc04a57e1f8 pc=0x25e707f github.com/minio/minio/cmd.saveConfig({0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}, {0xc001692040, 0x20}, {0xc0016ae000, 0x12f, 0x154}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/config-common.go:74 +0x2a7 fp=0xc04a57ea30 sp=0xc04a57e550 pc=0x2373d87 github.com/minio/minio/cmd.(*BucketMetadata).Save(0xc001540c88, {0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/bucket-metadata.go:368 +0x185 fp=0xc04a57ead0 sp=0xc04a57ea30 pc=0x23591a5 github.com/juicedata/juicefs/pkg/gateway.(*jfsObjects).MakeBucketWithLocation(0xc000fd5d10, {0x3ddd860, 0x566a400}, {0x30a36dc, 0xa}, {{0x0?, 0x0?}, 0x0?, 0x0?}) /go/src/github.com/juicedata/juicefs/pkg/gateway/gateway.go:222 +0x19f fp=0xc04a57ee28 sp=0xc04a57ead0 pc=0x25e707f github.com/minio/minio/cmd.saveConfig({0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}, {0xc00005a640, 0x20}, {0xc0000fa6e0, 0x12f, 0x154}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/config-common.go:74 +0x2a7 fp=0xc04a57f308 sp=0xc04a57ee28 pc=0x2373d87 github.com/minio/minio/cmd.(*BucketMetadata).Save(0xc001541560, {0x3ddd860, 0x566a400}, {0x3e0ff58, 0xc000fd5d10}) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/bucket-metadata.go:368 +0x185 fp=0xc04a57f3a8 sp=0xc04a57f308 pc=0x23591a5 github.com/juicedata/juicefs/pkg/gateway.(*jfsObjects).MakeBucketWithLocation(0xc000fd5d10, {0x3ddd860, 0x566a400}, {0x30a36dc, 0xa}, {{0xc0000143d0?, 0xe?}, 0xe0?, 0x6a?}) /go/src/github.com/juicedata/juicefs/pkg/gateway/gateway.go:222 +0x19f fp=0xc04a57f700 sp=0xc04a57f3a8 pc=0x25e707f github.com/juicedata/juicefs/cmd.gateway(0xc001586a80) /go/src/github.com/juicedata/juicefs/cmd/gateway.go:163 +0x4b7 fp=0xc04a57f940 sp=0xc04a57f700 pc=0x266d677 github.com/urfave/cli/v2.(*Command).Run(0xc00104cb40, 0xc0010256c0) /go/pkg/mod/github.com/juicedata/cli/v2@v2.19.4-0.20230605075551-9c9c5c0dce83/command.go:177 +0x6d4 fp=0xc04a57fa40 sp=0xc04a57f940 pc=0x5da9b4 github.com/urfave/cli/v2.(*App).RunContext(0xc000dd6e00, {0x3ddd860?, 0x566a400}, {0xc00001f700, 0xb, 0x10}) /go/pkg/mod/github.com/juicedata/cli/v2@v2.19.4-0.20230605075551-9c9c5c0dce83/app.go:389 +0x1069 fp=0xc04a57fd20 sp=0xc04a57fa40 pc=0x5d67e9 github.com/urfave/cli/v2.(*App).Run(...) /go/pkg/mod/github.com/juicedata/cli/v2@v2.19.4-0.20230605075551-9c9c5c0dce83/app.go:254 github.com/juicedata/juicefs/cmd.Main({0xc000052420, 0xb, 0xb}) /go/src/github.com/juicedata/juicefs/cmd/main.go:101 +0x38b1 fp=0xc04a57ff00 sp=0xc04a57fd20 pc=0x267eb31 main.main() /go/src/github.com/juicedata/juicefs/main.go:29 +0x28 fp=0xc04a57ff40 sp=0xc04a57ff00 pc=0x26bfbe8 runtime.main() /usr/local/go/src/runtime/proc.go:267 +0x2bb fp=0xc04a57ffe0 sp=0xc04a57ff40 pc=0x43f8bb runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc04a57ffe8 sp=0xc04a57ffe0 pc=0x473c41 goroutine 2 [force gc (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000088fa8 sp=0xc000088f88 pc=0x43fd2e runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:404 runtime.forcegchelper() /usr/local/go/src/runtime/proc.go:322 +0xb3 fp=0xc000088fe0 sp=0xc000088fa8 pc=0x43fb93 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000088fe8 sp=0xc000088fe0 pc=0x473c41 created by runtime.init.6 in goroutine 1 /usr/local/go/src/runtime/proc.go:310 +0x1a goroutine 3 [GC sweep wait]: runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000089778 sp=0xc000089758 pc=0x43fd2e runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:404 runtime.bgsweep(0x0?) /usr/local/go/src/runtime/mgcsweep.go:321 +0xdf fp=0xc0000897c8 sp=0xc000089778 pc=0x429b5f runtime.gcenable.func1() /usr/local/go/src/runtime/mgc.go:200 +0x25 fp=0xc0000897e0 sp=0xc0000897c8 pc=0x41eca5 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000897e8 sp=0xc0000897e0 pc=0x473c41 created by runtime.gcenable in goroutine 1 /usr/local/go/src/runtime/mgc.go:200 +0x66 goroutine 4 [GC scavenge wait]: runtime.gopark(0xc0000b2000?, 0x3da8e18?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000089f70 sp=0xc000089f50 pc=0x43fd2e runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:404 runtime.(*scavengerState).park(0x5637920) /usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000089fa0 sp=0xc000089f70 pc=0x4273c9 runtime.bgscavenge(0x0?) /usr/local/go/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc000089fc8 sp=0xc000089fa0 pc=0x427979 runtime.gcenable.func2() /usr/local/go/src/runtime/mgc.go:201 +0x25 fp=0xc000089fe0 sp=0xc000089fc8 pc=0x41ec45 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000089fe8 sp=0xc000089fe0 pc=0x473c41 created by runtime.gcenable in goroutine 1 /usr/local/go/src/runtime/mgc.go:201 +0xa5 goroutine 5 [finalizer wait]: runtime.gopark(0x198?, 0x3066880?, 0x1?, 0xe?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000088620 sp=0xc000088600 pc=0x43fd2e runtime.runfinq() /usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc0000887e0 sp=0xc000088620 pc=0x41dcc7 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000887e8 sp=0xc0000887e0 pc=0x473c41 created by runtime.createfing in goroutine 1 /usr/local/go/src/runtime/mfinal.go:163 +0x3d goroutine 6 [GC worker (idle)]: runtime.gopark(0x22619dfcbc3dd?, 0x3?, 0xdf?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00008a750 sp=0xc00008a730 pc=0x43fd2e runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc00008a7e0 sp=0xc00008a750 pc=0x420825 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00008a7e8 sp=0xc00008a7e0 pc=0x473c41 created by runtime.gcBgMarkStartWorkers in goroutine 1 /usr/local/go/src/runtime/mgc.go:1219 +0x1c goroutine 18 [GC worker (idle)]: runtime.gopark(0x22619dfcbbae9?, 0x1?, 0x5f?, 0xf3?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000084750 sp=0xc000084730 pc=0x43fd2e runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0000847e0 sp=0xc000084750 pc=0x420825 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000847e8 sp=0xc0000847e0 pc=0x473c41 created by runtime.gcBgMarkStartWorkers in goroutine 1 /usr/local/go/src/runtime/mgc.go:1219 +0x1c goroutine 19 [GC worker (idle)]: runtime.gopark(0x22619dba364a7?, 0x3?, 0x54?, 0x19?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000084f50 sp=0xc000084f30 pc=0x43fd2e runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc000084fe0 sp=0xc000084f50 pc=0x420825 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000084fe8 sp=0xc000084fe0 pc=0x473c41 created by runtime.gcBgMarkStartWorkers in goroutine 1 /usr/local/go/src/runtime/mgc.go:1219 +0x1c goroutine 34 [GC worker (idle)]: runtime.gopark(0x22619dfcbacb9?, 0x3?, 0x67?, 0x94?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000578750 sp=0xc000578730 pc=0x43fd2e runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0005787e0 sp=0xc000578750 pc=0x420825 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0005787e8 sp=0xc0005787e0 pc=0x473c41 created by runtime.gcBgMarkStartWorkers in goroutine 1 /usr/local/go/src/runtime/mgc.go:1219 +0x1c goroutine 20 [GC worker (idle)]: runtime.gopark(0x22619dfcbc6c6?, 0x3?, 0xb9?, 0xb3?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000085750 sp=0xc000085730 pc=0x43fd2e runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0000857e0 sp=0xc000085750 pc=0x420825 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000857e8 sp=0xc0000857e0 pc=0x473c41 created by runtime.gcBgMarkStartWorkers in goroutine 1 /usr/local/go/src/runtime/mgc.go:1219 +0x1c goroutine 35 [GC worker (idle)]: runtime.gopark(0x22619dfcbb737?, 0x1?, 0x59?, 0xee?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000578f50 sp=0xc000578f30 pc=0x43fd2e runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc000578fe0 sp=0xc000578f50 pc=0x420825 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000578fe8 sp=0xc000578fe0 pc=0x473c41 created by runtime.gcBgMarkStartWorkers in goroutine 1 /usr/local/go/src/runtime/mgc.go:1219 +0x1c goroutine 7 [GC worker (idle)]: runtime.gopark(0x22619dfcbbee0?, 0x3?, 0xd6?, 0x9?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00008af50 sp=0xc00008af30 pc=0x43fd2e runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc00008afe0 sp=0xc00008af50 pc=0x420825 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00008afe8 sp=0xc00008afe0 pc=0x473c41 created by runtime.gcBgMarkStartWorkers in goroutine 1 /usr/local/go/src/runtime/mgc.go:1219 +0x1c goroutine 36 [GC worker (idle)]: runtime.gopark(0x22619dfcbc23e?, 0x3?, 0xe4?, 0xb0?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000579750 sp=0xc000579730 pc=0x43fd2e runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0005797e0 sp=0xc000579750 pc=0x420825 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0005797e8 sp=0xc0005797e0 pc=0x473c41 created by runtime.gcBgMarkStartWorkers in goroutine 1 /usr/local/go/src/runtime/mgc.go:1219 +0x1c goroutine 8 [select]: runtime.gopark(0xc00057bf48?, 0x2?, 0x0?, 0x0?, 0xc00057bf14?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00057bdb0 sp=0xc00057bd90 pc=0x43fd2e runtime.selectgo(0xc00057bf48, 0xc00057bf10, 0x0?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc00057bed0 sp=0xc00057bdb0 pc=0x450485 github.com/baidubce/bce-sdk-go/util/log.NewLogger.func1() /go/pkg/mod/github.com/baidubce/bce-sdk-go@v0.9.150/util/log/logger.go:375 +0xa8 fp=0xc00057bfe0 sp=0xc00057bed0 pc=0x161e528 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00057bfe8 sp=0xc00057bfe0 pc=0x473c41 created by github.com/baidubce/bce-sdk-go/util/log.NewLogger in goroutine 1 /go/pkg/mod/github.com/baidubce/bce-sdk-go@v0.9.150/util/log/logger.go:368 +0x125 goroutine 9 [chan receive]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000085e98 sp=0xc000085e78 pc=0x43fd2e runtime.chanrecv(0xc00060e840, 0xc000085fa0, 0x1) /usr/local/go/src/runtime/chan.go:583 +0x3cd fp=0xc000085f10 sp=0xc000085e98 pc=0x40b00d runtime.chanrecv2(0x0?, 0x0?) /usr/local/go/src/runtime/chan.go:447 +0x12 fp=0xc000085f38 sp=0xc000085f10 pc=0x40ac32 github.com/rjeczalik/notify.(*nonrecursiveTree).dispatch(0xc00060e900, 0x0?) /go/pkg/mod/github.com/rjeczalik/notify@v0.9.3/tree_nonrecursive.go:36 +0x45 fp=0xc000085fc0 sp=0xc000085f38 pc=0x2047de5 github.com/rjeczalik/notify.newNonrecursiveTree.func1() /go/pkg/mod/github.com/rjeczalik/notify@v0.9.3/tree_nonrecursive.go:29 +0x25 fp=0xc000085fe0 sp=0xc000085fc0 pc=0x2047d65 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000085fe8 sp=0xc000085fe0 pc=0x473c41 created by github.com/rjeczalik/notify.newNonrecursiveTree in goroutine 1 /go/pkg/mod/github.com/rjeczalik/notify@v0.9.3/tree_nonrecursive.go:29 +0x145 goroutine 10 [chan receive]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000086638 sp=0xc000086618 pc=0x43fd2e runtime.chanrecv(0xc00060e8a0, 0xc000086738, 0x1) /usr/local/go/src/runtime/chan.go:583 +0x3cd fp=0xc0000866b0 sp=0xc000086638 pc=0x40b00d runtime.chanrecv2(0x0?, 0x0?) /usr/local/go/src/runtime/chan.go:447 +0x12 fp=0xc0000866d8 sp=0xc0000866b0 pc=0x40ac32 github.com/rjeczalik/notify.(*nonrecursiveTree).internal(0xc00060e900, 0xc00060e8a0) /go/pkg/mod/github.com/rjeczalik/notify@v0.9.3/tree_nonrecursive.go:81 +0x47 fp=0xc0000867c0 sp=0xc0000866d8 pc=0x2048607 github.com/rjeczalik/notify.newNonrecursiveTree.func2() /go/pkg/mod/github.com/rjeczalik/notify@v0.9.3/tree_nonrecursive.go:30 +0x25 fp=0xc0000867e0 sp=0xc0000867c0 pc=0x2047d05 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000867e8 sp=0xc0000867e0 pc=0x473c41 created by github.com/rjeczalik/notify.newNonrecursiveTree in goroutine 1 /go/pkg/mod/github.com/rjeczalik/notify@v0.9.3/tree_nonrecursive.go:30 +0x193 goroutine 11 [select]: runtime.gopark(0xc00057a790?, 0x2?, 0x80?, 0xf4?, 0xc00057a774?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00057a618 sp=0xc00057a5f8 pc=0x43fd2e runtime.selectgo(0xc00057a790, 0xc00057a770, 0x0?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc00057a738 sp=0xc00057a618 pc=0x450485 github.com/golang/glog.(*fileSink).flushDaemon(0x5636dd8) /go/pkg/mod/github.com/golang/glog@v1.1.0/glog_file.go:345 +0xb9 fp=0xc00057a7c8 sp=0xc00057a738 pc=0xdaecb9 github.com/golang/glog.init.1.func1() /go/pkg/mod/github.com/golang/glog@v1.1.0/glog_file.go:163 +0x25 fp=0xc00057a7e0 sp=0xc00057a7c8 pc=0xdadd85 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00057a7e8 sp=0xc00057a7e0 pc=0x473c41 created by github.com/golang/glog.init.1 in goroutine 1 /go/pkg/mod/github.com/golang/glog@v1.1.0/glog_file.go:163 +0x167 goroutine 12 [select]: runtime.gopark(0xc00057af88?, 0x3?, 0xe0?, 0xbe?, 0xc00057af72?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00057ae18 sp=0xc00057adf8 pc=0x43fd2e runtime.selectgo(0xc00057af88, 0xc00057af6c, 0xc000b56100?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc00057af38 sp=0xc00057ae18 pc=0x450485 go.opencensus.io/stats/view.(*worker).start(0xc000b56100) /go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:292 +0x9f fp=0xc00057afc8 sp=0xc00057af38 pc=0x18a4b7f go.opencensus.io/stats/view.init.0.func1() /go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x25 fp=0xc00057afe0 sp=0xc00057afc8 pc=0x18a3ea5 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00057afe8 sp=0xc00057afe0 pc=0x473c41 created by go.opencensus.io/stats/view.init.0 in goroutine 1 /go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x8d goroutine 13 [sleep]: runtime.gopark(0x2260d45401afd?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00057b788 sp=0xc00057b768 pc=0x43fd2e time.Sleep(0x8bb2c97000) /usr/local/go/src/runtime/time.go:195 +0x125 fp=0xc00057b7c8 sp=0xc00057b788 pc=0x470625 github.com/juicedata/juicefs/pkg/utils.init.0.func2() /go/src/github.com/juicedata/juicefs/pkg/utils/alloc.go:78 +0x1d fp=0xc00057b7e0 sp=0xc00057b7c8 pc=0x67a5fd runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00057b7e8 sp=0xc00057b7e0 pc=0x473c41 created by github.com/juicedata/juicefs/pkg/utils.init.0 in goroutine 1 /go/src/github.com/juicedata/juicefs/pkg/utils/alloc.go:76 +0x11a goroutine 51 [sleep]: runtime.gopark(0x2260d456040fe?, 0xc000ba2001?, 0x90?, 0xbf?, 0x412a5a?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000087760 sp=0xc000087740 pc=0x43fd2e time.Sleep(0x1a3185c5000) /usr/local/go/src/runtime/time.go:195 +0x125 fp=0xc0000877a0 sp=0xc000087760 pc=0x470625 github.com/minio/minio/cmd/logger.(*logOnceType).cleanupRoutine(0xc000f83aa0) /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/logger/logonce.go:64 +0x28 fp=0xc0000877c8 sp=0xc0000877a0 pc=0x20677e8 github.com/minio/minio/cmd/logger.newLogOnceType.func1() /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/logger/logonce.go:71 +0x25 fp=0xc0000877e0 sp=0xc0000877c8 pc=0x2067945 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000877e8 sp=0xc0000877e0 pc=0x473c41 created by github.com/minio/minio/cmd/logger.newLogOnceType in goroutine 1 /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/logger/logonce.go:71 +0x89 goroutine 52 [select]: runtime.gopark(0xc000086fa8?, 0x2?, 0x0?, 0x0?, 0xc000086f7c?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000086e18 sp=0xc000086df8 pc=0x43fd2e runtime.selectgo(0xc000086fa8, 0xc000086f78, 0x0?, 0x0, 0x40aaeb?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000086f38 sp=0xc000086e18 pc=0x450485 github.com/minio/minio/cmd/http.NewDNSCache.func1() /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/http/dial_dnscache.go:131 +0xf0 fp=0xc000086fe0 sp=0xc000086f38 pc=0x2054c70 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000086fe8 sp=0xc000086fe0 pc=0x473c41 created by github.com/minio/minio/cmd/http.NewDNSCache in goroutine 1 /go/pkg/mod/github.com/juicedata/minio@v0.0.0-20240523035739-2e3d9abd6140/cmd/http/dial_dnscache.go:127 +0x2e5 goroutine 82 [sleep]: runtime.gopark(0x2260d460105b1?, 0x409ef7?, 0xc0?, 0x40?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000576f58 sp=0xc000576f38 pc=0x43fd2e time.Sleep(0xdf8475800) /usr/local/go/src/runtime/time.go:195 +0x125 fp=0xc000576f98 sp=0xc000576f58 pc=0x470625 github.com/viki-org/dnscache.(*Resolver).autoRefresh(0x0?, 0xc0015840c0?) /go/pkg/mod/github.com/viki-org/dnscache@v0.0.0-20130720023526-c70c1f23c5d8/dnscache.go:74 +0x25 fp=0xc000576fc0 sp=0xc000576f98 pc=0x1d562c5 github.com/viki-org/dnscache.New.func1() /go/pkg/mod/github.com/viki-org/dnscache@v0.0.0-20130720023526-c70c1f23c5d8/dnscache.go:20 +0x25 fp=0xc000576fe0 sp=0xc000576fc0 pc=0x1d55e45 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000576fe8 sp=0xc000576fe0 pc=0x473c41 created by github.com/viki-org/dnscache.New in goroutine 1 /go/pkg/mod/github.com/viki-org/dnscache@v0.0.0-20130720023526-c70c1f23c5d8/dnscache.go:20 +0xb0 goroutine 83 [sleep]: runtime.gopark(0x22618ea40dd57?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0005776c8 sp=0xc0005776a8 pc=0x43fd2e time.Sleep(0x2540be400) /usr/local/go/src/runtime/time.go:195 +0x125 fp=0xc000577708 sp=0xc0005776c8 pc=0x470625 github.com/juicedata/juicefs/pkg/fuse.(*gidCache).cleanup(0xc000e4be90) /go/src/github.com/juicedata/juicefs/pkg/fuse/gidcache.go:60 +0x33 fp=0xc0005777c8 sp=0xc000577708 pc=0x264b2d3 github.com/juicedata/juicefs/pkg/fuse.newGidCache.func1() /go/src/github.com/juicedata/juicefs/pkg/fuse/gidcache.go:46 +0x25 fp=0xc0005777e0 sp=0xc0005777c8 pc=0x264b265 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0005777e8 sp=0xc0005777e0 pc=0x473c41 created by github.com/juicedata/juicefs/pkg/fuse.newGidCache in goroutine 1 /go/src/github.com/juicedata/juicefs/pkg/fuse/gidcache.go:46 +0x9b goroutine 84 [IO wait]: runtime.gopark(0xc00009db70?, 0x411ddd?, 0x78?, 0xdb?, 0x4b997d?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00009db08 sp=0xc00009dae8 pc=0x43fd2e runtime.netpollblock(0x7fa437e8f330?, 0x408186?, 0x0?) /usr/local/go/src/runtime/netpoll.go:564 +0xf7 fp=0xc00009db40 sp=0xc00009db08 pc=0x438437 internal/poll.runtime_pollWait(0x7fa43812e588, 0x72) /usr/local/go/src/runtime/netpoll.go:343 +0x85 fp=0xc00009db60 sp=0xc00009db40 pc=0x46d9c5 internal/poll.(*pollDesc).wait(0xc000b56000?, 0xc00009dc48?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc00009db88 sp=0xc00009db60 pc=0x4b25e7 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Accept(0xc000b56000) /usr/local/go/src/internal/poll/fd_unix.go:611 +0x2ac fp=0xc00009dc30 sp=0xc00009db88 pc=0x4b7acc net.(*netFD).accept(0xc000b56000) /usr/local/go/src/net/fd_unix.go:172 +0x29 fp=0xc00009dce8 sp=0xc00009dc30 pc=0x629e29 net.(*TCPListener).accept(0xc001560060) /usr/local/go/src/net/tcpsock_posix.go:152 +0x1e fp=0xc00009dd10 sp=0xc00009dce8 pc=0x6438be net.(*TCPListener).Accept(0xc001560060) /usr/local/go/src/net/tcpsock.go:315 +0x30 fp=0xc00009dd40 sp=0xc00009dd10 pc=0x642a70 net/http.(*onceCloseListener).Accept(0x3ddd860?) :1 +0x24 fp=0xc00009dd58 sp=0xc00009dd40 pc=0x896144 net/http.(*Server).Serve(0xc0001f2780, {0x3dd1810, 0xc001560060}) /usr/local/go/src/net/http/server.go:3056 +0x364 fp=0xc00009de88 sp=0xc00009dd58 pc=0x872344 net/http.(*Server).ListenAndServe(0xc0001f2780) /usr/local/go/src/net/http/server.go:2985 +0x71 fp=0xc00009deb8 sp=0xc00009de88 pc=0x871fb1 net/http.ListenAndServe(...) /usr/local/go/src/net/http/server.go:3239 github.com/juicedata/juicefs/cmd.setup.func1() /go/src/github.com/juicedata/juicefs/cmd/main.go:322 +0x26 fp=0xc00009df40 sp=0xc00009deb8 pc=0x26bb6a6 sync.(*Once).doSlow(0x0?, 0x0?) /usr/local/go/src/sync/once.go:74 +0xbf fp=0xc00009dfa0 sp=0xc00009df40 pc=0x47f21f sync.(*Once).Do(0x0?, 0x0?) /usr/local/go/src/sync/once.go:65 +0x19 fp=0xc00009dfc0 sp=0xc00009dfa0 pc=0x47f139 github.com/juicedata/juicefs/cmd.setup.func3() /go/src/github.com/juicedata/juicefs/cmd/main.go:318 +0x25 fp=0xc00009dfe0 sp=0xc00009dfc0 pc=0x2681d85 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00009dfe8 sp=0xc00009dfe0 pc=0x473c41 created by github.com/juicedata/juicefs/cmd.setup in goroutine 1 /go/src/github.com/juicedata/juicefs/cmd/main.go:318 +0x50a goroutine 86 [sleep]: runtime.gopark(0x2261a8bf81c0b?, 0x1?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000579e98 sp=0xc000579e78 pc=0x43fd2e time.Sleep(0x3b9aca00) /usr/local/go/src/runtime/time.go:195 +0x125 fp=0xc000579ed8 sp=0xc000579e98 pc=0x470625 github.com/juicedata/juicefs/pkg/meta.(*openfiles).cleanup(0xc000b15760) /go/src/github.com/juicedata/juicefs/pkg/meta/openfile.go:104 +0x53 fp=0xc000579fc8 sp=0xc000579ed8 pc=0x1383093 github.com/juicedata/juicefs/pkg/meta.newOpenFiles.func1() /go/src/github.com/juicedata/juicefs/pkg/meta/openfile.go:57 +0x25 fp=0xc000579fe0 sp=0xc000579fc8 pc=0x1383005 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000579fe8 sp=0xc000579fe0 pc=0x473c41 created by github.com/juicedata/juicefs/pkg/meta.newOpenFiles in goroutine 1 /go/src/github.com/juicedata/juicefs/pkg/meta/openfile.go:57 +0xaa goroutine 185 [sleep]: runtime.gopark(0x2261924ab68be?, 0xc0015bff38?, 0x25?, 0x1c?, 0xc000b5d020?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0015bff08 sp=0xc0015bfee8 pc=0x43fd2e time.Sleep(0x24179e912) /usr/local/go/src/runtime/time.go:195 +0x125 fp=0xc0015bff48 sp=0xc0015bff08 pc=0x470625 github.com/juicedata/juicefs/pkg/utils.SleepWithJitter(0x2540be400) /go/src/github.com/juicedata/juicefs/pkg/utils/general.go:26 +0x5e fp=0xc0015bff68 sp=0xc0015bff48 pc=0x67127e github.com/juicedata/juicefs/pkg/meta.(*baseMeta).InitMetrics.func1() /go/src/github.com/juicedata/juicefs/pkg/meta/base.go:266 +0x2c fp=0xc0015bffe0 sp=0xc0015bff68 pc=0x136528c runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0015bffe8 sp=0xc0015bffe0 pc=0x473c41 created by github.com/juicedata/juicefs/pkg/meta.(*baseMeta).InitMetrics in goroutine 1 /go/src/github.com/juicedata/juicefs/pkg/meta/base.go:258 +0x39a goroutine 207 [sleep]: runtime.gopark(0x2260d5b81cd0e?, 0xe0?, 0x0?, 0xee?, 0xc00191edb0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0015b9e50 sp=0xc0015b9e30 pc=0x43fd2e time.Sleep(0x34630b8a000) /usr/local/go/src/runtime/time.go:195 +0x125 fp=0xc0015b9e90 sp=0xc0015b9e50 pc=0x470625 github.com/juicedata/juicefs/pkg/usage.ReportUsage({0x3e15ff0, 0xc000c12f50}, {0xc00155cdf8, 0x18}) /go/src/github.com/juicedata/juicefs/pkg/usage/usage.go:91 +0x138 fp=0xc0015b9fb0 sp=0xc0015b9e90 pc=0x262a3b8 github.com/juicedata/juicefs/cmd.initBackgroundTasks.func2() /go/src/github.com/juicedata/juicefs/cmd/mount.go:386 +0x2c fp=0xc0015b9fe0 sp=0xc0015b9fb0 pc=0x2687f6c runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0015b9fe8 sp=0xc0015b9fe0 pc=0x473c41 created by github.com/juicedata/juicefs/cmd.initBackgroundTasks in goroutine 1 /go/src/github.com/juicedata/juicefs/cmd/mount.go:386 +0x496 goroutine 208 [sleep]: runtime.gopark(0x2261a8bf663d9?, 0x3?, 0x28?, 0xb7?, 0x413e45?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00008b6f8 sp=0xc00008b6d8 pc=0x43fd2e time.Sleep(0x3b9aca00) /usr/local/go/src/runtime/time.go:195 +0x125 fp=0xc00008b738 sp=0xc00008b6f8 pc=0x470625 github.com/juicedata/juicefs/pkg/vfs.(*dataReader).checkReadBuffer(0xc000c11680) /go/src/github.com/juicedata/juicefs/pkg/vfs/reader.go:728 +0x2c fp=0xc00008b7c8 sp=0xc00008b738 pc=0x1ec546c github.com/juicedata/juicefs/pkg/vfs.NewDataReader.func1() /go/src/github.com/juicedata/juicefs/pkg/vfs/reader.go:707 +0x25 fp=0xc00008b7e0 sp=0xc00008b7c8 pc=0x1ec5405 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00008b7e8 sp=0xc00008b7e0 pc=0x473c41 created by github.com/juicedata/juicefs/pkg/vfs.NewDataReader in goroutine 1 /go/src/github.com/juicedata/juicefs/pkg/vfs/reader.go:707 +0x1db goroutine 114 [sleep]: runtime.gopark(0x2261a8bf82c46?, 0x0?, 0xa8?, 0x2f?, 0x8ce58c?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc001012f48 sp=0xc001012f28 pc=0x43fd2e time.Sleep(0x3b9aca00) /usr/local/go/src/runtime/time.go:195 +0x125 fp=0xc001012f88 sp=0xc001012f48 pc=0x470625 github.com/juicedata/juicefs/pkg/chunk.NewCachedStore.func2() /go/src/github.com/juicedata/juicefs/pkg/chunk/cached_store.go:806 +0x30 fp=0xc001012fe0 sp=0xc001012f88 pc=0x1e7da70 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc001012fe8 sp=0xc001012fe0 pc=0x473c41 created by github.com/juicedata/juicefs/pkg/chunk.NewCachedStore in goroutine 1 /go/src/github.com/juicedata/juicefs/pkg/chunk/cached_store.go:798 +0x5ff goroutine 154 [sleep]: runtime.gopark(0x2261850d2b8d4?, 0xc000099e10?, 0x25?, 0x1c?, 0xc000b5d020?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000099de0 sp=0xc000099dc0 pc=0x43fd2e time.Sleep(0x2c0422b30) /usr/local/go/src/runtime/time.go:195 +0x125 fp=0xc000099e20 sp=0xc000099de0 pc=0x470625 github.com/juicedata/juicefs/pkg/utils.SleepWithJitter(0x2cb417800) /go/src/github.com/juicedata/juicefs/pkg/utils/general.go:26 +0x5e fp=0xc000099e40 sp=0xc000099e20 pc=0x67127e github.com/juicedata/juicefs/pkg/meta.(*baseMeta).refresh(0xc00007cf00) /go/src/github.com/juicedata/juicefs/pkg/meta/base.go:443 +0x3f fp=0xc000099fc8 sp=0xc000099e40 pc=0x1366bff github.com/juicedata/juicefs/pkg/meta.(*baseMeta).NewSession.func2() /go/src/github.com/juicedata/juicefs/pkg/meta/base.go:372 +0x25 fp=0xc000099fe0 sp=0xc000099fc8 pc=0x13668a5 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000099fe8 sp=0xc000099fe0 pc=0x473c41 created by github.com/juicedata/juicefs/pkg/meta.(*baseMeta).NewSession in goroutine 1 /go/src/github.com/juicedata/juicefs/pkg/meta/base.go:372 +0x6b goroutine 155 [select, locked to thread]: runtime.gopark(0xc001013fa8?, 0x2?, 0x0?, 0x0?, 0xc001013fa4?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc001013e38 sp=0xc001013e18 pc=0x43fd2e runtime.selectgo(0xc001013fa8, 0xc001013fa0, 0x0?, 0x0, 0x8?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc001013f58 sp=0xc001013e38 pc=0x450485 runtime.ensureSigM.func1() /usr/local/go/src/runtime/signal_unix.go:1014 +0x19f fp=0xc001013fe0 sp=0xc001013f58 pc=0x469eff runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc001013fe8 sp=0xc001013fe0 pc=0x473c41 created by runtime.ensureSigM in goroutine 1 /usr/local/go/src/runtime/signal_unix.go:997 +0xc8 goroutine 156 [chan receive]: runtime.gopark(0xc0005746c0?, 0x40a5f3?, 0xa0?, 0x47?, 0xc000fbe340?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0005746a0 sp=0xc000574680 pc=0x43fd2e runtime.chanrecv(0xc001585320, 0xc0005747c0, 0x1) /usr/local/go/src/runtime/chan.go:583 +0x3cd fp=0xc000574718 sp=0xc0005746a0 pc=0x40b00d runtime.chanrecv1(0xc000bdf6e0?, 0xc000574778?) /usr/local/go/src/runtime/chan.go:442 +0x12 fp=0xc000574740 sp=0xc000574718 pc=0x40ac12 github.com/juicedata/juicefs/cmd.initForSvc.func2() /go/src/github.com/juicedata/juicefs/cmd/gateway.go:246 +0x58 fp=0xc0005747e0 sp=0xc000574740 pc=0x266e638 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0005747e8 sp=0xc0005747e0 pc=0x473c41 created by github.com/juicedata/juicefs/cmd.initForSvc in goroutine 1 /go/src/github.com/juicedata/juicefs/cmd/gateway.go:245 +0x785 goroutine 23 [syscall]: runtime.notetsleepg(0x0?, 0x0?) /usr/local/go/src/runtime/lock_futex.go:236 +0x29 fp=0xc000577fa0 sp=0xc000577f68 pc=0x410729 os/signal.signal_recv() /usr/local/go/src/runtime/sigqueue.go:152 +0x29 fp=0xc000577fc0 sp=0xc000577fa0 pc=0x46fc29 os/signal.loop() /usr/local/go/src/os/signal/signal_unix.go:23 +0x13 fp=0xc000577fe0 sp=0xc000577fc0 pc=0x1efb6d3 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000577fe8 sp=0xc000577fe0 pc=0x473c41 created by os/signal.Notify.func1.1 in goroutine 1 /usr/local/go/src/os/signal/signal.go:151 +0x1f goroutine 163 [IO wait]: runtime.gopark(0xc000fb0c60?, 0x411ddd?, 0x68?, 0xc?, 0x4b997d?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000fb0bf8 sp=0xc000fb0bd8 pc=0x43fd2e runtime.netpollblock(0x7fa437bf5ff8?, 0x408186?, 0x0?) /usr/local/go/src/runtime/netpoll.go:564 +0xf7 fp=0xc000fb0c30 sp=0xc000fb0bf8 pc=0x438437 internal/poll.runtime_pollWait(0x7fa43812e398, 0x72) /usr/local/go/src/runtime/netpoll.go:343 +0x85 fp=0xc000fb0c50 sp=0xc000fb0c30 pc=0x46d9c5 internal/poll.(*pollDesc).wait(0xc000111500?, 0xc000fb0d38?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc000fb0c78 sp=0xc000fb0c50 pc=0x4b25e7 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Accept(0xc000111500) /usr/local/go/src/internal/poll/fd_unix.go:611 +0x2ac fp=0xc000fb0d20 sp=0xc000fb0c78 pc=0x4b7acc net.(*netFD).accept(0xc000111500) /usr/local/go/src/net/fd_unix.go:172 +0x29 fp=0xc000fb0dd8 sp=0xc000fb0d20 pc=0x629e29 net.(*TCPListener).accept(0xc000e0c400) /usr/local/go/src/net/tcpsock_posix.go:152 +0x1e fp=0xc000fb0e00 sp=0xc000fb0dd8 pc=0x6438be net.(*TCPListener).Accept(0xc000e0c400) /usr/local/go/src/net/tcpsock.go:315 +0x30 fp=0xc000fb0e30 sp=0xc000fb0e00 pc=0x642a70 net/http.(*onceCloseListener).Accept(0x3ddd860?) :1 +0x24 fp=0xc000fb0e48 sp=0xc000fb0e30 pc=0x896144 net/http.(*Server).Serve(0xc0010221e0, {0x3dd1810, 0xc000e0c400}) /usr/local/go/src/net/http/server.go:3056 +0x364 fp=0xc000fb0f78 sp=0xc000fb0e48 pc=0x872344 net/http.Serve(...) /usr/local/go/src/net/http/server.go:2595 github.com/juicedata/juicefs/cmd.exposeMetrics.func1() /go/src/github.com/juicedata/juicefs/cmd/mount.go:129 +0x45 fp=0xc000fb0fe0 sp=0xc000fb0f78 pc=0x26852a5 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000fb0fe8 sp=0xc000fb0fe0 pc=0x473c41 created by github.com/juicedata/juicefs/cmd.exposeMetrics in goroutine 1 /go/src/github.com/juicedata/juicefs/cmd/mount.go:128 +0x69c goroutine 209 [sleep]: runtime.gopark(0x2261aa602d5c3?, 0x2?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00008be28 sp=0xc00008be08 pc=0x43fd2e time.Sleep(0x5f5e100) /usr/local/go/src/runtime/time.go:195 +0x125 fp=0xc00008be68 sp=0xc00008be28 pc=0x470625 github.com/juicedata/juicefs/pkg/vfs.(*dataWriter).flushAll(0xc000c116e0) /go/src/github.com/juicedata/juicefs/pkg/vfs/writer.go:474 +0x31 fp=0xc00008bfc8 sp=0xc00008be68 pc=0x1edaed1 github.com/juicedata/juicefs/pkg/vfs.NewDataWriter.func1() /go/src/github.com/juicedata/juicefs/pkg/vfs/writer.go:444 +0x25 fp=0xc00008bfe0 sp=0xc00008bfc8 pc=0x1edae65 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00008bfe8 sp=0xc00008bfe0 pc=0x473c41 created by github.com/juicedata/juicefs/pkg/vfs.NewDataWriter in goroutine 1 /go/src/github.com/juicedata/juicefs/pkg/vfs/writer.go:444 +0x17f goroutine 210 [sleep]: runtime.gopark(0x2261a8bf8224a?, 0x36ea088?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00100edd8 sp=0xc00100edb8 pc=0x43fd2e time.Sleep(0x3b9aca00) /usr/local/go/src/runtime/time.go:195 +0x125 fp=0xc00100ee18 sp=0xc00100edd8 pc=0x470625 github.com/juicedata/juicefs/pkg/fs.(*FileSystem).cleanupCache(0xc000e87a40) /go/src/github.com/juicedata/juicefs/pkg/fs/fs.go:245 +0x31 fp=0xc00100efc8 sp=0xc00100ee18 pc=0x1f21291 github.com/juicedata/juicefs/pkg/fs.NewFileSystem.func1() /go/src/github.com/juicedata/juicefs/pkg/fs/fs.go:192 +0x25 fp=0xc00100efe0 sp=0xc00100efc8 pc=0x1f21045 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00100efe8 sp=0xc00100efe0 pc=0x473c41 created by github.com/juicedata/juicefs/pkg/fs.NewFileSystem in goroutine 1 /go/src/github.com/juicedata/juicefs/pkg/fs/fs.go:192 +0x5a5 goroutine 220 [chan receive]: runtime.gopark(0x45cf9e?, 0xc000073900?, 0x8?, 0x40?, 0x274a1d7af70a1?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0015bbd70 sp=0xc0015bbd50 pc=0x43fd2e runtime.chanrecv(0xc001700000, 0xc0015bbf38, 0x1) /usr/local/go/src/runtime/chan.go:583 +0x3cd fp=0xc0015bbde8 sp=0xc0015bbd70 pc=0x40b00d runtime.chanrecv2(0x4e94914f0000?, 0x0?) /usr/local/go/src/runtime/chan.go:447 +0x12 fp=0xc0015bbe10 sp=0xc0015bbde8 pc=0x40ac32 github.com/juicedata/juicefs/pkg/gateway.(*jfsObjects).cleanup(0xc000fd5d10) /go/src/github.com/juicedata/juicefs/pkg/gateway/gateway.go:1139 +0x6b fp=0xc0015bbfc8 sp=0xc0015bbe10 pc=0x25f33cb github.com/juicedata/juicefs/pkg/gateway.NewJFSGateway.func1() /go/src/github.com/juicedata/juicefs/pkg/gateway/gateway.go:68 +0x25 fp=0xc0015bbfe0 sp=0xc0015bbfc8 pc=0x25e5dc5 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0015bbfe8 sp=0xc0015bbfe0 pc=0x473c41 created by github.com/juicedata/juicefs/pkg/gateway.NewJFSGateway in goroutine 1 /go/src/github.com/juicedata/juicefs/pkg/gateway/gateway.go:68 +0x27a goroutine 774 [IO wait]: runtime.gopark(0x16000000fa?, 0xb?, 0x0?, 0x0?, 0xf?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0016115c8 sp=0xc0016115a8 pc=0x43fd2e runtime.netpollblock(0x48b018?, 0x408186?, 0x0?) /usr/local/go/src/runtime/netpoll.go:564 +0xf7 fp=0xc001611600 sp=0xc0016115c8 pc=0x438437 internal/poll.runtime_pollWait(0x7fa43812e0b0, 0x72) /usr/local/go/src/runtime/netpoll.go:343 +0x85 fp=0xc001611620 sp=0xc001611600 pc=0x46d9c5 internal/poll.(*pollDesc).wait(0xc0018a1200?, 0xc001652000?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc001611648 sp=0xc001611620 pc=0x4b25e7 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc0018a1200, {0xc001652000, 0x1800, 0x1800}) /usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a fp=0xc0016116e0 sp=0xc001611648 pc=0x4b38da net.(*netFD).Read(0xc0018a1200, {0xc001652000?, 0xc001652038?, 0x1a?}) /usr/local/go/src/net/fd_posix.go:55 +0x25 fp=0xc001611728 sp=0xc0016116e0 pc=0x627e05 net.(*conn).Read(0xc000b82b48, {0xc001652000?, 0x34?, 0xc000e46fb8?}) /usr/local/go/src/net/net.go:179 +0x45 fp=0xc001611770 sp=0xc001611728 pc=0x63a885 net.(*TCPConn).Read(0xc001611808?, {0xc001652000?, 0xc001664ed0?, 0x18?}) :1 +0x25 fp=0xc0016117a0 sp=0xc001611770 pc=0x64dca5 crypto/tls.(*atLeastReader).Read(0xc001664ed0, {0xc001652000?, 0xc001664ed0?, 0x0?}) /usr/local/go/src/crypto/tls/conn.go:805 +0x3b fp=0xc0016117e8 sp=0xc0016117a0 pc=0x7c807b bytes.(*Buffer).ReadFrom(0xc000e470a8, {0x3dba540, 0xc001664ed0}) /usr/local/go/src/bytes/buffer.go:211 +0x98 fp=0xc001611840 sp=0xc0016117e8 pc=0x50cc78 crypto/tls.(*Conn).readFromUntil(0xc000e46e00, {0x3db9fe0?, 0xc000b82b48}, 0x17cd?) /usr/local/go/src/crypto/tls/conn.go:827 +0xde fp=0xc001611880 sp=0xc001611840 pc=0x7c825e crypto/tls.(*Conn).readRecordOrCCS(0xc000e46e00, 0x0) /usr/local/go/src/crypto/tls/conn.go:625 +0x250 fp=0xc001611c20 sp=0xc001611880 pc=0x7c5830 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:587 crypto/tls.(*Conn).Read(0xc000e46e00, {0xc001607000, 0x1000, 0x857ca5?}) /usr/local/go/src/crypto/tls/conn.go:1369 +0x158 fp=0xc001611c90 sp=0xc001611c20 pc=0x7cbaf8 bufio.(*Reader).Read(0xc0018b9080, {0xc0018c0580, 0x9, 0x552ca80?}) /usr/local/go/src/bufio/bufio.go:244 +0x197 fp=0xc001611cc8 sp=0xc001611c90 pc=0x5f3d97 io.ReadAtLeast({0x3db79a0, 0xc0018b9080}, {0xc0018c0580, 0x9, 0x9}, 0x9) /usr/local/go/src/io/io.go:335 +0x90 fp=0xc001611d10 sp=0xc001611cc8 pc=0x4aaa10 io.ReadFull(...) /usr/local/go/src/io/io.go:354 net/http.http2readFrameHeader({0xc0018c0580, 0x9, 0xc00191ee10?}, {0x3db79a0?, 0xc0018b9080?}) /usr/local/go/src/net/http/h2_bundle.go:1635 +0x65 fp=0xc001611d60 sp=0xc001611d10 pc=0x831345 net/http.(*http2Framer).ReadFrame(0xc0018c0540) /usr/local/go/src/net/http/h2_bundle.go:1899 +0x85 fp=0xc001611e08 sp=0xc001611d60 pc=0x831a85 net/http.(*http2clientConnReadLoop).run(0xc001611f98) /usr/local/go/src/net/http/h2_bundle.go:9370 +0x11f fp=0xc001611f60 sp=0xc001611e08 pc=0x854c3f net/http.(*http2ClientConn).readLoop(0xc000e8a480) /usr/local/go/src/net/http/h2_bundle.go:9265 +0x65 fp=0xc001611fc8 sp=0xc001611f60 pc=0x8541c5 net/http.(*http2Transport).newClientConn.func3() /usr/local/go/src/net/http/h2_bundle.go:7937 +0x25 fp=0xc001611fe0 sp=0xc001611fc8 pc=0x84d085 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc001611fe8 sp=0xc001611fe0 pc=0x473c41 created by net/http.(*http2Transport).newClientConn in goroutine 773 /usr/local/go/src/net/http/h2_bundle.go:7937 +0xcbe
zhijian-pro commented 3 months ago

@Mangoyap Thanks for the feedback.It is a compatibility issue introduced in 1.2, we will fix it as soon as possible.