juicefs gc redis://some-host-name crashed

[steven-varga]

What happened: stack dump What you expected to happen: complete GC with an error code or OK How to reproduce it (as minimally and precisely as possible): root@master:~# juicefs gc redis://master 2022/01/29 13:16:49.609343 juicefs[805687] : Meta address: redis://master 2022/01/29 13:16:49.613709 juicefs[805687] : AOF is not enabled, you may lose data if Redis is not shutdown properly. 2022/01/29 13:16:49.613865 juicefs[805687] : Ping redis: 106.387µs 2022/01/29 13:16:49.614201 juicefs[805687] : Data use s3://h5-trading-us-east-1/io/ listed slices counter: 0 ⠴
listed slices counter: 17886 ⠧
2022/01/29 13:16:51.992812 juicefs[805687] : list slices: WRONGTYPE Operation against a key holding the wrong kind of value 2022/01/29 13:16:51.993014 juicefs[805687] : error: WRONGTYPE Operation against a key holding the wrong kind of value goroutine 1 [running]: runtime/debug.Stack(0x4000118050, 0x2768ece, 0x3) /opt/go/src/runtime/debug/stack.go:24 +0x88 github.com/juicedata/juicefs/pkg/meta.errno(0x2c59540, 0x4001b8ada0, 0x2786071) /mnt/ebs/CLUSTER/arm64/juicefs/pkg/meta/redis.go:888 +0xd8 github.com/juicedata/juicefs/pkg/meta.(redisMeta).ListSlices(0x400092e000, 0x2cd61e0, 0x4001ac7aa0, 0x4001aceae0, 0x0, 0x40002b34d0, 0x40007e5440) /mnt/ebs/CLUSTER/arm64/juicefs/pkg/meta/redis.go:2857 +0x46c main.gc(0x40007dc880, 0x4, 0x6) /mnt/ebs/CLUSTER/arm64/juicefs/cmd/gc.go:150 +0x754 github.com/urfave/cli/v2.(Command).Run(0x400065ac60, 0x40007dc700, 0x0, 0x0) /home/steven/go/pkg/mod/github.com/urfave/cli/v2@v2.3.0/command.go:163 +0x3d8 github.com/urfave/cli/v2.(App).RunContext(0x40006361a0, 0x2cad940, 0x4000052078, 0x40007dc2c0, 0x3, 0x4, 0x0, 0x0) /home/steven/go/pkg/mod/github.com/urfave/cli/v2@v2.3.0/app.go:313 +0x618 github.com/urfave/cli/v2.(App).Run(...) /home/steven/go/pkg/mod/github.com/urfave/cli/v2@v2.3.0/app.go:224 main.main() /mnt/ebs/CLUSTER/arm64/juicefs/cmd/main.go:100 +0x19c4

Anything else we need to know? Not sure how I got to this point; but here I am with a totally messed up filesystem Environment:

• JuiceFS version (use juicefs --version) or Hadoop Java SDK version:0.17.1 (2021-10-28 4d11c4bb)
• Cloud provider or hardware configuration running JuiceFS: AWS S3, Redis server v=6.2.5 sha=00000000:0 malloc=jemalloc-5.1.0 bits=64 build=f272b47bc83a1122
• OS (e.g cat /etc/os-release): 20.04.3 LTS
• Kernel (e.g. uname -a): Linux master 5.11.0-1027-aws
• Object storage (cloud provider and region, or self maintained): AWS S3
• Metadata engine info (version, cloud provider managed or self maintained): Redis server v=6.2.5 sha=00000000:0 malloc=jemalloc-5.1.0
• Network connectivity (JuiceFS to metadata engine, JuiceFS to object storage): same host
• Others:

[davies]

@steven-varga It did not crash, just failed to list all slices. It seems that there is a like with pattern c*_* but is not a List, can you adding some logging in the ListSlices of redis.go to find out the key?

Can you tell more on how it messed up?

[davies]

close stale issue