<ERROR>: error: WRONGTYPE Operation against a key holding the wrong kind of value

[letic]

What happened: Trying to gc or fsck end in redis error.

2023/03/24 01:34:31.537100 juicefs[1798186] : found leaked chunk ca6b19fb2021894a35ddbf17caf40357/imagePath-1fba7e075940fa8e318b8efcc0782d2b-user_status-app.svg [redis.go:2277] 2023/03/24 01:34:34.014683 juicefs[1798186] : list slices: WRONGTYPE Operation against a key holding the wrong kind of value [redis.go:2752] 2023/03/24 01:34:34.015778 juicefs[1798186] : error: WRONGTYPE Operation against a key holding the wrong kind of value goroutine 1 [running]: runtime/debug.Stack() /usr/local/go/src/runtime/debug/stack.go:24 +0x65 github.com/juicedata/juicefs/pkg/meta.errno({0x325bc60, 0xc0126ddc10}) /go/src/github.com/juicedata/juicefs/pkg/meta/utils.go:104 +0xc5 github.com/juicedata/juicefs/pkg/meta.(redisMeta).ListSlices(0xc000292140, {0x3285c30?, 0xc006b32640}, 0xc006b3c270, 0x0, 0xc006667c20) /go/src/github.com/juicedata/juicefs/pkg/meta/redis.go:2776 +0x1a9 github.com/juicedata/juicefs/cmd.fsck(0xc000bf6800) /go/src/github.com/juicedata/juicefs/cmd/fsck.go:111 +0x81b github.com/urfave/cli/v2.(Command).Run(0xc0008c2900, 0xc000bf6740) /go/pkg/mod/github.com/urfave/cli/v2@v2.4.0/command.go:163 +0x5dc github.com/urfave/cli/v2.(App).RunContext(0xc000682d00, {0x3271840?, 0xc000058090}, {0xc000bf6200, 0x4, 0x4}) /go/pkg/mod/github.com/urfave/cli/v2@v2.4.0/app.go:313 +0xb7d github.com/urfave/cli/v2.(App).Run(...) /go/pkg/mod/github.com/urfave/cli/v2@v2.4.0/app.go:224 github.com/juicedata/juicefs/cmd.Main({0xc000bf6080?, 0x4?, 0x4?}) /go/src/github.com/juicedata/juicefs/cmd/main.go:86 +0x246d main.main() /go/src/github.com/juicedata/juicefs/main.go:29 +0x2e [utils.go:104] 2023/03/24 01:34:34.015950 juicefs[1798186] : list all slices: input/output error [fsck.go:113]

What you expected to happen: That it would delete the bad chunk, or handle this kind of error in redis. How is it possible to get the redis key that make it crash ? This way I can just delete and re-run a gc/fsck. This is only used for backups so I can loose this chunk. How to reproduce it (as minimally and precisely as possible): I just need to run juicefs gc or fsck to make it crash

Environment:

• JuiceFS version: 1.0.3+2022-12-27.e4bf15a
• Cloud provider or hardware configuration running JuiceFS: Custom AMD Ryzen 7 3700X NAS build
• OS: TrueNas Scale Bluefin 22.12.1 (Debian 11 Bullseye)
• Kernel : 5.15.79+truenas
• Object storage : AWS S3
• Metadata engine info: redis/stable,stable-security 5:6.0.16-1+deb11u2
• Network connectivity: local network (LXC running on the same host)

[SandyXSD]

The slice keys added by JuiceFS must be a Redis list. You may find the invalid keys with:

for k in `redis-cli --scan --pattern 'c*_*'`; do t=`redis-cli type $k`; if [ $t != 'list' ]; then echo $k $t; fi; done

[letic]

Hey @SandyXSD ,

Thank you so much for your reactivity!

The redis key was the chunk "ca6b19fb2021894a35ddbf17caf40357/imagePath-1fba7e075940fa8e318b8efcc0782d2b-user_status-app.svg" id. Deleting it with the redis-cli solved it.

I don't know if you want to keep this issue open to handle these kind of errors more gracefully, or is it the expected behaviour ?

Thanks letic

[SandyXSD]

We'd better log more information to simplify the troubleshooting.