Permanent zpool errors after a few days of making natively encrypted zvol snapshots with sanoid

[rjycnfynby]

System information

Type	Version/Name
Distribution Name	Gentoo
Distribution Version	default/linux/amd64/17.1 (stable) profile
Kernel Version	6.6.13-gentoo-dist
Architecture	x86_64
OpenZFS Version	zfs-2.2.2-r1-gentoo / zfs-kmod-2.2.2-r0-gentoo

Describe the problem you're observing

After about several days of making autosnapshots on slightly over then twenty natively encrypted zvols hourly scheduled sanoid cronjob starts making errors "cannot iterate filesystems: I/O error" while making snapshots. "zpool status -vx" command gives an output "errors: Permanent errors have been detected in the following files:" and then blank. Getting the same "cannot iterate filesystems: I/O error" errors while trying to list snapshots on some of the zvols. Currently I have about 16 zvols in such error state. Total amount of snapshots is slightly less than a thousand.

"zfs list -t snap -r tank0 | wc -l" command gives me 23 "cannot iterate filesystems: I/O error" lines and the result of 991 to be exact. No errors in dmesg found. This particular pool was made out of three mirrored WD SA500 SSDs which support trim/unmap on LSI controllers but similar results were observed on a different servers and disks.

At least four different servers with a similar setup were failing the same way. Previously I was getting similar issues with ZFS version 2.1.14 and kernel 6.1.69-gentoo-dist with a slight difference that "zpool status -vx" command was giving more detailed output with the list of exact failing snapshots and I could also see an increasing kcf_ops_failed counter using command "grep . /proc/spl/kstat/kcf/*". Later Gentoo marked ZFS 2.2.2 as a stable and I decided to try it one more time with a newer version.

Different server pools started to failure after 3 or 5 days of uptime. All of them had less than a thousand snapshots.

Describe how to reproduce the problem

1. Install latest ZFS and binary distribution kernel;
2. Create zpool with enabled autotrim (probably irrelevant but that's what I'm doing on SSD pools);
3. Enable LZ4 compression on root dataset (probably irrelevant);
4. Create dataset with enabled autotrim (probably irrelevant);
5. Create an encrypted dataset named "encrypted" which will hold the rest of datasets and zvols;
6. Mount zvols to VMs using Xen 4.16.5 hypervisor (might be irrelevant);
7. Install sanoid and configure it to make and keep last 36 "hourly", 4 "weekly" and 2 "monthly" snapshots almost for each zvol;
8. Configure syncoid on a remote server to replicate snapshots from the source server (probably irrelevant);
9. Then wait for few days until the issue will start to appear on more and more zvols generating errors "cannot iterate filesystems: I/O error" during the snapshot operations;

Include any warning/errors/backtraces from the system logs

I couldn't find any related errors in system logs.

[ckruijntjens]

I have the same issue with Debian 12 zfs version 2.2.4-1

Any update on this zfs bug?

[rjycnfynby]

I had to migrate data back to an unencrypted pool since it was a constant problem.

[ckruijntjens]

I had to migrate data back to an unencrypted pool since it was a constant problem.

i understand but this is not the solution. there is a bug in zfs that is causing this. I hope the are looking at this problem soon as it is really anoying.

[IvanVolosyuk]

Is it the same as #12014? Can you try ZFS 2.2.5 or later?

[ckruijntjens]

Is it the same as #12014? Can you try ZFS 2.2.5 or later?

No it is not fullt the same. After some time i also get corruption errors in my encrypted pool. However when i look for the files that are corrupted i get an empty list.

If i then start a scrub cancel it after 1 procent, reboot and run a scrub again everything is normal again.

With zfs version 2.2.5 i had the same issue. I did a upgrade to the latest zfs version. Hopefully it is gene with the latest version.

[ckruijntjens]

With version 2.2.6 is still get the same errors after a couple of days with a native zfs encrypted pool.

This bug is really anoying.

[rjycnfynby]

With version 2.2.6 is still get the same errors after a couple of days with a native zfs encrypted pool.

This bug is really anoying.

I would say that it blocks us from using native encryption in production environments or any environment that is going to work longer than a couple of days with multiple replicating snapshots.

Has anybody managed to replicate this bug in TrueNAS on Linux by any chance?

[ckruijntjens]

With version 2.2.6 is still get the same errors after a couple of days with a native zfs encrypted pool. This bug is really anoying.

I would say that it blocks us from using native encryption in production environments or any environment that is going to work longer than a couple of days with multiple replicating snapshots.

Has anybody managed to replicate this bug in TrueNAS on Linux by any chance?

I agree. I think i am going to create a new pool without native encryption.

[aaltonenp]

Is it known in what version this started? I'm still running Ubuntu 20.04 LTS with zfs 0.8.3 and not seeing this problem. But I'm currently sending only one encrypted dataset to another host and from there to a third host where there's multiple encrypted datasets. I'm doing it with pyznap. Maybe my use case is simple enough with only a single dataset to not trigger this, or it started in a later version.

I dread updating to newer version when LTS updates end.

[tweax-sarl]

I read an article a couple months ago mentioning a bug in recent openzfs related to encrypted datasets. In the meanwhile I forgot... Installed two servers running Ubuntu 24.04.1 LTS (GNU/Linux 6.8.0-45-generic x86_64) for two months now. OpenZFS is 2.2.2. The machines run a couple of VMs in qemu-kvm. Everything seemed to work fine, until a week ago I started cross-copying the encrypted volumes between each other using sanoid / syncoid for a failover scenario. There is process control to ensure that only one syncoid job runs at a time. Syncoid does not create snapshots itself but only shuffles the latest one to the other server. It took not even one day for the first "cannot iterate filesystems: I/O error" to arrive on one of the two servers. No problem on Ubuntu 20.04.6 LTS (GNU/Linux 5.4.0-177-generic x86_64) with OpenZFS 0.8.3, though. Since I was not really keen on reinstalling the servers with Ubuntu 20.04.6 LTS I migrated the volumes onto non-encrypted datasets and hope that this will do the job.

@aaltonenp I think for the moment you are very well advised with Ubuntu 20.04 LTS.

[ckruijntjens]

When will the issue be fixed? The latest version still has this problem.........

[ckruijntjens]

@zfs team. Why are such critical bugs like this one months open. No fix? Still you push new versions of zfs.

[Germano0]

Cannot reproduce on a RHEL 9 system currently having ZFS 2.1.15, Sanoid 2.2.0 and running this OS since September 2022

[ckruijntjens]

Cannot reproduce on a RHEL 9 system currently having ZFS 2.1.15, Sanoid 2.2.0 and running this OS since September 2022

Are you also using syncoid to replicate to other machine?

I am having this issue every 5 days.

[Germano0]

Are you also using syncoid to replicate to other machine?

No, I am replicating only via zfs send/recv

[ckruijntjens]

Are you also using syncoid to replicate to other machine?

No, I am replicating only via zfs send/recv

Ok,

I am using the following versions and os.

Debian Bookworm Using bp kernel - Debian 6.9.7-1~bpo12+1

ZFS VERSION alsobackports.

zfs-2.2.6-1~bpo12+3 zfs-kmod-2.2.6-1~bpo12+3

sanoid version /usr/sbin/sanoid version 2.1.0

syncoid version /usr/sbin/syncoid version 2.1.0

Every 5 days syncoid stops replicating to the other machine because of io errors. My pool show errors however when i want to show what files are the problem with the -v option its an empty list. Ik i scrub 2 times the errors are gone.(first one needs to be canceled after 1 %)

And ofcourse i am using native zfs encryption. i dont understand some people are having problems with this like me and others dont.

[mcmilk]

Debian Bookworm Using bp kernel - Debian 6.9.7-1~bpo12+1

ZFS VERSION alsobackports.

zfs-2.2.6-1~bpo12+3 zfs-kmod-2.2.6-1~bpo12+3

sanoid version /usr/sbin/sanoid version 2.1.0

syncoid version /usr/sbin/syncoid version 2.1.0

Every 5 days syncoid stops replicating to the other machine because of io errors. My pool show errors however when i want to show what files are the problem with the -v option its an empty list. Ik i scrub 2 times the errors are gone.(first one needs to be canceled after 1 %)

And ofcourse i am using native zfs encryption. i dont understand some people are having problems with this like me and others dont.

What encryption and checksumming algo do you use?

[ckruijntjens]

Debian Bookworm Using bp kernel - Debian 6.9.7-1~bpo12+1 ZFS VERSION alsobackports. zfs-2.2.6-1~bpo12+3 zfs-kmod-2.2.6-1~bpo12+3 sanoid version /usr/sbin/sanoid version 2.1.0 syncoid version /usr/sbin/syncoid version 2.1.0 Every 5 days syncoid stops replicating to the other machine because of io errors. My pool show errors however when i want to show what files are the problem with the -v option its an empty list. Ik i scrub 2 times the errors are gone.(first one needs to be canceled after 1 %) And ofcourse i am using native zfs encryption. i dont understand some people are having problems with this like me and others dont.

What encryption and checksumming algo do you use?

Hi i use encryption,

aes-256-gcm

Checksum is default.

[mcmilk]

Okay, I will try to dig deeper into this thing. AES for ARM is also ongoing ;-)

[ckruijntjens]

Okay, I will try to dig deeper into this thing. AES for ARM is also ongoing ;-)

That would be great. I would be super of we get a fix for this problem. Thank you verry much.

[ckruijntjens]

Okay, I will try to dig deeper into this thing. AES for ARM is also ongoing ;-)

Ps if you need info from my system or if i can help with some info of some sort plsase feel free to ask. I am verry willing to help.

[ckruijntjens]

Okay, I will try to dig deeper into this thing. AES for ARM is also ongoing ;-)

Hi @mcmilk

Did you find anything? Do you need any help with log files or so?

Kind regards.

[ckruijntjens]

Okay, I will try to dig deeper into this thing. AES for ARM is also ongoing ;-)

Strangest thing here. normaly my system would create zfs errors (io) after a few days. always between 2 and 5 days. Now i set a --source-bwlimit=50M (syncoid) and now my system is not creating these errors now.

My system is now running for almost 7 days without zfs errors. I will keep track of it and inform you if the issue returns or not.

[ckruijntjens]

Okay, I will try to dig deeper into this thing. AES for ARM is also ongoing ;-)

Strangest thing here. normaly my system would create zfs errors (io) after a few days. always between 2 and 5 days. Now i set a --source-bwlimit=50M (syncoid) and now my system is not creating these errors now.

My system is now running for almost 7 days without zfs errors. I will keep track of it and inform you if the issue returns or not.

nevermind,

Now 7 days uptime and the errors are here again.

Time to reboot and scrub 2 times. i am reverting all to an unencrypted pool. Encryption is to buggy

[mcmilk]

@ckruijntjens - I am sorry, the time for me on OpenZFS is very short again.