Kernel 6.4 incompatibilty

[Moerten]

I allowed to open a new issue, due to you are ignoring existing ones since weeks. ZFS isn't compatible to actual linux kernels since months now.

Now I accidentally updated my Fedora 38 system getting kernel 6.4 installed. No problem I thought. Because your shitty ZFS broke my system dozen of times within the last years I found a solution to use "grubby" to boot into an older kernel. But because your ZFS is completely outdated there wasn't a 6.3 kernel installed anymore. And now I am stuck in a trapp. I couln't find a way to install an older kernel with apropriate kernel headers and thus it is impossible to access my data. It is also impossible to set up a newly installed system as there are existing kernel-headers 6.4 only. At least I know no solution to get a <= 6.3 kernel with headers installed.

This behaviour in completely inacceptable.

YOU ARE DESTROYING OTHER PEOPLE'S PROPERTY. So I will call you criminals.

And you do it again and again and again. It's a shame. And you even don't care for the problems you cause. I expect to be released a compatible ZFS version or a solution to access my data gain. ASAP!

Luckily I need new drives soon. I will migrate to btrfs then. Only have a look at the last two dozen issue requests. Your code base must be completely broken.

[andrewc12]

👍 Greetings from Australia

[outofforest]

YOU ARE CRIMINALS

I don't think so... Actually those people working hard on OpenZFS are the best people in the world. Delivering the best filesystem for free. They are also great educators, because from videos published on YouTube I've learnt a lot about the architecture of ZFS itself, learning a lot of great ideas I use in my other projects. So no, they are definitely not criminals :-)

Now I accidentally updated

Oh, so this is you who made a mistake, not OpenZFS team

... within the last years I found a solution ...

So this is your solution which doesn't work. OpenZFS clearly declares which kernels it is compatible to.

And now I am stuck in a trapp

The trap you created...

This behaviour in completely inacceptable.

I agree, the system you created to manage your OS is very shitty

YOU ARE DESTROYING OTHER PEOPLE'S PROPERTY

No, this is YOU who destroyed YOUR property

And you do it again and again and again.

Exactly, you do it again and again. And every time you do it, you share your frustration here.

I expect to be released a compatible ZFS version or a solution to access my data gain. ASAP!

I guess it's possible once you donate an appropriate sum to the team. Why anyone else should pay for fixing the mess you created?

Luckily I need new drives soon. I will migrate to btrfs then.

Good for you (and us)!

(maybe someone should warn the btrfs team)

[AllKind]

I'm not a Fedora user, but it took me one minute to figure out a solution:

Simple web search led to this: https://discussion.fedoraproject.org/t/how-do-i-install-an-old-kernel/76942 Which led to this: https://koji.fedoraproject.org/koji/buildinfo?buildID=2226968

Otherwise, not much to say other than which was already said here: https://github.com/openzfs/zfs/issues/15188

[outofforest]

@tonyhutter I could agree with one thing, not really mentioned by @Moerten. You could improve communication by sharing ETAs for the next version. I guess there is a good reason why 2.2.0 is still a release candidate and why 2.1.x is not upgraded to support new kernels. But at the same time it would be nice to simply tell us how the roadmap looks like.

[Moerten]

I found a more easy solution in the meanwhile to install the apropiate headers now. One can simply add to file "/ect/dnf/dnf.conf":

excludepkgs=kernel*-6.4*

But this takes all time to find out maybe installing the whole system from scratch multiple times. Now it installed the correct headers. I will check if ZFS can be installed later. But it is not acceptable that ZFS always brakes the system. There should be no update of the kernel allowed if it results in a broken system. What do you want with a system that cannot access its file system? Or it should explicitely be forced by --force to do so. And I don't know of a other software braking the system. It's only ZFS. So yes it's the fault of ZFS.

[rincebrain]

(I want to be clear that while it says "member", I speak for nobody but myself, I'm just marked that way so I can do random nonsense like add type tags on issues.)

If OpenZFS's packages on Fedora blocked upgrading the kernel, then there'd be even more issues objecting to that, and since RH sometimes backports wild things into their kernel trees, to say nothing of Linux upstream itself sometimes doing it, it still wouldn't stop things breaking.

My personal advice has been, for some time, that you should likely not run OpenZFS on a fast-moving distro that ships breaking kernel updates regularly, like Arch without using one of the special repos I believe exists to allow you to pin a kernel version that's known to work with the OpenZFS packages, or Fedora. There's only so much time in the day, for anyone, and anecdotally, though I haven't conducted a poll, a lot of the people paid to work on OpenZFS at their day jobs don't ship brand new kernels, so supporting the latest kernel is left to the available time that isn't on work priorities for them or people who aren't getting paid to do it.

Since I think an outright block on updating your kernel on Fedora or the like would be a nonstarter for multiple reasons, we could, conceivably, make it print a warning in the RPM packages on first install suggesting people consider pinning their kernel lest this happen (maybe? I know debs have a thing for this, but I don't recall if I've ever seen an RPM do it...), but I think that's as far as we can go - given that the kernel is a critical package, I think it'd just eject the ZFS packages unless you held them no matter how we marked them otherwise, and probably scream murder if it wanted to upgrade the kernel but that stopped it without an explicit hold on the kernel (and, as I said above, I don't think that would even prevent all or possibly even most breakages, with how much Linux upstream seems to like breaking changes).

So, I'm not sure what to suggest here. You're free to contribute or work on other people contributing faster kernel update support, though the other tradeoff with that being faster is that you'll more often encounter surprises with cases that might not have been discovered in testing (for example, right now, there's one on 6.3+ with native encryption and the memory rework that panics the kernel if it tries to compact RAM), or if there's some panacea I've overlooked because I'm as fallible as the next person, that'd be great too.

My final remark would be that we do, in fact, have a code of conduct around here, so while I understand you're quite frustrated, please try to be mindful of others in your replies.

[dmdx86]

Ubuntu has official ZFS support in their distribution, it just frozen at a particular version (I think it’s 2.1.9 in 22.04 LTS but don’t quote me on that). I would recommend using Ubuntu if you’re not prepared to deal with issues like the one you are dealing with.

Ubuntu also has a nice LiveCD. Assuming you don’t have any ZFS features enabled that break support with that ZFS version, you ought to be able to access your pool using the Ubuntu LiveCD.

The problem OpenZFS suffers from is that it is an out-of-tree filesystem and the Linux kernel is a fast-moving target, stuff breaks in new kernels all the time unfortunately. I would highly suggest switching to a distro that has an LTS kernel. I’m not a Fedora user. I would suggest either Ubuntu or RHEL / RHEL clone with an LTS kernel.

Just my $.02, I don’t speak for the project, just a user of ZFS for many years.

[AllKind]

It's a one-liner to configure your system to only update the kernel to a version compatible with the latest zfs version.

If that is too much of a demand, then free software is not for you. Install Oracle Solaris and pay for a support contract. Then you have more right to blame them and bitch at them all day. I'm sure such a big company will take good care of a little ordinary user!

I find this attitude rigorously annoying. It is free software, you didn't pay a dime for it. You don't have the right to demand anything! All you can do is be thankful and ask, how can I help. Where else do you get something for free? From Google, Meta, etc.. yes, but you pay with your personal data and they spam your head with ads. So many open source projects stop their development, because although their put in their lifetime for free, they rarely get a thank you (when everything is working). On the contrary they receive a lot of demands and bitching if things are not working as people expect.

[tomchiverton]

Since I think an outright block on updating your kernel on Fedora or the like would be a nonstarter for multiple reasons, we could, conceivably, make it print a warning in the RPM packages on first install suggesting people consider pinning their kernel lest this happen

I've also suggested in https://github.com/openzfs/openzfs-docs/pull/453 that the Fedora-specific documentation calls out this issue, and provides an optional step to prevent it occurring, as well as setting out the consequences.

[tonyhutter]

@Moerten can you post the error you were seeing? I'm able to install zfs-2.1.12 packages on F38 with the 6.4.14 kernel:

[hutter@fedora38 ~]$ cat /etc/fedora-release 
Fedora release 38 (Thirty Eight)

[hutter@fedora38 ~]$ sudo modprobe zfs

[hutter@fedora38 ~]$ uname -a
Linux fedora38 6.4.14-200.fc38.x86_64 #1 SMP PREEMPT_DYNAMIC Sat Sep  2 16:36:06 UTC 2023 x86_64 GNU/Linux

[hutter@fedora38 ~]$ rpm -qa | grep zfs
zfs-release-2-3.fc38.noarch
libzfs5-2.1.12-1.fc38.x86_64
zfs-dkms-2.1.12-1.fc38.noarch
zfs-2.1.12-1.fc38.x86_64

[hutter@fedora38 ~]$ lsmod | grep zfs
zfs                  4513792  0
zunicode              335872  1 zfs
zzstd                 606208  1 zfs
zlua                  229376  1 zfs
zavl                   16384  1 zfs
icp                   368640  1 zfs
zcommon               118784  2 zfs,icp
znvpair               126976  2 zfs,zcommon
spl                   139264  6 zfs,icp,zzstd,znvpair,zcommon,zavl

[tomchiverton]

How did you manage that ?

My current FC38 will not update with out removing zfs

$ uname -a
Linux bookcase 6.4.10-200.fc38.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Aug 11 12:20:29 UTC 2023 x86_64 GNU/Linux
# rpm -qa | grep zfs
zfs-release-2-3.fc38.noarch
libzfs5-2.1.12-1.fc38.x86_64
zfs-dkms-2.1.12-1.fc38.noarch
zfs-2.1.12-1.fc38.x86_64
# rpm -qa|grep kern|grep 6.4
kernel-tools-libs-6.4.4-200.fc38.x86_64
kernel-tools-6.4.4-200.fc38.x86_64
kernel-headers-6.4.4-200.fc38.x86_64
kernel-modules-core-6.4.7-200.fc38.x86_64
kernel-core-6.4.7-200.fc38.x86_64
kernel-modules-6.4.7-200.fc38.x86_64
kernel-6.4.7-200.fc38.x86_64
kernel-modules-core-6.4.9-200.fc38.x86_64
kernel-core-6.4.9-200.fc38.x86_64
kernel-modules-6.4.9-200.fc38.x86_64
kernel-devel-6.4.9-200.fc38.x86_64
kernel-6.4.9-200.fc38.x86_64
kernel-modules-core-6.4.10-200.fc38.x86_64
kernel-core-6.4.10-200.fc38.x86_64
kernel-modules-6.4.10-200.fc38.x86_64
kernel-devel-6.4.10-200.fc38.x86_64
kernel-devel-matched-6.4.10-200.fc38.x86_64
kernel-6.4.10-200.fc38.x86_64
# yum update
Last metadata expiration check: 4:12:17 ago on Tue 12 Sep 2023 07:44:53 BST.
Error: 
 Problem: The operation would result in removing the following protected packages: zfs

(I've done echo zfs > /etc/dnf/protected.d/zfs.conf to prevent kernel upgrades removing zfs)

[tomchiverton]

And more specifically :

[root@bookcase ~]# mv /etc/dnf/protected.d/zfs.conf /tmp/
[root@bookcase ~]# yum update
Last metadata expiration check: 0:03:39 ago on Tue 12 Sep 2023 12:11:06 BST.
Dependencies resolved.
=============================================================================================================
 Package                                        Arch        Version                 Repository          Size
=============================================================================================================
Installing:
 kernel                                         x86_64      6.4.14-200.fc38         updates            141 k
 kernel-core                                    x86_64      6.4.14-200.fc38         updates             16 M
 kernel-devel                                   x86_64      6.4.14-200.fc38         updates             19 M
 kernel-modules                                 x86_64      6.4.14-200.fc38         updates             56 M
 kernel-modules-core                            x86_64      6.4.14-200.fc38         updates             31 M
Upgrading:
 kernel-devel-matched                           x86_64      6.4.14-200.fc38         updates            141 k
 libheif                                        x86_64      1.16.2-2.fc38           updates            298 k
 mesa-dri-drivers                               x86_64      23.1.7-1.fc38           updates             19 M
 mesa-filesystem                                x86_64      23.1.7-1.fc38           updates             18 k
 mesa-libEGL                                    x86_64      23.1.7-1.fc38           updates            131 k
 mesa-libGL                                     x86_64      23.1.7-1.fc38           updates            173 k
 mesa-libgbm                                    x86_64      23.1.7-1.fc38           updates             45 k
 mesa-libglapi                                  x86_64      23.1.7-1.fc38           updates             54 k
 mesa-va-drivers                                x86_64      23.1.7-1.fc38           updates            3.4 M
 mesa-vulkan-drivers                            x86_64      23.1.7-1.fc38           updates            9.7 M
Removing:
 kernel                                         x86_64      6.4.7-200.fc38          @updates             0  
 kernel-core                                    x86_64      6.4.7-200.fc38          @updates            65 M
 kernel-devel                                   x86_64      6.2.9-300.fc38          @fedora             69 M
 kernel-modules                                 x86_64      6.4.7-200.fc38          @updates            54 M
 kernel-modules-core                            x86_64      6.4.7-200.fc38          @updates            30 M
Removing dependent packages:
 kmod-xtables-addons-6.4.7-200.fc38.x86_64      x86_64      3.24-1.fc38             @@commandline       70 k
 zfs                                            x86_64      2.1.12-1.fc38           @zfs               1.7 M
 zfs-dkms                                       noarch      2.1.12-1.fc38           @zfs                63 M

Transaction Summary
=============================================================================================================
Install   5 Packages
Upgrade  10 Packages
Remove    8 Packages

Total download size: 154 M
Is this ok [y/N]: N
Operation aborted.

[tonyhutter]

How did you manage that ?

My current FC38 will not update with out removing zfs

My setup was:

1. Vanilla F38 VM
2. dnf update
3. install zfs packages

I just wanted to confirm that zfs 2.1.12 could build against the F38 6.4.14 kernel, since https://github.com/openzfs/zfs/issues/15258#issue-1889891496 was indicating that it wasn't working.

[tomchiverton]

This is a physical machine, so maybe the kernel packages differ ?

Apparently ZFS 2.2 is nearly ready, which should hopefully resolve ? https://www.theregister.com/2023/08/16/openzfs_zfsbootmenu_2_2/

[oxyhyxo]

And more specifically :

[root@bookcase ~]# mv /etc/dnf/protected.d/zfs.conf /tmp/
[root@bookcase ~]# yum update
Last metadata expiration check: 0:03:39 ago on Tue 12 Sep 2023 12:11:06 BST.
Dependencies resolved.
=============================================================================================================
 Package                                        Arch        Version                 Repository          Size
=============================================================================================================
Installing:
 kernel                                         x86_64      6.4.14-200.fc38         updates            141 k
 kernel-core                                    x86_64      6.4.14-200.fc38         updates             16 M
 kernel-devel                                   x86_64      6.4.14-200.fc38         updates             19 M
 kernel-modules                                 x86_64      6.4.14-200.fc38         updates             56 M
 kernel-modules-core                            x86_64      6.4.14-200.fc38         updates             31 M
Upgrading:
 kernel-devel-matched                           x86_64      6.4.14-200.fc38         updates            141 k
 libheif                                        x86_64      1.16.2-2.fc38           updates            298 k
 mesa-dri-drivers                               x86_64      23.1.7-1.fc38           updates             19 M
 mesa-filesystem                                x86_64      23.1.7-1.fc38           updates             18 k
 mesa-libEGL                                    x86_64      23.1.7-1.fc38           updates            131 k
 mesa-libGL                                     x86_64      23.1.7-1.fc38           updates            173 k
 mesa-libgbm                                    x86_64      23.1.7-1.fc38           updates             45 k
 mesa-libglapi                                  x86_64      23.1.7-1.fc38           updates             54 k
 mesa-va-drivers                                x86_64      23.1.7-1.fc38           updates            3.4 M
 mesa-vulkan-drivers                            x86_64      23.1.7-1.fc38           updates            9.7 M
Removing:
 kernel                                         x86_64      6.4.7-200.fc38          @updates             0  
 kernel-core                                    x86_64      6.4.7-200.fc38          @updates            65 M
 kernel-devel                                   x86_64      6.2.9-300.fc38          @fedora             69 M
 kernel-modules                                 x86_64      6.4.7-200.fc38          @updates            54 M
 kernel-modules-core                            x86_64      6.4.7-200.fc38          @updates            30 M
Removing dependent packages:
 kmod-xtables-addons-6.4.7-200.fc38.x86_64      x86_64      3.24-1.fc38             @@commandline       70 k
 zfs                                            x86_64      2.1.12-1.fc38           @zfs               1.7 M
 zfs-dkms                                       noarch      2.1.12-1.fc38           @zfs                63 M

Transaction Summary
=============================================================================================================
Install   5 Packages
Upgrade  10 Packages
Remove    8 Packages

Total download size: 154 M
Is this ok [y/N]: N
Operation aborted.

It seems related to the dependency on kernel-devel 6.2.9-300.fc38.

Booting into the new kernel and reinstalling zfs and zfs-dkms packages reinstalls kernel-devel 6.2.9-300.fc38 but DKMS appears to happily build for the running kernel:

user@box ~> modinfo zfs 
filename:       /lib/modules/6.4.14-200.fc38.x86_64/extra/zfs.ko.xz
version:        2.1.12-1
license:        CDDL
author:         OpenZFS
description:    ZFS
alias:          devname:zfs
alias:          char-major-10-249
rhelversion:    9.99
srcversion:     0A38415F172773CFCCA60EC
depends:        znvpair,spl,zlua,icp,zcommon,zzstd,zunicode,zavl
retpoline:      Y
name:           zfs
vermagic:       6.4.14-200.fc38.x86_64 SMP preempt mod_unload 
sig_id:         PKCS#7
signer:         DKMS module signing key

[tomchiverton]

I don't fully understand.

So dnf remove kernel-devel-6.2.9-300.fc38.x86_64 remove zfs, because zfs requires devel 6.2.x, even on a 6.4.x kernel ? That's broken on the zfs dependencies side ?

[oxyhyxo]

I know almost nothing of the dark arts of rpm package management, however a quick look shows

user@box ~> dnf repoquery --requires zfs-dkms
/bin/sh
diffutils
dkms >= 2.2.0.3
gcc
kernel-devel <= 6.2.999
kernel-devel <= 6.3.999
kernel-devel >= 3.10
make
perl

at a guess the zfs package pulls in zfs-dkms as a dependency which itself then pulls in kernel-devel <= 6.2.99 as a dependency. A kernel update appears to update to the latest latest kernel-devel package (6.4.15-200.fc38) and uninstall the oldest kernel-devel (6.2.9-300.fc38) - which is required for zfs-dkms, which in turn is required for zfs.

I dont know what the impact of building the kernel modules against a later kernel-devel version is, and do not personally use Fedora in production (I use it for testing different softwares compatibility against more recent library/kernel versions and dont care if the box blows up).

[tonyhutter]

Booting into the new kernel and reinstalling zfs and zfs-dkms packages reinstalls kernel-devel 6.2.9-300.fc38 but DKMS appears to happily build for the running kernel:

Ok, so maybe there's a bug/feature that zfs-dkms just always builds against the current kernel no matter what the Requires:in the spec file says. That would explain the behavior we've all been seeing.

Note that zfs-2.1.13 with 6.5 kernel support is currently going though review (https://github.com/openzfs/zfs/pull/15268).

[RodoMa92]

Booting into the new kernel and reinstalling zfs and zfs-dkms packages reinstalls kernel-devel 6.2.9-300.fc38 but DKMS appears to happily build for the running kernel:

Ok, so maybe there's a bug/feature that zfs-dkms just always builds against the current kernel no matter what the Requires:in the spec file says. That would explain the behavior we've all been seeing.

Note that zfs-2.1.13 with 6.5 kernel support is currently going though review (#15268).

Yeah, but you should warn people that encryption is broken on >6.3 kernel, seems that another user lost their dataset while using on 6.5.2 here #15275

I wanted to test if it's gotten better on recent commits, but I'm not gonna risk it now unfortunately. My root pool is encrypted.

[AllKind]

Yeah, but you should warn people that encryption is broken on >6.3 kernel, seems that another user lost their dataset while using on 6.5.2 here https://github.com/openzfs/zfs/issues/15275

That user is using HEAD of master, which is version 2.2.x. There a lot of huge changes have been introduced. 2.1.13 is mainly fixes. Besides rincebrain recommended not to use kernel version >6.3 because of https://github.com/openzfs/zfs/issues/15140. Besides encryption has too many issues in general for a much longer time.

[RodoMa92]

Yeah, but you should warn people that encryption is broken on >6.3 kernel, seems that another user lost their dataset while using on 6.5.2 here https://github.com/openzfs/zfs/issues/15275

That user is using HEAD of master, which is version 2.2.x. There a lot of huge changes have been introduced. 2.1.13 is mainly fixes. Besides rincebrain recommended not to use kernel version >6.3 because of https://github.com/openzfs/zfs/issues/15140. Besides encryption has too many issues in general for a much longer time.

I didn't know that encryption had issues even on older kernel, I'll probably turn it off then at this point.

My two cents on this is that it should be at least mentioned in the release notes on the stable version, IMHO, since it claims 5.3 compat.

(Yeah, #15140 is an issue opened by me :))

[AllKind]

I didn't know that encryption had issues even on older kernel

Unfortunately quite a bunch of problems with encryption: https://github.com/openzfs/zfs/issues?q=is%3Aissue+is%3Aopen+label%3A%22Component%3A+Encryption%22+label%3A%22Type%3A+Defect%22

More unfortunately it doesn't look like they get addressed. Many users requested a warning in the documentation, but that hasn't happened. So people who don't follow issue trackers and such, are not aware :-( Sorry for getting off topic.

[outofforest]

Oh, that's not funny. I guess I'm not the only one storing the most important data on encrypted dataset

[admnd]

This behaviour in completely inacceptable.

If you have lost some data that means: no backup of it was done.

To quote your own word: not having backup of critical data is unacceptable. Nothing in this world is perfect and people around here are at hard work on various aspects (fixing bugs, testing, documenting and so on) and you get their work for free. Even commercial software agreements (unless you deal with very special circumstances) write this in capitals: software is provided as-is with no warranty and the vendor declines any kind of responsibility (damage, profit lost, ...).

Look, things can and will fail at some point, sometimes even for the dumbest possible reasons. If you are not sure about how things will end, testing in a breakable environment is far from being forbidden. At the end, you are responsible of the usage of a piece of software, not the community who works on a best effort principle. If you had taken the time to go through a couple a bugs reports, you would have seen that ZFS encryption has some issues then make your opinion to use it or not. Great news for you: like many things in life, you are not entitled to anything here. Even paying for a support contract would not mean seeing an issue being automagically fixed and getting your data back.

If what you are telling us is the truth and not something emphasized under frustration / anger, loosing data several times and not being able to take and restore a backup of it and not testing stuff before putting in production is a serious issue. If you do not learn the lesson and loose your data over and over again without taking counter-measures for that... well... too bad to sad. I can not even sympathize with you, to speak for myself. Assuming responsibilities is one the basic attitudes. Read: your data, your responsibility, your problem. Full-stop.

Next time, open a bug report and work in collaboration with the community rather than whine pointlessly on it. This will be more constructive and useful.

Being respectful and thankful for the work done by the community are not optional.

[tomchiverton]

I don't believe there are extensive encryption issues. A quick read shows either very old (>3y), not using a ZFS release (eg -git), doing unusual things (shared folder with Windows) and so forth

[RodoMa92]

I don't believe there are extensive encryption issues. A quick read shows either very old (>3y), not using a ZFS release (eg -git), doing unusual things (shared folder with Windows) and so forth

Well, the stable release claim 6.3, but it's kinda broken, see above with hugepages issues. Haven't looked at all the rest though. It's not as bad as others said, though, IMO.

[rincebrain]

I don't believe there are extensive encryption issues. A quick read shows either very old (>3y), not using a ZFS release (eg -git), doing unusual things (shared folder with Windows) and so forth

Bugs are still bugs if they never get fixed.

Several of the bugs predate OpenZFS merging native encryption into mainline git, and still exist.

[MasterCATZ]

Luckily I need new drives soon. I will migrate to btrfs then. Only have a look at the last two dozen issue requests. Your code base must be completely broken.

sadly "btfs" breaks even if its untouched I have lost of data because the tree's just randomly become corrupt I have "btrfs' on 120 drives and I am constantly going through trying to do data recovery across all off them the fs will just randomly break disks check out fine (single disks using mergerfs with snapper and snapraid) does not seem to matter if its table is gpt or btrfs using the entire disk , but unless gpt is used most data recovery programs will not work

zfs at least it only breaks when kernel updates are done :D but downside being if I was using zfs once the raid breaks I loose 100% everything (had that happen before using raidz2 now using raidz3) using btrfs I just loose the data on the failed disks

I just wish zfs would unmount correctly when I do a suspend during power failures every time I resume zfs goes I/O error mode and locks it so I have no choice but to reboot anyhow

[snajpa]

can this be closed? seems like 6.4 compat has been solved for a while, if there's any lingering problems which haven't been reported, can you please report them in a new issue?