search

openzfs / zfs

OpenZFS on Linux and FreeBSD
https://openzfs.github.io/openzfs-docs
Other
10.69k stars 1.76k forks source link

zfs-2.2.7 patchset #16720

Open tonyhutter opened 3 weeks ago

tonyhutter commented 3 weeks ago

Motivation and Context

Description

Proposed zfs-2.2.7 patchset. Support 6.11 kernel and github QEMU runners.

How Has This Been Tested?

Runners will test

Types of changes

Checklist:

robn commented 3 weeks ago

@tonyhutter I would like to include:

If you agree, I've pulled these in and fixed conflicts: https://github.com/robn/zfs/tree/zfs-2.2.7-staging

AllKind commented 3 weeks ago

@tonyhutter Fixing native-deb packages and the release tar ball was promised for 2.2.7, so I'm proposing these:

Fix missing packaging files from release tarballs c204c3f340104f39c873597a6ee110259d46ecff Disable parallel build for native-deb targets cc9e36a42e60145c3c571290d482b03005ae9f79 Update path for zed in zfs-zed.service for native debian packages 45addf760532a346d36b759801b2fda427f7be4a Only serialize native-deb targets 65a94ffa80466246f37322393c3b5bd460cb8987

Going back the commit list until 5th of September there are a lot of fixes. I did not test if they all would apply cleanly, also hard for me to tell their relevance for some, but IMHO 2.2.x should be stable, which makes fixes sound obligatory. I'll just put up the list here as an attempt to help. If I'm being too bold, just ignore it...

Verify parent_dev before calling udev_device_get_sysattr_value c8aed9f97329bb328ea4fe677defdcb20d703bb3 On the first vdev open ignore impossible ashift hints 6187b194349c5a728c9df8c6842f1866d5c9782a Fix gcc uninitialized warning in FreeBSD zio_crypt.c 2bf15202110767b6adbacc02d7e9dae1c305d944 Fix gcc unused value warning in FreeBSD simd.h c480e06d88e07fe5b8aaa920cc966498bb321a8d Fix dependency install on Debian 11 2e4e0928220f1a67d0412f8cbcc4b426ddfb58f2 Workaround issue of Linux vdev_disk.c aefc2da8a594d7a8059c862eab464d5f798393b3 (robns root cause fix should go into 2.3 - maybe this release (see above), but if not this will help many folks on LUKS meanwhile) Fix compile-time warnings caused by duplicate struct typedefs a9851ea3dd6af6f789e221f2ccd7ad43561eff81 Fix inconsistent mount options for ZFS root 27e8f5610262177567b9aaebc6c9448d783aadd7 Fallback to strerror() when strerror_l() isn't available 48dfe3974710aa9807980852aa5f46e1c311c579 FreeBSD: ignore some includes when not building kernel efeb60b86a22b4d58c9eaf73c862bbe0cd3a7fc3 Return boolean_t in inline functions of lib/libspl/include/sys/uio.h ab777f436ce228559f3c2480ebe8a6b5ab888c43 contrib/debian: add new manpages to installation list c59d5495fe171aac752ab5bf8b7be325a2ef861d Fix generation of kernel uevents for snapshot rename on linux 0b4dcbe5b425be70bc10d836d8e66711c690e8d4 ARC: Cache arc_c value during arc_evict() 4ebe674d91fa942751a51fa55fda5ebe7db6165e Avoid computing strlen() inside loops 86737c5927c204dc866f1de7c95ab964e98be6af man: update recordsize max size info 559150529909badcac80e72317067ea92aa4ef27 Properly release key in spa_keystore_dsl_key_hold_dd() 48d1be254fa3ec9dd9e998921fbc6939fe1d599e FreeBSD: Sync taskq_cancel_id() returns with Linux 832f66b218d21407a779a567dc4d984b4b861f92 Add missing guard defines for simd_stat ccc420acd5452ca6fffe3797ea7da5a5f1a64d2b Evicting too many bytes from MFU metadata d40d40913d33eb2d50cf363fa9f5022fc794cc9f Reduce and handle EAGAIN errors on AIO label reads 3014dcb76266c404ee97b8704f22996b56ba8888 Add compatibility file for GRUB versions up to v2.06 2aafc2ea1f887458fe84cf4e8042fb5891e176bf Fix handling of DNS names with '-' in them for sharenfs 29c9e6c3241f2508ad997e78eb82b00335822fcf sa_impl: fix SA header bitfield docs ec0209418fa613e0ee306daf91a9f7208aee6e10 Fix an uninitialized data access (#16511) 308f7c2f14deaa49b7b0d223564f48ccb6f5937b

https://github.com/openzfs/zfs/pull/16723 would be great to have, if not b052035990594408899fa32fd4ad6603b75b6c6d would at least cover the issue partly.

tonyhutter commented 3 weeks ago

@robn @AllKind thanks for the suggestions, I'll try to pull in as many as I can.

tonyhutter commented 3 weeks ago

I got everything in with the exception of these commits, which were either already merged or were related to a feature not present in 2.2.x: 2e4e09282 Fix dependency install on Debian 11 (#16683) a9851ea3d Fix compile-time warnings caused by duplicate struct typedefs efeb60b86 FreeBSD: ignore some includes when not building kernel ab777f436 Return boolean_t in inline functions of lib/libspl/include/sys/uio.h c59d5495f contrib/debian: add new manpages to installation list

@robn I eneded up "reverting the revert" (https://github.com/openzfs/zfs/pull/16720/commits/be0980f4552541612f8df8a929c76096dabb2237) to pull in https://github.com/openzfs/zfs/pull/16720/commits/ee40c21bd4a7f788de4bbe3b30805ddc731c804d

robn commented 3 weeks ago

@tonyhutter you don't need ee40c21 (orig aefc2da8a); it was only there until the proper fix went in, and was reverted in 3c650bec1.

You do need e7425ae62 to go with c25bf7a (orig 63bafe60e), or it'll corrupt memory pretty hard.

I'm not sure about be0980f; istr it made things worse for the classic IO submission (#15533), and since that's the default for 2.2 it seems likely that it's gonna immediately blow up again?

tonyhutter commented 3 weeks ago

@robn thanks, I just did the following in my latest push:

Remove: ee40c21 Workaround issue of Linux vdev_disk.c, (#16678) be0980f Reapply "Tune zio buffer caches and their alignments"

Add: e7425ae6248db21240049df859630d5044dcd58a vdev_disk: move abd return and free off the interrupt handler

Can you double check that I have it correct now?

robn commented 3 weeks ago

@tonyhutter looks right to the naked eye, I'm putting it through some stress tests now.

shodanshok commented 2 weeks ago

Can I ask about https://github.com/openzfs/zfs/pull/15457 and https://github.com/openzfs/zfs/pull/15451 ? Thanks.

AllKind commented 2 weeks ago

As far as I can tell "Read prefetched buffers from L2ARC https://github.com/openzfs/zfs/pull/15451" made it into 2.2.1. But "Increase L2ARC write rate and headroom" 887a3c533b94a4b70075e310f15c45b9dee19410 didn't make it into a 2.2 release.

Harry-Chen commented 2 weeks ago

It doesn't build on freebsd now:

  --- zfs_prop.o ---
  /usr/home/zfs/zfs/module/zcommon/zfs_prop.c:1065:28: error: too many arguments provided to function-like macro invocation
   1065 | extern void simd_stat_init(void);
        |                            ^
  /usr/home/zfs/zfs/include/os/freebsd/spl/sys/simd.h:53:9: note: macro 'simd_stat_init' defined here
     53 | #define simd_stat_init()        do {} while (0)
        |         ^
  /usr/home/zfs/zfs/module/zcommon/zfs_prop.c:1066:28: error: too many arguments provided to function-like macro invocation
   1066 | extern void simd_stat_fini(void);
        |                            ^
  /usr/home/zfs/zfs/include/os/freebsd/spl/sys/simd.h:54:9: note: macro 'simd_stat_fini' defined here
     54 | #define simd_stat_fini()        do {} while (0)
        |         ^
  2 errors generated.

Looks like we need https://github.com/openzfs/zfs/commit/f9d4f1b480beaaa67373a9cdff6652cff255fea8 to fix.

tonyhutter commented 2 weeks ago

@shodanshok @AllKind @Harry-Chen I will pull those in. Right now I'm trying to get the 6.12 kernel patches pulled in, but there's a lot of dependencies to resolve (like https://github.com/openzfs/zfs/pull/16479)

robn commented 2 weeks ago

@tonyhutter do you want a hand?

tonyhutter commented 2 weeks ago

@robn my latest push includes all the dependencies and builds locally. We'll see what the test suite says...

amotin commented 2 weeks ago

@tonyhutter Since you seem to be not on Slack let me repeat here: In context of https://github.com/openzfs/zfs/pull/16720#issuecomment-2460466442 could you check that we don't have "Tune zio buffer caches and their alignments" there unless we disable the "classic" Linux vdev code? I may be wrong, but seems like we may get broken combination. Rob asked to not include Reapply "Tune zio buffer caches and their alignments", but I don't see the revert commit here either.

robn commented 2 weeks ago

@tonyhutter guess it depends if your intent was to bump Linux min to 4.18 on this release? I was guessing not, but I don't know the rules for stable series. My offer was more, if you want 6.12 support, do you want me to work out a smaller patch set that also doesn't remove and reorganise a bunch of stuff?

I'll re-review my other stuff here too, mostly those userspace changes. I had not wanted them on 2.2.x mostly out of abundance of caution. istr there were some followup fixes too, I'll see what shakes out.

tonyhutter commented 2 weeks ago

@amotin thanks for catching that - I unwittingly pulled 847c0fc35 in to resolve a merge conflict, but you're right it shouldn't be in there. I'll take it out.

guess it depends if your intent was to bump Linux min to 4.18 on this release? I was guessing not, but I don't know the rules for stable series.

@robn I think it makes sense to pull in your big commit list and deprecate support for kernels < 4.18 on the 2.2.x branch. We're not doing any automated testing against kernels < 4.18, so we can't really claim to support those older kernels. Also, there's been so much churn with the deprecation commits, that if we don't pull them in now, we're just going to be fighting merge conflicts on future 2.2.x releases. Sure, 2.3.x is coming soon, but we historically still do a few releases for the older branch even after the new branch is released.

mabod commented 1 week ago

I have this PR running since 3 days on kernel 6.6.x on Endeavouros without any issue.

tonyhutter commented 1 week ago

Looks like module load isn't working on FreeBSD:

Test: /usr/local/share/zfs/zfs-tests/tests/functional/acl/off/setup (run as root) [00:00] [FAIL]
19:28:11.98 SUCCESS: add_group zfsgrp
19:28:12.00 SUCCESS: add_user zfsgrp staff1
19:28:12.02 SUCCESS: add_user zfsgrp staff2
19:28:12.02 NOTE: begin default_setup_noexit
19:28:12.14 Failed to load openzfs module: Exec format error
19:28:12.14 ERROR: zpool create -f testpool md0 exited 1
...

19:35:34.34 linker_load_file: /boot/modules/openzfs.ko - unsupported file type
19:35:34.34 link_elf_obj: symbol zfs_znode_update_vfs undefined
19:35:34.34 linker_load_file: /boot/modules/openzfs.ko - unsupported file type
19:35:34.34 link_elf_obj: symbol zfs_znode_update_vfs undefined
19:35:34.34 linker_load_file: /boot/modules/openzfs.ko - unsupported file type
19:35:34.34 link_elf_obj: symbol zfs_znode_update_vfs undefined
19:35:34.34 linker_load_file: /boot/modules/openzfs.ko - unsupported file type

I'm looking into it

robn commented 1 week ago

@tonyhutter You want 80645d658.

tonyhutter commented 1 week ago

@robn thanks, I included it in my latest push.

mabod commented 4 days ago

Do you want to add 6.12 compatibility with PR https://github.com/openzfs/zfs/pull/16793 ?

AllKind commented 4 days ago

@tonyhutter I think it would be great to have 38c0324c0fb6b53286f00b7e36d1edca70d863d8 included, as a lot of 2.2 users experience the underlying issue.

amotin commented 2 days ago

We could include also cdd53fea1e899585f141fb2d4885e88df9d6939f, 94a03dd1e49a1bfa464944b624cd718076a8c0ee, 483087b06ff29567c19a8b9fd53662c11af8f478, 1ee251bdde882feabc429ceb41769a66dbd7f228.

tonyhutter commented 2 days ago

@mabod @AllKind @amotin thanks, I included those commits in my latest push.

AllKind commented 2 days ago

For full 6.11+ kernel compatibility I guess we'd also need https://github.com/openzfs/zfs/pull/16805 - once it's merged....