Open FL140 opened 12 months ago
FL140
commented
12 months ago Small Update: The same error occurs for v2.2.0. In addition the file https://github.com/openzfs/zfs/releases/download/zfs-2.2.0/zfs-2.2.0.sha256.asc does not exist (is missing?).
rincebrain
commented
12 months ago @tonyhutter seems to have not yet read the bug reports since 2.2.0 came out that his process for generating release tarballs is producing things that will never work on FreeBSD and fail to build the debs on Linux.
Cloning the git repo it should work better.
luckylinux
commented
12 months ago I managed to compile 2.2.1 a few days ago on one system (Tarball downloaded yesterday 2023-11-26 at 06h27 GMT).
Now I am running into the same issue as @FL140 with make native-deb-utils.
Tested on two separate Rock 5B SBC (same Software Versions though). They both fail today.
Attached is the buildinfo of the successfull build of yesterday, in case it could help somebody. Arch is AARCH64/ARM64 on Rock 5B SBC.
openzfs-linux_2.2.1-1_arm64.buildinfo.txt
I am not sure if in the meantime some of my toolchain/system tools got updated though. If you want a diff / apt show packages etc I can provide that as well. Just let me know.
luckylinux
commented
12 months ago That being said the tarball from yesterday seemed to uncompress to zfs-zfs-2.2.1 (weird !), while the one from today seemed to uncompress to zfs-2.2.1.
Seems like the tarball on the Release page got changed ...
luckylinux
commented
12 months ago @FL140 : I just noticed when I installed yesterday I used wget https://github.com/openzfs/zfs/archive/refs/tags/zfs-$version.tar.gz -O zfs-$version.tar.gz
While today I was using wget https://github.com/openzfs/zfs/releases/download/zfs-$version/zfs-$version.tar.gz -O zfs-$version.tar.gz
Note that today still the first uncompresses to zfs-2.2.1 while the second uncompresses to zfs-zfs-2.2.1.
rincebrain
commented
12 months ago https://github.com/openzfs/zfs/releases/download/zfs-2.2.1/zfs-2.2.1.tar.gz seems to untar to zfs-2.2.1/*, for me, while the first one produces zfs-zfs-2.2.1, I assume because it's projectname-tagname dynamically generated.
luckylinux
commented
12 months ago And in one case you need to run ./autogen.sh while in the other you don't. But even if you run autogen.sh it seems it's not creating the necessary configuration for Debian. ./configure works, make does as well, but buildpkg fails.
luckylinux
commented
12 months ago In case somebody wants to replicate, I'm trying with this right now. You might not need to setup the requirements if you already have ...
#!/bin/bash
# Install Requirements
sudo apt --no-install-recommends install dkms
sudo apt install dh-dkms
sudo apt install aptitude libcurl4-openssl-dev libpam0g-dev lsb-release build-essential autoconf automake libtool libblkid-dev uuid-dev libudev-dev libssl-dev zlib1g-dev libaio-dev libattr1-dev libelf-dev python3 python3-dev python3-setuptools python3-cffi libffi-dev python3-packaging git libcurl4-openssl-dev debhelper-compat dh-python po-debconf python3-all-dev python3-sphinx
sudo apt install build-essential autoconf automake libtool gawk fakeroot libblkid-dev uuid-dev libudev-dev libssl-dev zlib1g-dev libaio-dev libattr1-dev libelf-dev python3 python3-dev python3-setuptools python3-cffi libffi-dev python3-packaging git libcurl4-openssl-dev debhelper-compat dh-python po-debconf python3-all-dev python3-sphinx
# Define Desired Version
version="2.2.1"
cd /usr/src
mkdir -p zfs
cd zfs
wget https://github.com/openzfs/zfs/archive/refs/tags/zfs-$version.tar.gz -O zfs-$version.tar.gz
#wget https://github.com/openzfs/zfs/releases/download/zfs-$version/zfs-$version.tar.gz -O zfs-$version.tar.gz
mkdir -p zfs-$version
#tar xvf zfs-$version.tar.gz
tar xvf zfs-$version.tar.gz -C zfs-$version --strip-components 1
cd zfs-$version
# Apply Patch in order to disable SIMD and Enable successfully ZFS Compile
wget https://raw.githubusercontent.com/chimera-linux/cports/master/main/zfs/patches/aarch64-disable-neon.patch -O aarch64-disable-neon.patch
patch -p1 < aarch64-disable-neon.patch
sh autogen.sh
./configure
make -s -j$(nproc)
make native-deb
make native-deb-utils native-deb-dkms
# Select Subset of Packages to prevent installation of default linux-image and linux-headers
cd ..
mkdir -p selected-packages
mkdir -p selected-packages/$version
cd selected-packages/$version/
mv ../../openzfs-libnvpair3_$version*.deb ./
mv ../../openzfs-libpam-zfs_$version*.deb ./
mv ../../openzfs-libuutil3_$version*.deb ./
mv ../../openzfs-libzfs4_$version*.deb ./
mv ../../openzfs-libzpool5_$version*.deb ./
mv ../../openzfs-zfs-dkms_$version*.deb ./
mv ../../openzfs-zfs-initramfs_$version*.deb ./
mv ../../openzfs-zfs-modules-*_$version*.deb ./
mv ../../openzfs-zfs-zed_$version*.deb ./
mv ../../openzfs-zfsutils_$version*.deb ./
sudo apt install --fix-missing ./*.debTrying to make both native deb packages at once breaks, would be my guess about what's breaking for you there.
luckylinux
commented
12 months ago Yeah keep in mind I was playing around a lot, so I kinda wrote them inside a script. But it doesn't meant it 100% works. It's more to remember what I did
native-deb-dkms I don't think it's valid anymore either
luckylinux
commented
12 months ago I managed to compile on one of the systems, the other one failed without very clear message.
On the system where it compiled successfully one warning was generated though
debconf: Unknown template field '_description', in stanza #1 of /var/lib/dpkg/info/openzfs-zfs-dkms.templates
debconf: Unknown template field '_description', in stanza #2 of /var/lib/dpkg/info/openzfs-zfs-dkms.templates
debconf: Unknown template field '_description', in stanza #3 of /var/lib/dpkg/info/openzfs-zfs-dkms.templatesThe exact commands used were (just to make sure I didn't update anything since last time)
#!/bin/bash
# Install Requirements
sudo apt --no-install-recommends install dkms
sudo apt install dh-dkms
sudo apt install aptitude libcurl4-openssl-dev libpam0g-dev lsb-release build-essential autoconf automake libtool libblkid-dev uuid-dev libudev-dev libssl-dev zlib1g-dev libaio-dev libattr1-dev libelf-dev python3 python3-dev python3-setuptools python3-cffi libffi-dev python3-packaging git libcurl4-openssl-dev debhelper-compat dh-python po-debconf python3-all-dev python3-sphinx
sudo apt install build-essential autoconf automake libtool gawk fakeroot libblkid-dev uuid-dev libudev-dev libssl-dev zlib1g-dev libaio-dev libattr1-dev libelf-dev python3 python3-dev python3-setuptools python3-cffi libffi-dev python3-packaging git libcurl4-openssl-dev debhelper-compat dh-python po-debconf python3-all-dev python3-sphinx
# Define Desired Version
version="2.2.1"
cd /usr/src
mkdir -p zfs
cd zfs
wget https://github.com/openzfs/zfs/archive/refs/tags/zfs-$version.tar.gz -O zfs-$version.tar.gz
#wget https://github.com/openzfs/zfs/releases/download/zfs-$version/zfs-$version.tar.gz -O zfs-$version.tar.gz
mkdir -p zfs-$version
#tar xvf zfs-$version.tar.gz
tar xvf zfs-$version.tar.gz -C zfs-$version --strip-components 1
cd zfs-$version
# Apply Patch in order to disable SIMD and Enable successfully ZFS Compile
wget https://raw.githubusercontent.com/chimera-linux/cports/master/main/zfs/patches/aarch64-disable-neon.patch -O aarch64-disable-neon.patch
patch -p1 < aarch64-disable-neon.patch
sh autogen.sh
./configure
make -s -j$(nproc)
make native-deb
make native-deb-utils native-deb-dkms
# Select Subset of Packages to prevent installation of default linux-image and linux-headers
cd ..
mkdir -p selected-packages
mkdir -p selected-packages/$version
cd selected-packages/$version/
mv ../../openzfs-libnvpair3_$version*.deb ./
mv ../../openzfs-libpam-zfs_$version*.deb ./
mv ../../openzfs-libuutil3_$version*.deb ./
mv ../../openzfs-libzfs4_$version*.deb ./
mv ../../openzfs-libzpool5_$version*.deb ./
mv ../../openzfs-zfs-dkms_$version*.deb ./
mv ../../openzfs-zfs-initramfs_$version*.deb ./
#mv ../../openzfs-zfs-modules-*_$version*.deb ./
mv ../../openzfs-zfs-zed_$version*.deb ./
mv ../../openzfs-zfsutils_$version*.deb ./
sudo apt install --fix-missing ./*.debOn another note, is it possible to omit the "openzfs" prefix when building deb packages ? So that the name would be the same as the "standard" e.g. zfs-dkms package as in APT Repository ?
I had a brief look at autogen.sh, configure & Makefile files, but didn't see any obvious "prefix" directive.
rincebrain
commented
12 months ago The entire point is to make them separate so that you can clearly say "you cannot mix and match these" and not have to worry about people's version numbers overlapping.
luckylinux
commented
12 months ago So it's a "feature" not a "bug", alright, I guess I can live with that
FL140
commented
12 months ago After a bit of sleep and reinvestigating again it turns out the source of the problem is exact the same as in https://github.com/openzfs/zfs/issues/15404
But marking it as a pure duplicate is probably not the best thing as the the two scripts provided here will provide a way to compile the package for Ubuntu 22.10 (hopefully). I will merge the new information from the issue above and provide an update.
FL140
commented
12 months ago @luckylinux I will try to merge our scripts to make it more generic.
And I think we should switch over to the git clone approach instead of the release tar.gz since those are broken (at least for 2.2.0 and 2.2.1) and no one needs them in this state.
Just a small hint, you should prefere apt-get over apt in scripts as per recommendation.
luckylinux
commented
12 months ago @FL140 actually I thought that apt-get like apt-cache was getting deprecated in favor of simply apt.
I many script (besides this one) I actually use aptitude.
I just saw today that apt is not really recommended in scripts. Duh.
Also sometimes apt and aptitude yield different dependencoes resolution...
FL140
commented
12 months ago @FL140 actually I thought that apt-get like apt-cache was getting deprecated in favor of simply apt.
I many script (besides this one) I actually use aptitude.
I just saw today that apt is not really recommended in scripts. Duh.
Also sometimes apt and aptitude yield different dependencoes resolution...
I also got recently bite by that, that's why I wrote the comment here in the first place. I also prefer aptitude for daily work, but it is not the default package manager, apt (in the apt-get variant) is and therefor the reference to use in scripts over aptitude, apt and aptitude AKAIK don't work on the same database, I ran into this when I set holds for some packages with apt which got removed when running aptitude the next time. Apart from that problem the recommendation was to use apt-get (which uses the apt database AFAIK) because it "has a stable user interface made for scripting compared to apt".
FL140
commented
11 months ago The following script builds an Ubuntu package for 2.2.2 from tonyhutter and will be updated to the official branch after the official 2.2.2 release.
Note: currently the script exits prior the deb package installation because either dracut or initramfs is used, act according to your setup.
#!/bin/bash
# Ubuntu OpenZFS DKMS Kernel Module Install Script
typeset -r OPENZFS_VERSION=${1:-2.2.2}
typeset -r TMP_DIR="/tmp"
typeset -r BUILD_JOBS=$(( $(nproc) / 2 )) # use half of what we have for building
# Installed required packages (Debian, Ubuntu)
# Note:
# TBD: hmm, I don't think we need package(s) alien and related packages any longer for versions >= v2.2.
sudo apt-get install -y build-essential autoconf automake libtool gawk \
fakeroot libblkid-dev uuid-dev libudev-dev libssl-dev zlib1g-dev \
libaio-dev libattr1-dev libelf-dev linux-headers-generic \
python3 python3-dev python3-setuptools python3-cffi libffi-dev \
python3-packaging debhelper-compat dh-python po-debconf \
python3-all-dev python3-sphinx \
git libcurl4-openssl-dev libpam0g-dev \
dkms dh-dkms
# Create working directory
[ -d "${TMP_DIR}/zfs-${OPENZFS_VERSION}" ] && { rm -rf "${TMP_DIR}/zfs-${OPENZFS_VERSION}"; } # !! use a fixed sting in rm -rf!
install -d -m 0770 -o root -g root "${TMP_DIR}/zfs-${OPENZFS_VERSION}"
pushd "${TMP_DIR}/zfs-${OPENZFS_VERSION}" > /dev/null || { echo "Error while pushd directory \"${TMP_DIR}/zfs-${OPENZFS_VERSION}\"!"; exit 2; }
## Get Source Code (Released Tarball) # deprecated, .tar.gz release file is broken
## Note:
## The download location given in the instructions does not exist, get the tar package directly from GitHub.
##wget http://archive.zfsonlinux.org/downloads/zfsonlinux/zfs/zfs-${OPENZFS_VERSION}.tar.gz
#wget https://github.com/openzfs/zfs/releases/download/zfs-${OPENZFS_VERSION}/zfs-${OPENZFS_VERSION}.tar.gz
#wget https://github.com/openzfs/zfs/releases/download/zfs-${OPENZFS_VERSION}/zfs-${OPENZFS_VERSION}.sha256.asc
#sha256sum --check zfs-${OPENZFS_VERSION}.sha256.asc || { echo "Error checksum(s) for zfs-${OPENZFS_VERSION} does not match!"; exit 3; }
#gpg --verify zfs-${OPENZFS_VERSION}.sha256.asc zfs-${OPENZFS_VERSION}.tar.gz # TBD add error handling
#tar -xzf zfs-${OPENZFS_VERSION}.tar.gz
# Get Source Code (GitHub)
# TBD FIXME git clone --branch zfs-${OPENZFS_VERSION} https://github.com/openzfs/zfs zfs-${OPENZFS_VERSION} || { echo "Error cloning zfs-${OPENZFS_VERSION} from GitHub!"; exit 3; }
git clone --branch zfs-${OPENZFS_VERSION}-hutter https://github.com/tonyhutter/zfs zfs-${OPENZFS_VERSION} || { echo "Error cloning zfs-${OPENZFS_VERSION} from GitHub!"; exit 3; }
cd zfs-${OPENZFS_VERSION}
# Get Patches (optional)
# see: https://openzfs.github.io/openzfs-docs/Developer%20Resources/OpenZFS%20Patches.html#openzfs-patches
#
# The following and 5 other patches, but all of those are finally added to the
# zfs-2.2.2 branch of tonyhutter waiting to be merged into openzfs/zfs-2.2.2
# as of 01.12.2023
# => so we use tonyhutter/zfs-2.2.2 branch for now
#
# Notes:
# * If commented out, the following commits are already included into 2.2.2.
## Revert "Tune zio buffer caches and their alignments"
## https://github.com/openzfs/zfs/commit/89fcb8c6f969cbfa237849d9b33610124d361186
## fixes: CKSUM and WRITE errors with 2.2.1 stable, when vdevs are atop LUKS #15533
## https://github.com/openzfs/zfs/issues/15533
# git cherry-pick 89fcb8c6f969cbfa237849d9b33610124d361186
## dnode_is_dirty: check dnode and its data for dirtiness #15571
## https://github.com/openzfs/zfs/commit/9b9b09f452a469458451c221debfbab944e7f081
## fixes: some copied files are corrupted (chunks replaced by zeros) #15526
# git cherry-pick 9b9b09f452a469458451c221debfbab944e7f081
## brt: lift internal definitions into _impl header
## https://github.com/openzfs/zfs/commit/803a9c12c911b240f1bb20ac9b542349d49bea3b
#git cherry-pick 803a9c12c911b240f1bb20ac9b542349d49bea3b
## zdb: show BRT statistics and dump its contents
## https://github.com/openzfs/zfs/commit/213d6829673ce28095f7c15b8fb65e1ade91d20a
#git cherry-pick 213d6829673ce28095f7c15b8fb65e1ade91d20a
## dmu_buf_will_clone: fix race in transition back to NOFILL
## https://github.com/openzfs/zfs/commit/2a953e0ac928563166ad7caa89e48b8d257724d4
## closes: dmu_buf_will_clone: fix race in transition back to NOFILL #15566
## closes: some copied files are corrupted (chunks replaced by zeros) #15526
#git cherry-pick 2a953e0ac928563166ad7caa89e48b8d257724d4
## openzfs/zfs: zfs-dkms: fix shell-init error message
## https://github.com/openzfs/zfs/commit/95b68eb693ba5b4785b260d592d8b5094f4c9576
#git cherry-pick 95b68eb693ba5b4785b260d592d8b5094f4c9576
## ZTS: Fix zfs_load-key failures on F39
## https://github.com/openzfs/zfs/commit/3551a32e5ed6c79cad192f66f5faf3d4aae8c682
#git cherry-pick 3551a32e5ed6c79cad192f66f5faf3d4aae8c682
# Build OpenZFS native deb packages
# see: https://openzfs.github.io/openzfs-docs/Developer%20Resources/Building%20ZFS.html
#
# Notes:
# * Starting from openzfs-2.2 release, native deb-based DKMS and user packages can be built.
# => We are compiling for => v2.2.1 so this is the way to go.
# * make native-deb builds native-deb-kmod and native-deb-utils.
# * Native Debian packages build with pre-configured paths for Debian and Ubuntu.
# It’s best not to override the paths during configure.
# * KVERS, KSRC and KOBJ environment variables can be exported
# to specify the kernel installed in non-default location.
./autogen.sh || { echo "Error during autgen.sh for zfs-${OPENZFS_VERSION} package build!"; exit 4; }
./configure || { echo "Error while configuring zfs-${OPENZFS_VERSION} package build!"; exit 5; }
make -s -j${BUILD_JOBS}
# Build the userland utilities and the DKMS package
# Note:
# If kernel modules are to be installed via DKMS package, this target should be sufficient.
make native-deb-utils || { echo "Error while making native-deb-utils for zfs-${OPENZFS_VERSION}!"; exit 6; }
# Build the kernel modules and package the binaries
# TBD FIXME hmm, KVERS is not enough, what to set for KSRC and KOBJ!?
#export KVERS=6.5.0-13-generic
#export KSRC=/usr/src/linux-headers-6.5.0-13-generic # hmm, looks like that doesn't work or is not enough
#export KOBJ=TBD what to set here?
# => Let's skip kmod generation for now and use dkms.
#make native-deb-kmod || { echo "Error while making native-deb-kmod for zfs-${OPENZFS_VERSION}!"; exit 7; }
# Install OpenZFS native deb packages
# TBD FIXME, hmm, installing everything doesn't make any sense at all
# * Install either openzfs-zfs-initramfs or openzfs-zfs-dracut
# * Install either openzfs-zfs-dkms or openzfs-zfs-modules-*
exit 99
sudo apt-get install --fix-missing ../*.deb
# Clean Up
popd > /dev/null || { echo "Error while popd directory!"; exit 2; }
rm -rf "${TMP_DIR}/zfs-${OPENZFS_VERSION}" # !! use a fixed sting in rm -rf!
exit 0Apparently 2.2.2 is released, but if I got (using my script) using git clone approach
# Use git and clone zfs-$version tag
git clone https://github.com/openzfs/zfs.git --depth 1 --tag zfs-$versionThen it compiles to 2.2.99 for some reason
FL140
commented
11 months ago @luckylinux please be careful with 2.2.2 and/or the above script for now! I ran into a bad pool crash and currently can't say where it comes from! see: https://github.com/openzfs/zfs/pull/15602#issuecomment-1835474484
FL140
commented
11 months ago Apparently 2.2.2 is released, but if I got (using my script) using git clone approach
# Use git and clone zfs-$version tag git clone https://github.com/openzfs/zfs.git --depth 1 --tag zfs-$versionThen it compiles to 2.2.99 for some reason
Well you can try my script from above, it is mostly comments to keep track of what has been changed and why. You only need to comment out the git clone command cloning from tonyhutter and enable the line above from the official repo.
The script currently exits prior the actual deb package install because the packages depend on if you use dracut or initramfs. Just move the one you don't need out of the way. The kmod package is not built currently in favor of dkms because I was not able to set the correct parameters for KVERS and friends as you can see from the comments in the script. And dkms also works on kernel updates from Ubuntu without regenerating a kernel module package.
Apart from a few _Description errors during install, which just seem to be the leading _ as a problem, the packages build and install fine.
luckylinux
commented
11 months ago Well with my script & initramfs it just finished installing the packages - at which point I saw your comment about the pool crash -_-.
luckylinux
commented
11 months ago My updated script for ZFS 2.2.2 and AMD64 with apt-get instead of apt (thanks again !)
#!/bin/bash
# Install Requirements
sudo apt-get --no-install-recommends install dkms
sudo apt-get install dh-dkms
sudo apt-get install aptitude libcurl4-openssl-dev libpam0g-dev lsb-release build-essential autoconf automake libtool libblkid-dev uuid-dev libudev-dev libssl-dev zlib1g-dev libaio-dev libattr1-dev libelf-dev python3 python3-dev python3-setuptools python3-cffi libffi-dev python3-packaging git libcurl4-openssl-dev debhelper-compat dh-python po-debconf python3-all-dev python3-sphinx
sudo apt-get install build-essential autoconf automake libtool gawk fakeroot libblkid-dev uuid-dev libudev-dev libssl-dev zlib1g-dev libaio-dev libattr1-dev libelf-dev python3 python3-dev python3-setuptools python3-cffi libffi-dev python3-packaging git libcurl4-openssl-dev debhelper-compat dh-python po-debconf python3-all-dev python3-sphinx
sudo apt-get install linux-headers-generic
# Define Desired Version
version="2.2.2"
cd /usr/src
mkdir -p zfs
cd zfs
# Save basedir
basedir=$(pwd)
# Use git and clone zfs-$version tag
#git clone https://github.com/openzfs/zfs.git --depth 1 --tag zfs-$version
#
# Use tar (working)
wget https://github.com/openzfs/zfs/archive/refs/tags/zfs-$version.tar.gz -O zfs-$version.tar.gz
mkdir -p zfs-$version
tar xvf zfs-$version.tar.gz -C zfs-$version --strip-components 1
#
# Use tar (currently broken archive)
#wget https://github.com/openzfs/zfs/releases/download/zfs-$version/zfs-$version.tar.gz -O zfs-$version.tar.gz
#tar xvf zfs-$version.tar.gz
# Change working direectory
cd zfs-$version
# Apply Patch in order to disable SIMD and Enable successfully ZFS Compile
#wget https://raw.githubusercontent.com/chimera-linux/cports/master/main/zfs/patches/aarch64-disable-neon.patch -O aarch64-disable-neon.patch
#patch -p1 < aarch64-disable-neon.patch
sh autogen.sh
./configure
make -s -j$(nproc)
make native-deb
make native-deb-utils native-deb-dkms
# Select Subset of Packages to prevent installation of default linux-image and linux-headers
cd $basedir
mkdir -p selected-packages
mkdir -p selected-packages/$version
cd selected-packages/$version/
mv ../../openzfs-libnvpair3_$version*.deb ./
mv ../../openzfs-libpam-zfs_$version*.deb ./
mv ../../openzfs-libuutil3_$version*.deb ./
mv ../../openzfs-libzfs4_$version*.deb ./
mv ../../openzfs-libzpool5_$version*.deb ./
mv ../../openzfs-zfs-dkms_$version*.deb ./
mv ../../openzfs-zfs-initramfs_$version*.deb ./
mv ../../openzfs-zfs-zed_$version*.deb ./
mv ../../openzfs-zfsutils_$version*.deb ./
sudo apt-get install --fix-missing ./*.debWell with my script & initramfs it just finished installing the packages - at which point I saw your comment about the pool crash -_-.
Sorry, we are writing in parallel and I am currently very tired and therefor, slow.... ;-(
BUT the panic occured during running of the block clone info script.
I booted again and I am running on 2.2.2 here since that (1-2h) without any complaints AFAIK.
FL140
commented
11 months ago OK, so apart from a few debug error messages I can't recall from the tonyhutter branch
dddc79cd349d97015a876ecf521da792b8ee7b51
c235ba91328ae18f2793b6b07e5d4cf17cce50f5
0629eb8462b7c8b74df095afd9fc326a34f27302
f5ece938dc6a5ed57904c7857c257d22038c0c3a
debugedit: debian/openzfs-libzfs4/lib/x86_64-linux-gnu/libzfs.so.4.1.0: Unknown DWARF DW_FORM_0x1f21
21ed593e6a37824ba0d07cc409a1a1d39d5554f5
94ba0342db297363d3ec91bf13f6cb85ec88677b
debugedit: debian/openzfs-zfsutils/usr/lib/zfs-linux/zpool_influxdb: Unknown DWARF DW_FORM_0x1f21
dc1b0725c000418a218591240f8b2d301f122750
debugedit: debian/openzfs-zfs-test/usr/share/zfs/zfs-tests/bin/zfs_diff-socket: Unknown DWARF DW_FORM_0x1f20
debugedit: debian/openzfs-zfsutils/sbin/zpool: Unknown DWARF DW_FORM_0x1f21
9da6e608a68ca0e87462a985635bbae9a4a8856f
6e2efaea2933723ece6405b7edb3369c701af1b7
04b1b99f5ef0d9551d14b0ad049f163bf16d6735
...The building of the deb packages went just fine. Installing and rebootin now...
luckylinux
commented
11 months ago I'm also tempted to try with the newly built ZFS 2.2.2 - wish me luck :D
FL140
commented
11 months ago I'm also tempted to try with the newly built ZFS 2.2.2 - wish me luck :D
Wait 5 minutes and let me be the guinea pig, the sh** already hit the fan here...
luckylinux
commented
11 months ago I'm also tempted to try with the newly built ZFS 2.2.2 - wish me luck :D
Wait 5 minutes and let me be the guinea pig, the sh** already hit the fan here...
Sorry to hear that. Well thanks to the other boot bug I usually suspend (for the past 16 days) I never shutdown the PC.
I already built ZFS 2.2.2 and it installed without complaints. Re-generated DKMS modules and re-generated initramfs.
FL140
commented
11 months ago I already built ZFS 2.2.2 and it installed without complaints. Re-generated DKMS modules and re-generated initramfs.
Same here official 2.2.2 compiles and installed fine (using dracut with a custom EFI setup and NOT the official GRUB / initramfs combo).
Reboot was uneventful and system is up and running so far, I just don't dare to run the block clone info script for now. ;-)
luckylinux
commented
11 months ago Well that's promising. But I'm starting to get more and more disappointed that a file system that proud itself for preventing data corruption now (and not for the first time) is causing data corruption after an update.
Bugs seems to cause more data corruption than bad ram / disks / cables it seems
rincebrain
commented
11 months ago I'm not really sure where you're getting the idea that the update is causing more corruption, and not that the existing issue on disk is causing a panic when you try to read it later. It's hard to guess without the actual crash message, but that would be my bet.
Also, zdb doesn't touch the running kernel, so if something is causing the running kernel to panic, it's not directly zdb. It might be that the script is walking the filesystem and hitting metadata that's munged, or something, but zdb itself can't mangle your bits.
FL140
commented
11 months ago Well that's promising. But I'm starting to get more and more disappointed that a file system that proud itself for preventing data corruption now (and not for the first time) is causing data corruption after an update.
Bugs seems to cause more data corruption than bad ram / disks / cables it seems
Well TBH I also had to hold my temper over the last two weeks sometimes, but at the end everyone is just trying to get over with the damage done. To be fair the underlying bug is time critical and those are very hard to find if you don't know what you are looking for. Over the last decades of unsing file systems I ran into a bunch of very unpleasant things with different file systems (ntfs, xfs, extN,...) so nothing is perfect. But I have to admit the amount of bugs that manifested over the last two weeks where a lot and I wonder if more in depth (heavy parallel load!) testing would have saved us from what we are all in now. My problem is that everyday when I think now we have all that data corruption crap fixed another thing pops up and ruins the day. I really hope this has an end soon and I will be able to fix my effected pools.
0x5c
commented
11 months ago It might be that the script is walking the filesystem and hitting metadata that's munged, or something, but zdb itself can't mangle your bits.
Yeah, seeing the output in the other issue it's either crashing when walking the filesystem or when trying to find the path/filename of a specific object.
FL140
commented
11 months ago I'm not really sure where you're getting the idea that the update is causing more corruption, and not that the existing issue on disk is causing a panic when you try to read it later. It's hard to guess without the actual crash message, but that would be my bet.
Also, zdb doesn't touch the running kernel, so if something is causing the running kernel to panic, it's not directly zdb. It might be that the script is walking the filesystem and hitting metadata that's munged, or something, but zdb itself can't mangle your bits.
I also think, the pool panic came from the data that got corrupted between 2.2.0 and 2.2.1 but I need to investigate as already written. The "panic" was related to the pool not the kernel. So while the machine crashed, I expect that happened because the root pool was put in panic mode. And not due to a kernel crash, but I also have to investigate that further. But after 16h into this I need a break.
luckylinux
commented
11 months ago Well that's promising. But I'm starting to get more and more disappointed that a file system that proud itself for preventing data corruption now (and not for the first time) is causing data corruption after an update. Bugs seems to cause more data corruption than bad ram / disks / cables it seems
Well TBH I also had to hold my temper over the last two weeks sometimes, but at the end everyone is just trying to get over with the damage done. To be fair the underlying bug is time critical and those are very hard to find if you don't know what you are looking for. Over the last decades of unsing file systems I ran into a bunch of very unpleasant things with different file systems (ntfs, xfs, extN,...) so nothing is perfect. But I have to admit the amount of bugs that manifested over the last two weeks where a lot and I wonder if more in depth (heavy parallel load!) testing would have saved us from what we are all in now. My problem is that everyday when I think now we have all that data corruption crap fixed another thing pops up and ruins the day. I really hope this has an end soon and I will be able to fix my effected pools.
I am thankful I wasn't hit yet (knock on wood)... But if we cannot trust zfs copy on write mechanism, we cannot trust our backups to be made in and with zfs and sent over to a remove host (zfs snapshot, zfs send/receive), I am not sure what we can trust.
Yesterday I once again got stuck with podman / docker because of ACLs permission working weirdly with zfs. Then programs left hanging and only solution is to reboot once every 10 minutes.
I don't know... It just feels like everything keep breaking.
You should always update for security and vulnerabilities reasons, but if every time stuff breaks 🙄
FL140
commented
11 months ago I am thankful I wasn't hit yet (knock on wood)... But if we cannot trust zfs copy on write mechanism, we cannot trust our backups to be made in and with zfs and sent over to a remove host (zfs snapshot, zfs send/receive), I am not sure what we can trust.
Well that is my biggest taking after the last two weeks that all my remote backups are also in danger, which is the worst case scenario. / paranoid mode on / So in the future I think I will additionally make backups with and rsync script (as prior zfs) to a remote site with having the backup on a different filesystem. Just in case everything blows up at once. / paranoid mode off /
rincebrain
commented
11 months ago You should always update for security and vulnerabilities reasons, but if every time stuff breaks 🙄
No, you should wait a little while and see what breaks, unless it fixes something that you need in your environment, and either way, test it somewhere less important before updating the important things.
I'm not saying people don't try to make things reliable to the best of their ability, but there are reasons I advise people to not install the latest and greatest immediately on things where any breakage could be bad. By definition, any amount of testing is going to be less than "everyone running it on all their varied systems" for anything popular enough, and no matter how well you test it, your tests are only as good as how varied the places you run them are.
If it's not broke, don't fix it.
(None of the above is specific to ZFS in any way - just, you know, the lower your tolerance for risk, the more cautious you need to be.)
luckylinux
commented
11 months ago You should always update for security and vulnerabilities reasons, but if every time stuff breaks 🙄
No, you should wait a little while and see what breaks, unless it fixes something that you need in your environment, and either way, test it somewhere less important before updating the important things.
I'm not saying people don't try to make things reliable to the best of their ability, but there are reasons I advise people to not install the latest and greatest immediately on things where any breakage could be bad. By definition, any amount of testing is going to be less than "everyone running it on all their varied systems" for anything popular enough, and no matter how well you test it, your tests are only as good as how varied the places you run them are.
If it's not broke, don't fix it.
(None of the above is specific to ZFS in any way - just, you know, the lower your tolerance for risk, the more cautious you need to be.)
I agree to some extent to the "if it isn't broken, don't fix it" approach.
I am usually not that scared of updating Debian for instance. Track record is good so far (knock on wood).
Ever since Ubuntu managed to self-uninstall of system tools during a distro upgrade I have become vary and skeptical of it.
But ZFS I used to have more confidence.
Sometimes upgrade is required. But then you have to upgrade kernel. But zfs is not supported natively. Then you need to manually compile zfs. And so on and so forth. Then I have to upgrade podman, but I also need all of the other tools etc.
I am usually quite happy with the distro shipped version of zfs. Except this buggy one on Ubuntu which is still a development version nonetheless 🙄.
FL140
commented
11 months ago I am usually quite happy with the distro shipped version of zfs. Except this buggy one on Ubuntu which is still a development version nonetheless 🙄.
That is exactly the problem for users of Ubuntu. And what is really unacceptable IMHO is how all of the last two weeks is just ignored by Ubuntu. The know of it because 2.2.1 is there for nobel, but still no hotfix 2.2.1 for mantic. I am not even starting to talk about 2.2.2.
So while the testing with the 2.2.2 patches suggest that the "block cloning" bug is very rare in real life for everything prior 2.2.1 and fixed in 2.2.2. I at least can confirm serious problems (pool panic) with a real world notebook, non heavy used and Ubuntu 23.10 after Upgrade to 23.10 (2.2.0 rc3? IIRC, as shipped with Mantic).
It is still too early to tell what really is the exact problem, but 2.2.0 was the cause for 99% (the pool gets regular scrubs at least once a month and the pool panic is caused by accessing BRT information with resulting system freeze) and since "we just don't know yet" and need to look further into this, just ignoring those Updates, is not a smart move, to put it mildly, by Ubuntu.
So since I have hard evidence now, I am going to fill a bug report with Ubuntu, but I just can hope this one gets taken more seriously finally by Ubuntu.
0x5c
commented
11 months ago and the pool panic is caused by accessing BRT information with resulting system freeze
It is not, since the crash you're seeing happens in a different command that dumps the list of all blocks* without any BRT-related information
*rather, it dumps the whole list of all objects which includes all blocks but none of the information from the BRT
FL140
commented
11 months ago and the pool panic is caused by accessing BRT information with resulting system freeze
It is not, since the crash you're seeing happens in a different command that dumps the list of all blocks* without any BRT-related information
*rather, it dumps the whole list of all objects which includes all blocks but none of the information from the BRT
@0x5c Sorry looks like I am not so deep into this to see that on first glance, but fact is that this simple script triggers a pool panic and a following system freeze. All on a notebook running stock Ubuntu 23.10 (fast installed 2.2.1 and now with 2.2.2) which was scrubed on a regular base without prior errors. For sure this was OK prior updrade from 23.04 to 23.10. This is good enough for me to take that all very seriously.
So any help on nailing this down is highly appreciated!
luckylinux
commented
11 months ago @0x5c if you can help me getting the script to work with my system, I can try to see if I can replicate @FL140 before I reboot. I'm kinda afraid of rebooting as it is now given what @FL140 experienced.
FL140
commented
11 months ago @0x5c if you can help me getting the script to work with my system, I can try to see if I can replicate @FL140 before I reboot. I'm kinda afraid of rebooting as it is now given what @FL140 experienced.
@luckylinux after having some sleep and having I deeper look into the zdb man page I should be now up dot date. So what is your problem and with what do you need help?
FL140
commented
11 months ago Cleaned up script for building the deb packages:
#!/bin/bash
# Ubuntu OpenZFS DKMS Kernel Module Install Script
typeset -r OPENZFS_VERSION=${1:-2.2.2}
typeset -r BOOT_TOOL=${2:-"initramfs"} # initramfs is the default on Ubuntu
#typeset BOOT_TOOL=${2:-"dracut"}
typeset -r TMP_DIR="/tmp"
typeset -r BUILD_JOBS=$(( $(nproc) / 2 )) # use half of what we have for building
if [[ ! "${BOOT_TOOL}" =~ ^initramfs|dracut$ ]]; then
echo "Invalid boot tool \"${BOOT_TOOL}\" [initramfs|dracut] parameter!"
exit 1
fi
# Installed required packages (Debian, Ubuntu)
# Note:
# TBD: hmm, I don't think we need package(s) alien and related packages any longer for versions >= v2.2.
sudo apt-get install -y build-essential autoconf automake libtool gawk \
fakeroot libblkid-dev uuid-dev libudev-dev libssl-dev zlib1g-dev \
libaio-dev libattr1-dev libelf-dev linux-headers-generic \
python3 python3-dev python3-setuptools python3-cffi libffi-dev \
python3-packaging debhelper-compat dh-python po-debconf \
python3-all-dev python3-sphinx \
git libcurl4-openssl-dev libpam0g-dev \
dkms dh-dkms
# Create working directory
[ -d "${TMP_DIR}/zfs-${OPENZFS_VERSION}" ] && { rm -rf "${TMP_DIR}/zfs-${OPENZFS_VERSION}"; } # !! use a fixed sting in rm -rf!
install -d -m 0770 -o root -g root "${TMP_DIR}/zfs-${OPENZFS_VERSION}"
pushd "${TMP_DIR}/zfs-${OPENZFS_VERSION}" > /dev/null || { echo "Error while pushd directory \"${TMP_DIR}/zfs-${OPENZFS_VERSION}\"!"; exit 2; }
# Get Source Code (GitHub)
git clone --branch zfs-${OPENZFS_VERSION} https://github.com/openzfs/zfs zfs-${OPENZFS_VERSION} || { echo "Error cloning zfs-${OPENZFS_VERSION} from GitHub!"; exit 3; }
cd zfs-${OPENZFS_VERSION}
# Get Patches (optional)
# see: https://openzfs.github.io/openzfs-docs/Developer%20Resources/OpenZFS%20Patches.html#openzfs-patches
#
# Official zfs-2.2.2 just (01.12.2023) released use it, no new patches yet.
#
# Notes:
# * If commented out, the following commits are already included into 2.2.2.
## dnode_is_dirty: check dnode and its data for dirtiness #15571
## https://github.com/openzfs/zfs/commit/9b9b09f452a469458451c221debfbab944e7f081
## fixes: some copied files are corrupted (chunks replaced by zeros) #15526
# git cherry-pick 9b9b09f452a469458451c221debfbab944e7f081
# Build OpenZFS native deb packages
# see: https://openzfs.github.io/openzfs-docs/Developer%20Resources/Building%20ZFS.html
#
# Notes:
# * Starting from openzfs-2.2 release, native deb-based DKMS and user packages can be built.
# => We are compiling for => v2.2.1 so this is the way to go.
# * make native-deb builds native-deb-kmod and native-deb-utils.
# * Native Debian packages build with pre-configured paths for Debian and Ubuntu.
# It’s best not to override the paths during configure.
# * KVERS, KSRC and KOBJ environment variables can be exported
# to specify the kernel installed in non-default location.
./autogen.sh || { echo "Error during autgen.sh for zfs-${OPENZFS_VERSION} package build!"; exit 4; }
./configure || { echo "Error while configuring zfs-${OPENZFS_VERSION} package build!"; exit 5; }
make -s -j${BUILD_JOBS}
# Build the userland utilities and the DKMS package
# Note:
# If kernel modules are to be installed via DKMS package, this target should be sufficient.
make native-deb-utils || { echo "Error while making native-deb-utils for zfs-${OPENZFS_VERSION}!"; exit 6; }
popd > /dev/null || { echo "Error while popd directory!"; exit 2; }
# Install OpenZFS native deb packages
# Note:
# * Install either openzfs-zfs-initramfs or openzfs-zfs-dracut
# * Install either openzfs-zfs-dkms or openzfs-zfs-modules-*
# => dkms is more flexible so use it as default.
sudo apt-get install --fix-missing -y \
./openzfs-lib*_${OPENZFS_VERSION}*.deb \
./openzfs-py*_${OPENZFS_VERSION}*.deb \
./openzfs-zfs-dkms_${OPENZFS_VERSION}*.deb \
./openzfs-zfsutils_${OPENZFS_VERSION}*.deb \
./openzfs-zfs-zed_${OPENZFS_VERSION}*.deb \
./openzfs-zfs-test_${OPENZFS_VERSION}*.deb \
./openzfs-zfs-${BOOT_TOOL}_${OPENZFS_VERSION}*.deb
# Clean Up
rm -rf "${TMP_DIR}/zfs-${OPENZFS_VERSION}" # !! use a fixed sting in rm -rf!
exit 0The script builds 2.2.2 with initramfs in default mode. But you can provide the version to build and the build tool (either initramfs or dracut).
The script was tested sucessfully on Ubuntu 23.10 running a personalized dracut based boot process using UKI. (Which shouldn't matter for you, initramfs is just not tested.)
luckylinux
commented
11 months ago @0x5c if you can help me getting the script to work with my system, I can try to see if I can replicate @FL140 before I reboot. I'm kinda afraid of rebooting as it is now given what @FL140 experienced.
@luckylinux after having some sleep and having I deeper look into the zdb man page I should be now up dot date. So what is your problem and with what do you need help?
I described it in the other thread actually https://github.com/openzfs/zfs/issues/15404#issuecomment-1835640633
But TLDR is I cannot get the zdb script to work and list the corrupted blocks (if any).
luckylinux
commented
11 months ago Cleaned up script for building the deb packages:
#!/bin/bash # Ubuntu OpenZFS DKMS Kernel Module Install Script typeset -r OPENZFS_VERSION=${1:-2.2.2} typeset -r BOOT_TOOL=${2:-"initramfs"} # initramfs is the default on Ubuntu #typeset BOOT_TOOL=${2:-"dracut"} typeset -r TMP_DIR="/tmp" typeset -r BUILD_JOBS=$(( $(nproc) / 2 )) # use half of what we have for building if [[ ! "${BOOT_TOOL}" =~ ^initramfs|dracut$ ]]; then echo "Invalid boot tool \"${BOOT_TOOL}\" [initramfs|dracut] parameter!" exit 1 fi # Installed required packages (Debian, Ubuntu) # Note: # TBD: hmm, I don't think we need package(s) alien and related packages any longer for versions >= v2.2. sudo apt-get install -y build-essential autoconf automake libtool gawk \ fakeroot libblkid-dev uuid-dev libudev-dev libssl-dev zlib1g-dev \ libaio-dev libattr1-dev libelf-dev linux-headers-generic \ python3 python3-dev python3-setuptools python3-cffi libffi-dev \ python3-packaging debhelper-compat dh-python po-debconf \ python3-all-dev python3-sphinx \ git libcurl4-openssl-dev libpam0g-dev \ dkms dh-dkms # Create working directory [ -d "${TMP_DIR}/zfs-${OPENZFS_VERSION}" ] && { rm -rf "${TMP_DIR}/zfs-${OPENZFS_VERSION}"; } # !! use a fixed sting in rm -rf! install -d -m 0770 -o root -g root "${TMP_DIR}/zfs-${OPENZFS_VERSION}" pushd "${TMP_DIR}/zfs-${OPENZFS_VERSION}" > /dev/null || { echo "Error while pushd directory \"${TMP_DIR}/zfs-${OPENZFS_VERSION}\"!"; exit 2; } # Get Source Code (GitHub) git clone --branch zfs-${OPENZFS_VERSION} https://github.com/openzfs/zfs zfs-${OPENZFS_VERSION} || { echo "Error cloning zfs-${OPENZFS_VERSION} from GitHub!"; exit 3; } cd zfs-${OPENZFS_VERSION} # Get Patches (optional) # see: https://openzfs.github.io/openzfs-docs/Developer%20Resources/OpenZFS%20Patches.html#openzfs-patches # # Official zfs-2.2.2 just (01.12.2023) released use it, no new patches yet. # # Notes: # * If commented out, the following commits are already included into 2.2.2. ## dnode_is_dirty: check dnode and its data for dirtiness #15571 ## https://github.com/openzfs/zfs/commit/9b9b09f452a469458451c221debfbab944e7f081 ## fixes: some copied files are corrupted (chunks replaced by zeros) #15526 # git cherry-pick 9b9b09f452a469458451c221debfbab944e7f081 # Build OpenZFS native deb packages # see: https://openzfs.github.io/openzfs-docs/Developer%20Resources/Building%20ZFS.html # # Notes: # * Starting from openzfs-2.2 release, native deb-based DKMS and user packages can be built. # => We are compiling for => v2.2.1 so this is the way to go. # * make native-deb builds native-deb-kmod and native-deb-utils. # * Native Debian packages build with pre-configured paths for Debian and Ubuntu. # It’s best not to override the paths during configure. # * KVERS, KSRC and KOBJ environment variables can be exported # to specify the kernel installed in non-default location. ./autogen.sh || { echo "Error during autgen.sh for zfs-${OPENZFS_VERSION} package build!"; exit 4; } ./configure || { echo "Error while configuring zfs-${OPENZFS_VERSION} package build!"; exit 5; } make -s -j${BUILD_JOBS} # Build the userland utilities and the DKMS package # Note: # If kernel modules are to be installed via DKMS package, this target should be sufficient. make native-deb-utils || { echo "Error while making native-deb-utils for zfs-${OPENZFS_VERSION}!"; exit 6; } popd > /dev/null || { echo "Error while popd directory!"; exit 2; } # Install OpenZFS native deb packages # Note: # * Install either openzfs-zfs-initramfs or openzfs-zfs-dracut # * Install either openzfs-zfs-dkms or openzfs-zfs-modules-* # => dkms is more flexible so use it as default. sudo apt-get install --fix-missing -y \ ./openzfs-lib*_${OPENZFS_VERSION}*.deb \ ./openzfs-py*_${OPENZFS_VERSION}*.deb \ ./openzfs-zfs-dkms_${OPENZFS_VERSION}*.deb \ ./openzfs-zfsutils_${OPENZFS_VERSION}*.deb \ ./openzfs-zfs-zed_${OPENZFS_VERSION}*.deb \ ./openzfs-zfs-test_${OPENZFS_VERSION}*.deb \ ./openzfs-zfs-${BOOT_TOOL}_${OPENZFS_VERSION}*.deb # Clean Up rm -rf "${TMP_DIR}/zfs-${OPENZFS_VERSION}" # !! use a fixed sting in rm -rf! exit 0The script builds 2.2.2 with initramfs in default mode. But you can provide the version to build and the build tool (either initramfs or dracut).
The script was tested sucessfully on Ubuntu 23.10 running a personalized dracut based boot process using UKI. (Which shouldn't matter for you, initramfs is just not tested.)
Well actually, besides a few debugging differences (my script is just "open-loop" command after command, yours actually handles some of the errors which might occur) we should be getting the same result. I'm not that advanced in bash 😉.
I can also test yours. Only question is does it build 2.2.2 when checked out from git with the tag? It built 2.2.99 for me, that's why I fell back on the tar tarball which is working (if you download the correct one).
luckylinux
commented
11 months ago Cleaned up script for building the deb packages:
#!/bin/bash # Ubuntu OpenZFS DKMS Kernel Module Install Script typeset -r OPENZFS_VERSION=${1:-2.2.2} typeset -r BOOT_TOOL=${2:-"initramfs"} # initramfs is the default on Ubuntu #typeset BOOT_TOOL=${2:-"dracut"} typeset -r TMP_DIR="/tmp" typeset -r BUILD_JOBS=$(( $(nproc) / 2 )) # use half of what we have for building if [[ ! "${BOOT_TOOL}" =~ ^initramfs|dracut$ ]]; then echo "Invalid boot tool \"${BOOT_TOOL}\" [initramfs|dracut] parameter!" exit 1 fi # Installed required packages (Debian, Ubuntu) # Note: # TBD: hmm, I don't think we need package(s) alien and related packages any longer for versions >= v2.2. sudo apt-get install -y build-essential autoconf automake libtool gawk \ fakeroot libblkid-dev uuid-dev libudev-dev libssl-dev zlib1g-dev \ libaio-dev libattr1-dev libelf-dev linux-headers-generic \ python3 python3-dev python3-setuptools python3-cffi libffi-dev \ python3-packaging debhelper-compat dh-python po-debconf \ python3-all-dev python3-sphinx \ git libcurl4-openssl-dev libpam0g-dev \ dkms dh-dkms # Create working directory [ -d "${TMP_DIR}/zfs-${OPENZFS_VERSION}" ] && { rm -rf "${TMP_DIR}/zfs-${OPENZFS_VERSION}"; } # !! use a fixed sting in rm -rf! install -d -m 0770 -o root -g root "${TMP_DIR}/zfs-${OPENZFS_VERSION}" pushd "${TMP_DIR}/zfs-${OPENZFS_VERSION}" > /dev/null || { echo "Error while pushd directory \"${TMP_DIR}/zfs-${OPENZFS_VERSION}\"!"; exit 2; } # Get Source Code (GitHub) git clone --branch zfs-${OPENZFS_VERSION} https://github.com/openzfs/zfs zfs-${OPENZFS_VERSION} || { echo "Error cloning zfs-${OPENZFS_VERSION} from GitHub!"; exit 3; } cd zfs-${OPENZFS_VERSION} # Get Patches (optional) # see: https://openzfs.github.io/openzfs-docs/Developer%20Resources/OpenZFS%20Patches.html#openzfs-patches # # Official zfs-2.2.2 just (01.12.2023) released use it, no new patches yet. # # Notes: # * If commented out, the following commits are already included into 2.2.2. ## dnode_is_dirty: check dnode and its data for dirtiness #15571 ## https://github.com/openzfs/zfs/commit/9b9b09f452a469458451c221debfbab944e7f081 ## fixes: some copied files are corrupted (chunks replaced by zeros) #15526 # git cherry-pick 9b9b09f452a469458451c221debfbab944e7f081 # Build OpenZFS native deb packages # see: https://openzfs.github.io/openzfs-docs/Developer%20Resources/Building%20ZFS.html # # Notes: # * Starting from openzfs-2.2 release, native deb-based DKMS and user packages can be built. # => We are compiling for => v2.2.1 so this is the way to go. # * make native-deb builds native-deb-kmod and native-deb-utils. # * Native Debian packages build with pre-configured paths for Debian and Ubuntu. # It’s best not to override the paths during configure. # * KVERS, KSRC and KOBJ environment variables can be exported # to specify the kernel installed in non-default location. ./autogen.sh || { echo "Error during autgen.sh for zfs-${OPENZFS_VERSION} package build!"; exit 4; } ./configure || { echo "Error while configuring zfs-${OPENZFS_VERSION} package build!"; exit 5; } make -s -j${BUILD_JOBS} # Build the userland utilities and the DKMS package # Note: # If kernel modules are to be installed via DKMS package, this target should be sufficient. make native-deb-utils || { echo "Error while making native-deb-utils for zfs-${OPENZFS_VERSION}!"; exit 6; } popd > /dev/null || { echo "Error while popd directory!"; exit 2; } # Install OpenZFS native deb packages # Note: # * Install either openzfs-zfs-initramfs or openzfs-zfs-dracut # * Install either openzfs-zfs-dkms or openzfs-zfs-modules-* # => dkms is more flexible so use it as default. sudo apt-get install --fix-missing -y \ ./openzfs-lib*_${OPENZFS_VERSION}*.deb \ ./openzfs-py*_${OPENZFS_VERSION}*.deb \ ./openzfs-zfs-dkms_${OPENZFS_VERSION}*.deb \ ./openzfs-zfsutils_${OPENZFS_VERSION}*.deb \ ./openzfs-zfs-zed_${OPENZFS_VERSION}*.deb \ ./openzfs-zfs-test_${OPENZFS_VERSION}*.deb \ ./openzfs-zfs-${BOOT_TOOL}_${OPENZFS_VERSION}*.deb # Clean Up rm -rf "${TMP_DIR}/zfs-${OPENZFS_VERSION}" # !! use a fixed sting in rm -rf! exit 0The script builds 2.2.2 with initramfs in default mode. But you can provide the version to build and the build tool (either initramfs or dracut).
The script was tested sucessfully on Ubuntu 23.10 running a personalized dracut based boot process using UKI. (Which shouldn't matter for you, initramfs is just not tested.)
I tried your script. Not sure what's wrong, but the install part is buggy.
Attached the build log. Last ~ 10 lines of the log are most relevant.
rincebrain
commented
11 months ago apt-get needs full paths, not relative.
luckylinux
commented
11 months ago apt-get needs full paths, not relative.
With my script it also works with relative paths:
root@UbuntuWorkstation02:/usr/src/zfs# cd selected-packages/2.2.2
root@UbuntuWorkstation02:/usr/src/zfs/selected-packages/2.2.2# sudo apt-get install --fix-missing ./*.deb
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Note, selecting 'openzfs-libnvpair3' instead of './openzfs-libnvpair3_2.2.2-1_amd64.deb'
Note, selecting 'openzfs-libpam-zfs' instead of './openzfs-libpam-zfs_2.2.2-1_amd64.deb'
Note, selecting 'openzfs-libuutil3' instead of './openzfs-libuutil3_2.2.2-1_amd64.deb'
Note, selecting 'openzfs-libzfs4' instead of './openzfs-libzfs4_2.2.2-1_amd64.deb'
Note, selecting 'openzfs-libzpool5' instead of './openzfs-libzpool5_2.2.2-1_amd64.deb'
Note, selecting 'openzfs-zfs-dkms' instead of './openzfs-zfs-dkms_2.2.2-1_all.deb'
Note, selecting 'openzfs-zfs-initramfs' instead of './openzfs-zfs-initramfs_2.2.2-1_all.deb'
Note, selecting 'openzfs-zfsutils' instead of './openzfs-zfsutils_2.2.2-1_amd64.deb'
Note, selecting 'openzfs-zfs-zed' instead of './openzfs-zfs-zed_2.2.2-1_amd64.deb'
openzfs-libnvpair3 is already the newest version (2.2.2-1).
openzfs-libpam-zfs is already the newest version (2.2.2-1).
openzfs-libuutil3 is already the newest version (2.2.2-1).
openzfs-libzfs4 is already the newest version (2.2.2-1).
openzfs-libzpool5 is already the newest version (2.2.2-1).
openzfs-zfs-dkms is already the newest version (2.2.2-1).
openzfs-zfs-initramfs is already the newest version (2.2.2-1).
openzfs-zfsutils is already the newest version (2.2.2-1).
openzfs-zfs-zed is already the newest version (2.2.2-1).
0 upgraded, 0 newly installed, 0 to remove and 78 not upgraded.(it was already installed - re-ran the command to prove my point)
System information
Describe the problem you're observing
I am trying to build a deb zfs dkms package for Ubuntu 23.10 from the release 2.2.1 tar package from GitHub according to the recommendation given in https://github.com/openzfs/zfs/pull/15529#issuecomment-1823169348 if followed the instructions given in https://openzfs.github.io/openzfs-docs/Developer%20Resources/Custom%20Packages.html .
I am trying this because of the block clone bug and the following fixes introduced with 2.2.1, unfortunately there is still no package for Ubuntu available and I need to fix the systems I am responsible for ASAP.
Unfortunately it seems as the instructions and release package have issues.
The following issues were found:
contrib/debian/controlis missing in the release tar package.zfs-${OPENZFS_VERSION}notzfsalienpackage (and others?) should no longer be needed since 2.2.0 (if I understand the instructions correctly). (Note: The control file not found error pops up with and without packagealieninstalled.)Only the first issue needs solving, I already solved the other issues (except that the documentation needs an update).
Describe how to reproduce the problem
I wrote an installation script:
The command
make native-deb-utilstriggers the error:Include any warning/errors/backtraces from the system logs
Script output: