HankB / Linux_ZFS_Root

Commands/script to install Linux on ZFS root
10 stars 6 forks source link

Clarification of the steps #17

Closed humphreylee closed 3 years ago

humphreylee commented 3 years ago

First of all, thanks for the excellent script. I'm trying to do ZFS root on Debian Bullseye, by running the script line-by-line, which will be a dual boot with Win10. There are two scripts this1 and this2 - which script to use?

Appreciate your clarification on some steps in this1, from line 333 to 350. I guess these are ZFS related commands to import ZFS's bpool. It is a lot clearer in your script compared to Nourlin's blog

Do I need to run initial, then this1 and finally post reboot script? The env.sh usage is not that clear, Thanks.

HankB commented 3 years ago

"this1" is a script I wrote. "this2" is someone else' script.

333:350 creates a systemd service file to import bpool using /etc/zfs/zpool.cache. The "4.9" tag refers to the instructions at https://openzfs.github.io/openzfs-docs/Getting%20Started/Debian/Debian%20Buster%20Root%20on%20ZFS.html at the time I wrote the script. I see the instructions have now been renumbered and that is now "11"

"initial" is a minor convenience to install/start an SSH server in the live environment in case you want to SSH in to perform the rest of the procedure.

"post-reboot" is similarly a convenience to perform the post reboot operations that I usually need to do.

"env.sh" sets up environment variables that control how the script operates, in other words, does it do whole disk, selected partitions or previously prepared pools. "env.sh" is the default file name for these. If you look in the .../testing directory you will find a bunch of variants of this that configure for particular combinations of configurations. I find it convenient to choose one of these as a starting point when getting ready to install. For example, if I were installing dual boot to selected partitions and using ZFS native encryption I would

Since I wrote the script, ZFS now supports zstd compression and I would use that instead of lz4.

Does this answer your questions?

best,

On Wed, Sep 1, 2021 at 9:13 PM humphreylee @.***> wrote:

First of all, thanks for the excellent script. I'm trying to do ZFS root on Debian Bullseye, by running the script line-by-line, which will be a dual boot with Win10. There are two scripts this1 https://github.com/HankB/Linux_ZFS_Root/blob/master/Debian/install_Debian_to_ZFS_root.sh and this2 https://raw.githubusercontent.com/hn/debian-buster-zfs-root/master/debian-buster-zfs-root.sh

  • which script to use?

Appreciate your clarification on some steps in this1 https://github.com/HankB/Linux_ZFS_Root/blob/master/Debian/install_Debian_to_ZFS_root.sh, from line 333 to 350. I guess these are ZFS related commands to import ZFS's bpool. It is a lot clearer in your script compared to Nourlin's blog https://noulin.net/blog/linux/2021/06/26/setting-up-root-on-zfs-in-debian-bullseye.html

Do I need to run initial https://github.com/HankB/Linux_ZFS_Root/blob/master/Debian/initial.sh, then this1 https://github.com/HankB/Linux_ZFS_Root/blob/master/Debian/install_Debian_to_ZFS_root.sh and finally post reboot https://github.com/HankB/Linux_ZFS_Root/blob/master/Debian/post-reboot.sh script? The env.sh usage is not that clear, Thanks.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/HankB/Linux_ZFS_Root/issues/17, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZM47VDRS4BSMC7KUT256TT73MUNANCNFSM5DH3QYTA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- '03 BMW F650CS - hers '98 Dakar K12RS - "BABY K" grew up. '93 R100R w/ Velorex 700 (MBD starts...) '95 Miata - "OUR LC" polish visor: apply squashed bugs, rinse, repeat Beautiful Sunny Winfield, Illinois

humphreylee commented 3 years ago

Thank you very much. You've answered most of my questions. Good to know about zstd - will include that in my script. Some other customisation includes encryption for /home rpool/datasets

I've pre-created boot (1 GiB), root (200+ GiB) partitions for this installation. The EFI is on partition-part1. Let me check out the part.std.zfs_enc.sh script.

As for SecureBoot system, my plan is as outlined below. Do you think this is doable?

  1. Disable SecureBoot
  2. Install Debian with ZFS (your script). Before reboot, a critically important step is to register dkms modules (e.g zfs) key into MOK.
  3. Enable SecureBoot and register the key.
  4. Boot normally.

Thank you.

humphreylee commented 3 years ago

May I have some understanding for script part.std.zfs_enc.sh

  1. Since I will be using Bullseye (ZFS version = 2.0.3), line 13 to 15 will just have all enabled like below? export ZFS_CRYPT="yes" export LUKS_CRYPT="no" export ENCRYPT=yes # must be set

  2. Line 37 to 39 - do I disable all?

  3. Since I will be using selected partitions, do I disable like 43 & 44?

  4. Since I already have EFI, then I disable like 46 to 51 and enable like 67?

HankB commented 3 years ago

Line numbers from: https://github.com/HankB/Linux_ZFS_Root/blob/master/Debian/testing/part.std.zfs_enc.sh

On Thu, Sep 2, 2021 at 12:19 AM humphreylee @.***> wrote:

May I have some understanding for script part.std.zfs_enc.sh

1.

Since I will be using Bullseye (ZFS version = 2.0.3 https://packages.debian.org/bullseye/zfs-dkms), line 13 to 15 will just have all enabled like below? export ZFS_CRYPT="yes" export LUKS_CRYPT="no" export ENCRYPT=yes # must be set

Lines 15-17 can be left as is. "ENCRYPT=yes" is a remnant from when LUKS was the only encryption option and is no longer used.

1.

Line 37 to 39 - do I disable all?

43-49 wipe the entire drive and create partitions for testing and you do not want for an existing system.

1. 2.

Since I will be using selected partitions, do I disable like 43 & 44?

If you have already created the target partitions, you do not want to execute any sgdisk commands.

1. 2.

Since I already have EFI, then I disable like 46 to 51 and enable like 67?

You will need to have the three partition variables EFI_PART, BOOT_PART and ROOT_PART pointing to your target partitions.

Apologies for the confusion. I see that I have a mix of commands that produce the test environment (and not desired for normal use of the script) and the commands to set up the environment for the script. (And some old cruft that is no longer needed.) I should clarify those before proposing that someone use it as a starting point. Thank you for pointing this out!

best,

-- '03 BMW F650CS - hers '98 Dakar K12RS - "BABY K" grew up. '93 R100R w/ Velorex 700 (MBD starts...) '95 Miata - "OUR LC" polish visor: apply squashed bugs, rinse, repeat Beautiful Sunny Winfield, Illinois

HankB commented 3 years ago

I have updated my script to support Bullseye and clarified testing/ part.std.zfs_enc.sh and tested this along with whole disk, no encryption. These updates are in the branch "bullseye".

Please review and if this is clear, you may close the issue.

On Thu, Sep 2, 2021 at 7:12 AM Hank Barta @.***> wrote:

Line numbers from:

https://github.com/HankB/Linux_ZFS_Root/blob/master/Debian/testing/part.std.zfs_enc.sh

On Thu, Sep 2, 2021 at 12:19 AM humphreylee @.***> wrote:

May I have some understanding for script part.std.zfs_enc.sh

1.

Since I will be using Bullseye (ZFS version = 2.0.3 https://packages.debian.org/bullseye/zfs-dkms), line 13 to 15 will just have all enabled like below? export ZFS_CRYPT="yes" export LUKS_CRYPT="no" export ENCRYPT=yes # must be set

Lines 15-17 can be left as is. "ENCRYPT=yes" is a remnant from when LUKS was the only encryption option and is no longer used.

1.

Line 37 to 39 - do I disable all?

43-49 wipe the entire drive and create partitions for testing and you do not want for an existing system.

1. 2.

Since I will be using selected partitions, do I disable like 43 & 44?

If you have already created the target partitions, you do not want to execute any sgdisk commands.

1. 2.

Since I already have EFI, then I disable like 46 to 51 and enable like 67?

You will need to have the three partition variables EFI_PART, BOOT_PART and ROOT_PART pointing to your target partitions.

Apologies for the confusion. I see that I have a mix of commands that produce the test environment (and not desired for normal use of the script) and the commands to set up the environment for the script. (And some old cruft that is no longer needed.) I should clarify those before proposing that someone use it as a starting point. Thank you for pointing this out!

best,

-- '03 BMW F650CS - hers '98 Dakar K12RS - "BABY K" grew up. '93 R100R w/ Velorex 700 (MBD starts...) '95 Miata - "OUR LC" polish visor: apply squashed bugs, rinse, repeat Beautiful Sunny Winfield, Illinois

-- '03 BMW F650CS - hers '98 Dakar K12RS - "BABY K" grew up. '93 R100R w/ Velorex 700 (MBD starts...) '95 Miata - "OUR LC" polish visor: apply squashed bugs, rinse, repeat Beautiful Sunny Winfield, Illinois

humphreylee commented 3 years ago

Sorry for my ignorant. I can't find the bullseye branch. Would you kindly put the link?

HankB commented 3 years ago

Try https://github.com/HankB/Linux_ZFS_Root/tree/bullseye

On Thu, Sep 2, 2021 at 7:32 PM humphreylee @.***> wrote:

Sorry for my ignorant. I can't find the bullseye branch. Would you kindly put the link?

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/HankB/Linux_ZFS_Root/issues/17#issuecomment-912159341, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZM47UWAOAKQGYTBD73ZVTUAAJQPANCNFSM5DH3QYTA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- '03 BMW F650CS - hers '98 Dakar K12RS - "BABY K" grew up. '93 R100R w/ Velorex 700 (MBD starts...) '95 Miata - "OUR LC" polish visor: apply squashed bugs, rinse, repeat Beautiful Sunny Winfield, Illinois

humphreylee commented 3 years ago

Thanks for the update. I used latest bullseye script to install on selected partitions (with encryption). However, upon boot-up, error occurred - as shown in image below. In addition, the grub-install failed to detect and add my Win10 OS into the grub menu. Appreciate your help.

zfs

humphreylee commented 3 years ago

I managed to get it running by entering the following command zpool import -aN -d /dev/disk/by-id -o cachefile=none. How to incorporate this, such that it can boot normally?

HankB commented 3 years ago

That's good. Did you try rebooting? I've seen this before and normally it just requires import bpool on the first boot. I recall some systems that required a delay during boot to allow the NVME drive to be ready but can't find that in my notes at the moment. (Getting ready to run before it gets too warm here.)

On Sat, Sep 4, 2021 at 6:56 AM humphreylee @.***> wrote:

I managed to get it running by entering the following command zpool import -aN -d /dev/disk/by-id -o cachefile=none. How to incorporate this, such that it can boot normally?

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/HankB/Linux_ZFS_Root/issues/17#issuecomment-912961594, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZM47UQAAVMVLP465HRKLTUAICOZANCNFSM5DH3QYTA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- '03 BMW F650CS - hers '98 Dakar K12RS - "BABY K" grew up. '93 R100R w/ Velorex 700 (MBD starts...) '95 Miata - "OUR LC" polish visor: apply squashed bugs, rinse, repeat Beautiful Sunny Winfield, Illinois

HankB commented 3 years ago

On one of my systems it was necessary to delay mounting root: @.:~$ grep rootdelay /etc/default/grub GRUB_CMDLINE_LINUX_DEFAULT="rootdelay=10 quiet" @.:~$ (Need to run update-grub for changes in this file to take effect.)

On Sat, Sep 4, 2021 at 7:00 AM Hank Barta @.***> wrote:

That's good. Did you try rebooting? I've seen this before and normally it just requires import bpool on the first boot. I recall some systems that required a delay during boot to allow the NVME drive to be ready but can't find that in my notes at the moment. (Getting ready to run before it gets too warm here.)

On Sat, Sep 4, 2021 at 6:56 AM humphreylee @.***> wrote:

I managed to get it running by entering the following command zpool import -aN -d /dev/disk/by-id -o cachefile=none. How to incorporate this, such that it can boot normally?

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/HankB/Linux_ZFS_Root/issues/17#issuecomment-912961594, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZM47UQAAVMVLP465HRKLTUAICOZANCNFSM5DH3QYTA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- '03 BMW F650CS - hers '98 Dakar K12RS - "BABY K" grew up. '93 R100R w/ Velorex 700 (MBD starts...) '95 Miata - "OUR LC" polish visor: apply squashed bugs, rinse, repeat Beautiful Sunny Winfield, Illinois

-- '03 BMW F650CS - hers '98 Dakar K12RS - "BABY K" grew up. '93 R100R w/ Velorex 700 (MBD starts...) '95 Miata - "OUR LC" polish visor: apply squashed bugs, rinse, repeat Beautiful Sunny Winfield, Illinois

humphreylee commented 3 years ago

Subsequent boots still same. Will try the recommendations.

humphreylee commented 3 years ago

Doesn't seems to work. After fresh reboot, it still requires the magic line.

HankB commented 3 years ago

The first thing I would try is a longer bootdelay. While you are doing that, review the instructions for section 5 step 7 to see if the cache files exist. It probably wouldn't hurt to repeat those procedures that create the cache file.

I always update grub and refresh initramfs when I'm in the rescue mode.

Also be sure to repeat the commands in Step 6, item 4 to unmount the filesystems and export the pool. They're in the section "First Boot" but need to be performed before shutting down the live/rescue system (and rebooting.)

As far as detecting Windows (when you get to that point) make sure that the package os-prober is installed.

I'm configuring a laptop to test on real hardware but am not there yet.

On Sat, Sep 4, 2021 at 8:34 AM humphreylee @.***> wrote:

Doesn't seems to work. After fresh reboot, it still requires the magic line.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/HankB/Linux_ZFS_Root/issues/17#issuecomment-912975339, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZM47TBNK7MS5SL4CX5WOTUAIN5HANCNFSM5DH3QYTA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- '03 BMW F650CS - hers '98 Dakar K12RS - "BABY K" grew up. '93 R100R w/ Velorex 700 (MBD starts...) '95 Miata - "OUR LC" polish visor: apply squashed bugs, rinse, repeat Beautiful Sunny Winfield, Illinois

humphreylee commented 3 years ago

The first thing I would try is a longer bootdelay

rootdelay=30. No different.

Also be sure to repeat the commands in Step 6, item 4 to unmount the filesystems and export the pool. They're in the section "First Boot" but need to be performed before shutting down the live/rescue system (and rebooting.)

Are you referring to line 464 to 485 of install_Debian_to_ZFS_root.sh?

As far as detecting Windows (when you get to that point) make sure that the package os-prober is installed.

Win10 has been added to the menu after running update-grub from the successful boot.

HankB commented 3 years ago

rootdelay=30. No different.

Not the problem then.

Are you referring to line 464 to 485 of install_Debian_to_ZFS_root.sh?

lines 471 through 475 in https://github.com/HankB/Linux_ZFS_Root/blob/bullseye/Debian/install_Debian_to_ZFS_root.sh .

On Sun, Sep 5, 2021 at 6:44 AM humphreylee @.***> wrote:

In install_Debian_to_ZFS_root.sh, why it does not do zpool create for "$INSTALL_TYPE" == "use_partitions" case, i.e. something like like line 119-135 as it has done the wipefs?

For "$INSTALL_TYPE" == "use_partitions" wipefs is executed for the boot and root pool partitions on lines 102 and 103.

-- '03 BMW F650CS - hers '98 Dakar K12RS - "BABY K" grew up. '93 R100R w/ Velorex 700 (MBD starts...) '95 Miata - "OUR LC" polish visor: apply squashed bugs, rinse, repeat Beautiful Sunny Winfield, Illinois

HankB commented 3 years ago

I suggest focusing on the process that prepares the cache files, lines 424 through 461.

Can you show the contents f/etc/zfs/zfs-list.cache/bpooland/etc/zfs/zfs-list.cache/rpool` (can be posted to something like pastebin)

Can you set the environment variables for ROOT_POOL_NAME and BOOT_POOL_NAME and repeat these steps and capture the output as well as the contents of the two cache files as well?

It might be worthwhile asking about this on the mailing list. I have not had this much difficulty importing pools at boot but I know it has been discussed on the list.

HankB commented 3 years ago

I performed a similar installation on an old Macbook (2008 aluminum) with the following setup

I did not experience this issue. I did uncomment both "set -x" commands and captured a complete record of the install. If you decide to repeat the install, please do this so I can compare results and hopefully gain some insight into what is not working as desired.

On Sat, Sep 4, 2021 at 11:36 AM Hank Barta @.***> wrote:

The first thing I would try is a longer bootdelay. While you are doing that, review the instructions for section 5 step 7 to see if the cache files exist. It probably wouldn't hurt to repeat those procedures that create the cache file.

I always update grub and refresh initramfs when I'm in the rescue mode.

Also be sure to repeat the commands in Step 6, item 4 to unmount the filesystems and export the pool. They're in the section "First Boot" but need to be performed before shutting down the live/rescue system (and rebooting.)

As far as detecting Windows (when you get to that point) make sure that the package os-prober is installed.

I'm configuring a laptop to test on real hardware but am not there yet.

On Sat, Sep 4, 2021 at 8:34 AM humphreylee @.***> wrote:

Doesn't seems to work. After fresh reboot, it still requires the magic line.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/HankB/Linux_ZFS_Root/issues/17#issuecomment-912975339, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZM47TBNK7MS5SL4CX5WOTUAIN5HANCNFSM5DH3QYTA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- '03 BMW F650CS - hers '98 Dakar K12RS - "BABY K" grew up. '93 R100R w/ Velorex 700 (MBD starts...) '95 Miata - "OUR LC" polish visor: apply squashed bugs, rinse, repeat Beautiful Sunny Winfield, Illinois

-- '03 BMW F650CS - hers '98 Dakar K12RS - "BABY K" grew up. '93 R100R w/ Velorex 700 (MBD starts...) '95 Miata - "OUR LC" polish visor: apply squashed bugs, rinse, repeat Beautiful Sunny Winfield, Illinois

humphreylee commented 3 years ago

Can you show the contents f/etc/zfs/zfs-list.cache/bpooland/etc/zfs/zfs-list.cache/rpool` (can be posted to something like pastebin)

bpool.txt rpool.txt

Ignore the .txt extension, which I added to allow the upload.

humphreylee commented 3 years ago

It might be worthwhile asking about this on the mailing list. I have not had this much difficulty importing pools at boot but I know it has been discussed on the list.

I googled many. One of the other's fix I tried was fix. I could the error pointed in this thread, but still cannot fix my issues

humphreylee commented 3 years ago

I did not experience this issue. I did uncomment both "set -x" commands and captured a complete record of the install. If you decide to repeat the install, please do this so I can compare results and hopefully gain some insight into what is not working as desired.

Will do this in next re-install... probably this coming weekend. I most likely do the re-install if/after I find the fix to this issue. I had re-installed 4 times, but all ended almost same issue. My first attempt was install Debian with ext4 (/boot and /), then copy out the / into separate partitions, do zfs and copy back. 2nd attempt was install by hand each command line. 3rd attempt was push straight to Debian sid and last attempt was following (almost) blindly your script (of course change some local parameters). If I recall correctly, all ended same error.

When I dig around of current install, I found some zfs relics of previous install, e.g. hostname which I change in some installs. Can't remember which one now.. thus I asked about wipefs.

humphreylee commented 3 years ago

After many reboots, this is still the only thing I need to do -> zpool import -aN -d /dev/disk/by-id -o cachefile=none. Once this is done, I got normal Debian Bullseye.

HankB commented 3 years ago

If you can boot and get Bullseye running, you can collect the information I asked for above (without doing a reinstall.) You won't need the rescue procedure, just run the commands once up and running.

When you do, please close this issue and start a new one with the information linked and ask on the mailing list. I'm pretty sure there will be a fix simpler than a reinstall. I just don't know it.

On Mon, Sep 6, 2021 at 6:09 AM humphreylee @.***> wrote:

After many reboots, this is still the only thing I need to do -> zpool import -aN -d /dev/disk/by-id -o cachefile=none. Once this is done, I got normal Debian Bullseye.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/HankB/Linux_ZFS_Root/issues/17#issuecomment-913562667, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZM47UHWIAZDOKSQIN54DTUASOP3ANCNFSM5DH3QYTA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- '03 BMW F650CS - hers '98 Dakar K12RS - "BABY K" grew up. '93 R100R w/ Velorex 700 (MBD starts...) '95 Miata - "OUR LC" polish visor: apply squashed bugs, rinse, repeat Beautiful Sunny Winfield, Illinois

humphreylee commented 3 years ago

Thanks very much Hank for all the helps. Much appreciated. I will follow-up my case in zfsonlinux forum,