laurent85v / archuseriso

A set of bash script programs to create bootable Arch Linux images and to create bootable USB flash drives with a desktop environment.
GNU General Public License v3.0
223 stars 30 forks source link

it's not possible to boot via network #57

Closed 0petrik closed 2 hours ago

0petrik commented 3 months ago

Dear laurent85v, please check the program for the next issue. net_boot_error

When I'm creating iso file using last two releases of archuseriso (0.8.3 and 0.8.4) it's not possible to boot via network. Last work release is 0.8.2. For boot I use my own iPXE server and iVentoy. The error mesage attached. Please, fix it.

Thanks in advance.

laurent85v commented 3 months ago

This issue is not related to Archuseriso. Maybe a recent package update broke pxe booting. I tried booting Arch Linux latest iso and Manjaro latest iso over the network using iventoy and faced the same error.

0petrik commented 3 months ago

But why when I use Archuseriso 0.8.2 and latest packages everything Ok and it possible to boot via network? I created Arch Linux iso on Friday booted via network without problem. Maybe you're right and some dependency does't work correctly with Archuseriso (0.8.3 and 0.8.4) but work correctly with Archuseriso 0.8.2. We'll see it in the future.

laurent85v commented 3 months ago

But why when I use Archuseriso 0.8.2 and latest packages everything Ok and it possible to boot via network? I created Arch Linux iso on Friday booted via network without problem.

I didn't mentioned that I also created two iso images using version 0.8.2 for the Console and Xfce profiles. Both don't work and fail with the exact same error message in first post.

It seems this bug only affects Arch-based distros, other distributions iso boot correctly using iventoy. I suspect a recent package update in the Arch repositories broke pxe booting via iventoy.

Can you provide the profile you used for creating your iso image using version 0.8.2 that works for you ?

0petrik commented 3 months ago

I also use the Console and Xfce profiles, I'll make the iso files today then check them and lat you know the results. For me more important to boot via iPXE server then via iVentoy, but I'll check both options.

0petrik commented 3 months ago

I've checked boot via iPXE server is Ok. Boot via iVentoy fail . I've used my own profiles 0.8.2 for the Console and Xfce . Also I created two iso images using version 0.8.4 for the Console and Xfce profiles and tested them on iPXE server and on iVentoy. Both don't work and fail on iVentoy. On iPXE server Console-iso work, Xfce-iso doesn't work and fail. Do you still need my profiles 0.8.2 ? They work only on iPXE server.

laurent85v commented 3 months ago

I set up a pxe server and a client virtual machine to do some testing.

I can correctly boot the latest Xfce iso available online generated by version 0.8.4. My first attempt failed because the pxe client ran out of memory. I could successfully boot after increasing the virtual machine ram size.

Can you check you have enough ram on the client for your profile ?

0petrik commented 3 months ago

Yes I have enough ram on the clients, these are servers and each has up to 768 Gb of ram Did you set up a pxe server or ipxe one? These are slightly different technologies.

laurent85v commented 3 months ago

Today I tried pxe boot and ipxe boot, both worked. I can try to boot your iso or your profile if available.

0petrik commented 2 months ago

I tried default profile without any changes and xfce iso by this link http://dl.gnutux.fr/archuseriso/iso/ I've been busy lately and haven't had time for tests. Mey be the problem in my ipxe loader. I'll check it on the next week and let you know.

0petrik commented 2 months ago

I tried xfce iso by this link http://dl.gnutux.fr/archuseriso/iso/aui-xfce-linux_6_9_6-0623-x64.iso. On my iPXE server it doesn't work and fail. Also I created an iso image using version 0.8.2 for the Xfce profiles and I could successfully boot. May be problem in my iPXE server. Here is my ipxe menu:

!ipxe

console --x 1024 --y 768 console --picture http://192.168.136.70/logo.png

set url http://192.168.136.70/ set iso ${url}iso/ set linux ${url}linux/ set kernel ${linux}archiso/arch/boot/x86_64/ set ucode ${linux}archiso/arch/boot/ set initrd ${linux}archiso/arch/boot/x86_64/ set kernel2 ${linux}archiso2/arch/boot/x86_64/ set ucode2 ${linux}archiso2/arch/boot/ set initrd2 ${linux}archiso2/arch/boot/x86_64/ set extrabootoptions ip=dhcp net.ifnames=0 BOOTIF=01-${netX/mac}

:MENU menu item --gap -- *** item exit Exit iPXE and continue BIOS boot item --gap -- --------------- iPXE Start Menu --------------- item arch_gui Arch Linux GUI item arch_con Arch Linux Console item clonezilla Clonezilla item reboot Reboot item arch_gui_test Arch Linux GUI Test

item --gap -- ----------------------------------------------- choose target && goto ${target}

:arch_gui kernel ${kernel}vmlinuz-linux initrd ${ucode}amd-ucode.img initrd ${ucode}intel-ucode.img initrd ${initrd}initramfs-linux.img imgargs vmlinuz-linux initrd=amd-ucode.img initrd=intel-ucode.img initrd=initramfs-linux.img archiso_http_srv=${linux}archiso/ archisobasedir=arch loglevel=7 cms_verify=n ${extrabootoptions}

boot

:arch_con dhcp initrd ${iso}cons.iso chain utilits/memdisk iso raw exit

:clonezilla dhcp initrd ${iso}clonezilla.iso chain utilits/memdisk iso raw exit

:arch_gui_test kernel ${kernel2}vmlinuz-linux initrd ${ucode2}amd-ucode.img initrd ${ucode2}intel-ucode.img initrd ${initrd2}initramfs-linux.img imgargs vmlinuz-linux initrd=amd-ucode.img initrd=intel-ucode.img initrd=initramfs-linux.img archiso_http_srv=${linux}archiso2/ archisobasedir=arch loglevel=7 cms_verify=n ${extrabootoptions}

boot

:reboot reboot

:exit exit

Please, check my menu, thanks.

laurent85v commented 2 months ago

Since version 0.8.3 mkinitcpio's microcode hook is used instead of external microcode images. mkinitcpio now supports packing the microcode initramfs together with the main initramfs into one file. Remove the lines and boot parameters related to ucode.

#!ipxe

console --x 1024 --y 768
console --picture http://192.168.136.70/logo.png

set url http://192.168.136.70/
set iso ${url}iso/
set linux ${url}linux/
set kernel ${linux}archiso/arch/boot/x86_64/
set initrd ${linux}archiso/arch/boot/x86_64/
set kernel2 ${linux}archiso2/arch/boot/x86_64/
set initrd2 ${linux}archiso2/arch/boot/x86_64/
set extrabootoptions ip=dhcp net.ifnames=0 BOOTIF=01-${netX/mac}

:MENU
menu
item --gap -- ***********************************
item exit Exit iPXE and continue BIOS boot
item --gap -- --------------- iPXE Start Menu ---------------
item arch_gui Arch Linux GUI
item arch_con Arch Linux Console
item clonezilla Clonezilla
item reboot Reboot
item arch_gui_test Arch Linux GUI Test

item --gap -- -----------------------------------------------
choose target && goto ${target}

:arch_gui
kernel ${kernel}vmlinuz-linux
initrd ${initrd}initramfs-linux.img
imgargs vmlinuz-linux initrd=initramfs-linux.img archiso_http_srv=${linux}archiso/ archisobasedir=arch loglevel=7 cms_verify=n ${extrabootoptions}

boot

:arch_con
dhcp
initrd ${iso}cons.iso
chain utilits/memdisk iso raw
exit

:clonezilla
dhcp
initrd ${iso}clonezilla.iso
chain utilits/memdisk iso raw
exit

:arch_gui_test
kernel ${kernel2}vmlinuz-linux
initrd ${initrd2}initramfs-linux.img
imgargs vmlinuz-linux initrd=initramfs-linux.img archiso_http_srv=${linux}archiso2/ archisobasedir=arch loglevel=7 cms_verify=n ${extrabootoptions}

boot

:reboot
reboot

:exit
exit
0petrik commented 2 months ago

thanks a lot I'll check and let you know

0petrik commented 2 months ago

Cool! Great! I'm happy! It works, million thanks. I have one question. Is it possible to make a program like Archuseriso, witch can create an iso file from my Current Installed System on hdd or usb without downloading and installing the packages from internet just to turn my Current System to a boot iso image? Because it takes a lot of time to customize, add changes and make an iso file after each update.

laurent85v commented 2 months ago

What are your more specific needs? You can customize an existing archuseriso profile.

0petrik commented 2 months ago

Yes, I can customize an existing archuseriso profile and I do it. But it takes a lot of time to customize, add changes, make an iso and not very convenient. Also I have the packages from AUR it also takes a lot of time to assemble the packages and add them to local repo. About my specific needs: I have a boot usb stick with installed ArchLinux. The system is installed on this usb. It's the same like installed ArchLinux on your PC on hdd, ssd or nvme. When I boot from this usb I can do any changes like update the system, create files and all changes will be saved. Usb has the portions (uefi, boot and root). At the input the program receives this partitions (uefi, boot and root) and makes a boot iso. Something like this program "penguins-eggs". https://penguins-eggs.net/ https://www.linuxuprising.com/2023/03/how-to-turn-your-current-system-to.html I tried penguins-eggs two times and got errors and no an iso file.

laurent85v commented 2 months ago

I see. The idea looks interesting. I managed to manually turn an Arch Linux system installed on a hard drive into a bootable iso image. I will look into this to automate the process. A bootable disk image instead of an ISO image should work as well even for ipxe booting and easier to maintain.

0petrik commented 2 months ago

Thank you. Please, let me know the name of program when it will be ready.

0petrik commented 1 month ago

If you need help to test the program, I'm ready. Also I want to add, it will be good if the program gives the choices:

  1. From which directory make the iso file. For example the program installed to PC but I want to make the iso file from USB.
  2. I can manager "work" directory (like in Archuseriso).
  3. I can manager "out" directory (like in Archuseriso).
laurent85v commented 1 month ago

I am working on it. The program will integrate Archuseriso.

The system running the program and the system to be compressed into a bootable disk image must be different.

The required argument of the program is the path to the root filesystem, which means you will need to mount the root filesystem yourself under a mount point of your choice before running the program. Working directory and output directory are command options.

The resulting disk image is a gpt disk image suitable for an external usb disk or thumb drive, not for dvds. Will also work on a server for ipxe booting.

0petrik commented 1 month ago

thank you, I'm looking forward to it.

laurent85v commented 1 month ago

A draft version of the program aui-hd2aui is ready for testing in the draft branch.

aui-hd2aui command line help:

$ aui-hd2aui --help
aui-hd2aui: Compress an Arch Linux system to a bootable disk image.

Synopsis:
aui-hd2aui [options] <path to root filesystem>

Options:
-h, --help                  Command line help
-w, --workdir               Set the Working directory
-o, --outdir                Set the output directory
--config                    Path to hd2aui configuration files (default: /usr/share/archuseriso/hd2aui)

To test the program, clone the Archuseriso repository and switch to the draft branch.

git clone https://github.com/laurent85v/archuseriso.git
cd archuseriso
git checkout -b draft origin/draft

Before running the program, the root filesystem to be compressed into a bootable disk image must be mounted under a mount point, for example under the mount point /mnt/rootfs The working and output directories are command options.

sudo ./archuseriso/aui-hd2aui  --config=./hd2aui /mnt/rootfs

The resulting disk image is a bootable GPT disk image suitable for external usb drives or thumb drives, but not DVDs. I also tested iPXE booting, it works.

The gpt disk image partitions layout:

1 is the efi and boot partition, vfat filesystem.

2 is the partition containing the squashfs disk image, ext4 filesystem.

3 is the copy on write partition for data persistence, ext4 filesystem.

To write the image for example on an external usb disk identified as /dev/disk/by-id/usb-Samsung_PSSD_T7_Shield_xxxxxxxxxxxxxxx-0:0 use:

# cat ./out/hd2aui-0812-xxx.img > /dev/disk/by-id/usb-Samsung_PSSD_T7_Shield_xxxxxxxxxxxxxxx-0:0

The identifier /dev/disk/by-id/... avoids to overwrite by mistake an internal disk by using the classical identifiers /dev/sdX

For ipxe booting copy the contents of partitions #1 and #2 to the destination directory of the ipxe server.

0petrik commented 4 weeks ago

I tried but couldn't make an image. Below there are all my actions. these are parts of my usb stick sdc
├─sdc1
├─sdc2 f2fs 1.15 arch-usb 669a3032-f8be-4daa-8380-9124297fc2f6
├─sdc3 ext4 1.0 boot c816599b-497b-45d2-a955-cd56d51afce3
└─sdc4 vfat FAT32 uefi 702A-7CE2
mount USB stick zbook:~ gwaing $ sudo mount /dev/sdc2 /mnt/rootfs zbook:~ gwaing $ sudo mount /dev/sdc3 /mnt/rootfs/boot zbook:~ gwaing $ sudo mount /dev/sdc4 /mnt/rootfs/boot/efi

zbook:~ gwaing $ cd archuseriso zbook:archuseriso gwaing $ git checkout -b draft origin/draft fatal: a branch named 'draft' already exists zbook:archuseriso gwaing $ sudo ./archuseriso/aui-hd2aui --config=./hd2aui /mnt/rootfs [sudo] пароль для gwaing:

[aui-hd2aui] INFO: Path to root filesystem: /mnt/rootfs

Confirm operation (N/y)? y [aui-hd2aui] INFO: Preparing the overlay filesystem [aui-hd2aui] INFO: Done! [aui-hd2aui] INFO: Configuring overlay filesystem for live startup install: failed to perform stat for '/home/gwaing/archuseriso/auiwork.TY9MpcN4/overlay/usr/lib/modules/6.10.5.arch1-1/vmlinuz': There is no such file or catalog umount: /home/gwaing/archuseriso/auiwork.TY9MpcN4/overlay: target is busy.

It's strange because I have this file ./home/gwaing/archuseriso/auiwork.TY9MpcN4/overlay/usr/lib/modules/6.10.5.arch1-1/vmlinuz' see the picture

archuseriso

laurent85v commented 3 weeks ago

Issue fixed, draft branch updated. Delete the local git repository archuseriso before testing again.

rm -rf archuseriso
0petrik commented 2 weeks ago

I did the same like the first time and I have a problem again. zbook:archuseriso gwaing $ sudo ./archuseriso/aui-hd2aui --config=./hd2aui /mnt/rootfs install: cannot create directory ‘’: No such file or directory I deleted the local git repository archuseriso and downloaded new one. What I do the wrong?

thanks

laurent85v commented 2 weeks ago

Thanks for reporting. It's a bug when workdir and outdir are not set. I uploaded a fix, you can retry again with the new version. Delete the local archuseriso git repository before trying again.

0petrik commented 2 weeks ago

I'm sorry for being pain in the butt but I have a trouble still . Below there are all my actions.

These are the mount points sdc 8:32 1 7,4G 0 disk
├─sdc1 8:33 1 10M 0 part
├─sdc2 8:34 1 6,8G 0 part /home/gwaing/archuseriso/auiwork.onL9N40j/rootfsbind │ /mnt/rootfs ├─sdc3 8:35 1 457M 0 part /mnt/rootfs/boot └─sdc4 8:36 1 99M 0 part /mnt/rootfs/boot/efi

zbook:~ gwaing $ sudo mount /dev/sdc2 /mnt/rootfs [sudo] пароль для gwaing: zbook:~ gwaing $ sudo mount /dev/sdc3 /mnt/rootfs/boot zbook:~ gwaing $ sudo mount /dev/sdc4 /mnt/rootfs/boot/efi zbook:~ gwaing $ git clone https://github.com/laurent85v/archuseriso.git Клонирование в «archuseriso»... remote: Enumerating objects: 14143, done. remote: Counting objects: 100% (1560/1560), done. remote: Compressing objects: 100% (129/129), done. remote: Total 14143 (delta 1467), reused 1467 (delta 1427), pack-reused 12583 (from 1) Получение объектов: 100% (14143/14143), 1.80 МиБ | 4.68 МиБ/с, готово. Определение изменений: 100% (8674/8674), готово. zbook:~ gwaing $ cd archuseriso zbook:archuseriso gwaing $ git checkout -b draft origin/draft branch 'draft' set up to track 'origin/draft'. Переключились на новую ветку «draft» zbook:archuseriso gwaing $ sudo ./archuseriso/aui-hd2aui --config=./hd2aui /mnt/rootfs

[aui-hd2aui] INFO: Path to root filesystem: /mnt/rootfs

Confirm operation (N/y)? y [aui-hd2aui] INFO: Preparing the overlay filesystem [aui-hd2aui] INFO: Done! [aui-hd2aui] INFO: Configuring overlay filesystem for live startup [aui-hd2aui] INFO: Done! [aui-hd2aui] INFO: Installing necessary packages to overlay filesystem [aui-hd2aui] INFO: Done! [aui-hd2aui] INFO: Configuring boot files [aui-hd2aui] INFO: Done! [aui-hd2aui] INFO: Filesystem cleanup [aui-hd2aui] INFO: Done! [aui-hd2aui] INFO: Creating the compressed filesystem image [/ ] 0/1073743068 0% [- ] 0/1073743068 0% [\ ] 0/1073743068 0% [| ] 0/1073743068 0% [- ] 0/1073743068 0% [| ] 0/1073743068 0% [/ ] 0/1073743068 0% [- ] 0/1073743068 0% [\ ] 0/1073743068 0% [| ] 0/1073966352 0% [- ] 1094/1073967037 0% [\ ] 1322/1073967037 0% [| ] 1347/1073967037 0% [/ ] 1347/1073967037 0% [- ] 1347/1073967037 0% [\ ] 1347/1073967037 0% [| ] 1347/1073967037 0% [/ ] 1347/1073967037 0% [- ] 1347/1073967037 0% [\ ] 1347/1073967037 0% [| ] 1347/1073967037 0% [/ ] 1347/1073967037 0% [- ] 1347/1073967037 0% [\ ] 1347/1073967037 0% [| ] 1347/1073967037 0% [/ ] 1347/1073967037 0% [- ] 1347/1073967037 0% [\ ] 1347/1073967037 0% [| ] 1347/1073967037 0% [/ ] 1347/1073967037 0% [- ] 1347/1073967037 0% [\ ] 1347/1073967037 0% [| ] 1347/1073967037 0% [/ ] 1347/1073967037 0% [- ] 1347/1073967037 0% [\ ] 1347/1073967037 0% [| ] 1347/1073967037 0% [/ ] 1347/1073967037 0% [- ] 1347/1073967037 0% [\ ] 1347/1073967037 0% [| ] 1347/1073967037 0% [/ ] 1347/1073967037 0% [- ] 1347/1073967037 0% [\ ] 1347/1073967037 0% [| ] 1347/1073967037 0% [/ ] 1347/1073967037 0% [- ] 1347/1073967037 0% [\ ] 1347/1073967037 0% [| ] 1347/1073967037 0% [/ ] 1347/1073967037 0% [- ] 1347/1073967037 0% [\ ] 1347/1073967037 0% [| ] 1347/1073967037 0% [/ ] 1347/1073967037 0% [- ] 1357/1073967037 0% [==============/ ] 200403189/1073967037 18% [aui-hd2aui] ERROR: Making squashfs image failed! Aborting. umount: /home/gwaing/archuseriso/auiwork.onL9N40j/overlay: target is busy. zbook:archuseriso gwaing $

laurent85v commented 2 weeks ago

What you did seems correct.

1347/1073967037 0% [- ] 1357/1073967037 0% [==============/ ] 200403189/1073967037 18% [aui-hd2aui] ERROR: Making squashfs image failed! Aborting. umount: /home/gwaing/archuseriso/auiwork.onL9N40j/overlay: target is busy. zbook:archuseriso gwaing $

The numbers 200403189/1073967037 of the mksquashfs progress bar are insane! That does not look consistent. My tests only show numbers under 120 000 for a basic archlinux installed on an external usb disk. It is several thousand times more in your case.

Try creating a squashfs image from the command line to get more details.

cd ~
sudo mount /dev/sdc2 /mnt/rootfs
sudo mksquashfs /mnt/rootfs ./airootfs.sfs -noappend -comp zstd

Can you upload to a pastebin the debug output of the aui-hd2aui program like this:

sudo bash -x ./archuseriso/aui-hd2aui --config=./hd2aui /mnt/rootfs 2>&1 | tee output.txt
curl -F 'file=@-' 0x0.st < output.txt

Then share the link resulting from the curl command.

0petrik commented 5 days ago

I did an image, everything fine and work. Boot via iPXE server also is Ok. Thanks. How to install the program on PC permanently? Is it possible to get ISO file instead of IMG? ISO file more useful and easy in use.

laurent85v commented 3 days ago

Thank you for your feedback.

How to install the program on PC permanently?

Wait for the next version of Archuseriso.

Is it possible to get ISO file instead of IMG?

This should work too, but it requires more coding and is harder to maintain due to the limitations of the iso format. Most people use USB devices for live booting, the iso format was designed for optical discs.

laurent85v commented 2 hours ago

New version released, including new aui-hd2aui program.