Closed davidcassany closed 2 years ago
scdbackup
commented
2 years ago Hi,
i am the developer of xorriso. Your xorriso run is supposed to keep the boot equipment of myimage.iso as is. So i wonder what goes wrong for you.
Be invited to discuss this on the bug-xorriso@gnu.org mailing list or in private with me via scdbackup@gmx.net . It would be helpful if i had the ISO image which is base of your experiment. Consider to give me a link.
Have a nice day :)
Thomas
davidcassany
commented
2 years ago i am the developer of xorriso. Your xorriso run is supposed to keep the boot equipment of myimage.iso as is. So i wonder what goes wrong for you.
@scdbackup it is really cool and appreciated you contacted us, thanks much. We create the EFI only ISO with the following command:
xorriso -volid COS_LIVE -padding 0 -outdev elemental.iso -map <path_to_root_tree> / -chmod 0755 --
-append_partition 2 0xef <path_to_efi_image_file>
-boot_image any cat_path=/boot/boot.catalog
-boot_image any cat_hidden=on
-boot_image any efi_path=--interval:appended_partition_2:all::
-boot_image any platform_id=0xef
-boot_image any appended_part_as=gpt
-boot_image any partition_offset=16and with the resulting image we call
xorriso -boot_image any keep -dev elemental.iso -map <pathtofile>/registration.yaml /livecd-cloud-config.yamlto include a new file and the image is created, however xorriso complains with
xorriso 1.4.6 : RockRidge filesystem manipulator, libburnia project.
xorriso : NOTE : Loading ISO image tree from LBA 0
xorriso : UPDATE : 12 nodes read in 1 seconds
libisofs: WARNING : Found hidden El-Torito image. Its size could not be figured out, so image modify or boot image patching may lead to bad results.
xorriso : NOTE : Detected El-Torito boot information which currently is set to be kept unchanged
Drive current: -dev 'elemental.iso'
Media current: stdio file, overwriteable
Media status : is written , is appendable
Boot record : El Torito , MBR protective-msdos-label cyl-align-off GPT
Media summary: 1 session, 307400 data blocks, 600m data, 51.5g free
Volume id : 'COS_LIVE'
xorriso : UPDATE : 1 files added in 1 seconds
Added to ISO image: file '/livecd-cloud-config.yaml'='<local_path>/livecd-cloud-config.yaml'
xorriso : NOTE : Keeping boot image unchanged
libisofs: NOTE : Cannot arrange content of data files in surely reproducible way
libisofs: FAILURE : Appended partition which shall serve as boot image does not exist
libisofs: MISHAP : File "BOOT CATALOG" can't be opened. Filling with 0s.
libisofs: NOTE : > Caused by: Trying to use an invalid file as boot image
ISO image produced: 30 sectors
Written to medium : 192 sectors at LBA 307424
Writing to 'elemental.iso' completed successfully.
xorriso : NOTE : -return_with SORRY 32 triggered by problem severity FAILURETo me it feels like there is some inconsistency in the former ISO creation. I am far from an expert regarding ISOs so I struggle to find the proper settings.
scdbackup
commented
2 years ago Hi,
David Cassany Viladomat wrote:
@scdbackup [github.com] it is really cool
I don't doubt that and do not try to lure people or discussions away from it. It's just that it's obviously about a potential xorriso problem and that i am used to discuss those via mail. My favorite workstation is old and does not play well with the buttons and menus of github web interface.
We create the EFI only ISO with the following command: xorriso ... append_partition 2 0xef
An ISO with appended partition is not really suitable for adding another session. At least the size of the ISO filesystem will get expanded over the size of its partition 1 and include the range of partition 2.
Therefore i would advise to let xorriso create a new ISO which uses the old one as base but will not change it. Regrettably your old xorriso version 1.4.6 is not able to perform the job properly unless you give it the original boot options and an EFI image on disk. See below at the end of this mail.
xorriso -boot_image any keep -dev elemental.iso -map
/registration.yaml /livecd-cloud-config.yaml
So my advice for the current release xorriso-1.5.4 translates to:
test -e new.iso && rm new.iso
xorriso -indev elemental.iso -outdev new.iso \
-map
Note that the -boot_image "replay" command has to be given after -indev. "keep" is a setting which may be defined before -indev or -dev. "replay" is an action which depends on what was detected during action -indev.
xorriso 1.4.6 : RockRidge filesystem manipulator, libburnia project.
That's causing nostalgic feelings in me. Released 6 years ago when the world was quite a different one ... There were bugs in -boot_image "replay" which only showed up, wenn the distro packagers began to give up the old isohybrid layout of ISOLINUX.
Consider to get the current release 1.5.4 as standalone source tarball https://www.gnu.org/software/xorriso/xorriso-1.5.4.pl02.tar.gz or the development snapshot https://www.gnu.org/software/xorriso/xorriso-1.5.5.tar.gz
Installation instructions are in https://www.gnu.org/software/xorriso/README_xorriso Essentially:
cd ...somewhere... tar xzf xorriso-1.5.4.pl02.tar.gz cd xorriso-1.5.4 ./configure --prefix=/usr make
First sign of life would then be the output of this run
xorriso/xorriso -version
which should say something like GNU xorriso 1.5.4.pl02 : RockRidge filesystem manipulator, libburnia project. ... Version timestamp : 2021.02.06.123001 Build timestamp : -none-given- libisofs in use : 1.5.4 (min. 1.5.4) libjte in use : 2.0.0 (min. 2.0.0) libburn in use : 1.5.4 (min. 1.5.4) libburn OS adapter: internal GNU/Linux SG_IO adapter sg-linux libisoburn in use : 1.5.4 (min. 1.5.4) License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html. ...
The listed libraries are statically linked into the xorriso binary and will not interfere with the locally installed older lib.so. . You may run the binary from where it was built
...somewhere.../xorriso-1.5.4/xorriso/xorriso
or you may replace the local older /usr/bin/xorriso by it. A classic make install as superuser is possible, too. You'd first have to uninstall the older xorriso, in order not to confuse your distro's package management. Vice versa you'd need to uninstall GNU xorriso before letting your package management install its own xorriso package.
libisofs: FAILURE : Appended partition which shall serve as boot image does not exist libisofs: MISHAP : File "BOOT CATALOG" can't be opened. Filling with 0s. libisofs: NOTE : > Caused by: Trying to use an invalid file as boot image
Yep. I get these too with xorriso-1.4.6 and ubuntu-22.04-desktop-amd64.iso as base ISO:
cp ubuntu-22.04-desktop-amd64.iso test.iso xorriso -boot_image any keep -dev test.iso \ -map /...some.path... /...some.path... \
The result still has a recognizable EFI partition for booting from USB stick but a damaged El Torito boot catalog for booting from CD/DVD/BD media. (I did not check whether content of the EFI partition is undamaged. Chances are not too bad, though.)
The same run with xorriso-1.5.4 reports no FAILURE or MISHAP errors. Inspection of the resulting ISO by xorriso-1.5.4 looks encouraging:
xorriso -indev test.iso -report_system_area plain -report_el_torito plain
yields
... MBR partition table: N Status Type Start Blocks MBR partition : 1 0x00 0xee 1 7138587 ... GPT partition name : 2 41007000700065006e006400650064003200 GPT partname local : 2 Appended2 GPT partition GUID : 2 b8b29da0f6b5ae43afb191e0a90189a1 GPT type GUID : 2 28732ac11ff8d211ba4b00a0c93ec93b GPT partition flags: 2 0x0000000000000000 GPT start and size : 2 7129428 8496 ... El Torito images : N Pltf B Emul Ld_seg Hdpt Ldsiz LBA El Torito boot img : 1 BIOS y none 0x0000 0x00 4 697 El Torito boot img : 2 UEFI y none 0x0000 0x00 8496 1782357 El Torito img path : 1 /boot/grub/i386-pc/eltorito.img El Torito img opts : 1 boot-info-table grub2-boot-info El Torito img blks : 2 2124
So xorriso still recognizes the GPT and the EFI partition in it. The partition's range is also recognized as El Torito EFI boot image (ISO 9660 LBA is in blocks of 2048 bytes, GPT counts in blocks of 512.) fdisk still recognizes GPT and EFI partition:
/sbin/fdisk -l test.iso
yields
GPT PMBR size mismatch (7138587 != 7140095) will be corrected by w(rite).
Disk test.iso: 3.4 GiB, 3655729152 bytes, 7140096 sectors ... Disklabel type: gpt ... Device Start End Sectors Size Type test.iso1 64 7129427 7129364 3.4G Microsoft basic data test.iso2 7129428 7137923 8496 4.2M EFI System test.iso3 7137924 7138523 600 300K Microsoft basic data
The protest about size is mainly a consequence of adding a session to an ISO with appended partition. Smaller size deviations stem from zero padding at the end of the ISO image file.
Above proposal to create a new ISO from the old one by
test -e new.iso && rm new.iso xorriso -indev ubuntu-22.04-desktop-amd64.iso -outdev new.iso \ -map /...some.path... /...some.path... \ -boot_image any replay
yields a result that causes slightly different numbers in the report:
MBR partition table: N Status Type Start Blocks MBR partition : 1 0x00 0xee 1 7136955 ... GPT partition name : 2 41007000700065006e006400650064003200 GPT partname local : 2 Appended2 GPT partition GUID : 2 8c780e0b5f985345bb0d15c582b0c36a GPT type GUID : 2 28732ac11ff8d211ba4b00a0c93ec93b GPT partition flags: 2 0x0000000000000000 GPT start and size : 2 7128396 8496 ... El Torito images : N Pltf B Emul Ld_seg Hdpt Ldsiz LBA El Torito boot img : 1 BIOS y none 0x0000 0x00 4 436 El Torito boot img : 2 UEFI y none 0x0000 0x00 8496 1782099 El Torito img path : 1 /boot/grub/i386-pc/eltorito.img El Torito img opts : 1 boot-info-table grub2-boot-info El Torito img blks : 2 2124
fdisk is still complaining about a smaller mismatch of partitioned size and image file size. This comes from padding up to full blocks of 64 KiB and from the default padding to protect the ISO's end from the Linux TAO CD Readahead bug.
Nevertheless, in this case the ISO filesystem does not reach over the end of its partition 1 and there is no meaningful data after partition 2. So i deem it better than the result of -boot_image "keep".
As stated above, xorriso-1.4.6 has bugs with -boot_image "replay" on appended partitions. If you need to stay with that version, try:
test -e new.iso && rm new.iso
xorriso -indev elemental.iso -outdev new.iso \
-map
If you don't have
dd if=ubuntu-22.04-desktop-amd64.iso bs=512 skip=7129428 count=8496 \
of='
Have a nice day :)
Thomas
davidcassany
commented
2 years ago @scdbackup thanks much for the detailed explanation and I just confirmed that using Xorriso 1.5.4 works as expected, I also removed the partitions from my ISO as I noted they are no longer necessary, I think they just stayed there from a time we attempted to build BIOS & EFI capable images bootable from a block storage.
scdbackup
commented
2 years ago Hi,
i sent this by mail yesterday but it does not show up on the web page:
David Cassany Viladomat wrote:
I also removed the partitions from my ISO as I noted they are no longer necessary, I think they just stayed there from a time we attempted to build BIOS & EFI capable images bootable from a block storage.
EFI specs demand an EFI partition with VFAT filesystem for booting from USB stick or hard disk. From CD, DVD, or BD media it is supposed to work via an El Torito catalog entry which advertises a VFAT filesystem image as boot image. Usually in a modern bootable ISO, partition table and El Torito catalog mark the same storage area for EFI. Your shown xorriso run does this by
-append_partition 2 0xef
The virtual machine EFI OVMF is known to be too liberal by booting via El Torito from virtual hard disk and by partition table from virtual DVD. So better do not take success on a virtual machine as reliable prediction of success on real hardware.
Have a nice day :)
Thomas
davidcassany
commented
2 years ago After several tests and the insides discussed here it is clear that safe way to go is rewriting the ISO with xorriso. This can simply be done with something like:
time xorriso -indev elemental.iso -outdev elemental_modified.iso -map overlay/livecd-cloud-config.yaml /livecd-cloud-config.yaml -boot_image any replay
xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project.
xorriso : NOTE : Loading ISO image tree from LBA 0
xorriso : UPDATE : 12 nodes read in 1 seconds
libisofs: NOTE : Found hidden El-Torito image for EFI.
libisofs: NOTE : EFI image start and size: 307386 * 2048 , 8192 * 512
xorriso : NOTE : Detected El-Torito boot information which currently is set to be discarded
Drive current: -indev 'elemental.iso'
Media current: stdio file, overwriteable
Media status : is written , is appendable
Boot record : El Torito , MBR protective-msdos-label cyl-align-off GPT
Media summary: 1 session, 309450 data blocks, 604m data, 297g free
Volume id : 'COS_LIVE'
Drive current: -outdev 'elemental_modified.iso'
Media current: stdio file, overwriteable
Media status : is blank
Media summary: 0 sessions, 0 data blocks, 0 data, 297g free
xorriso : UPDATE : 1 files added in 1 seconds
Added to ISO image: file '/livecd-cloud-config.yaml'='/home/deivit/work/elemental-cli/overlay/livecd-cloud-config.yaml'
xorriso : NOTE : Replayed 16 boot related commands
xorriso : NOTE : Copying to System Area: 32768 bytes from file '--interval:imported_iso:0s-15s:zero_mbrpt,zero_gpt:elemental.iso'
xorriso : UPDATE : Writing: 16384s 5.3% fifo 100% buf 50%
xorriso : UPDATE : Writing: 90592s 29.3% fifo 99% buf 50% 121.7xD
xorriso : UPDATE : Writing: 159840s 51.6% fifo 99% buf 50% 102.2xD
xorriso : UPDATE : Writing: 229376s 74.1% fifo 100% buf 50% 102.6xD
xorriso : UPDATE : Writing: 298464s 96.4% fifo 100% buf 50% 102.0xD
ISO image produced: 309403 sectors
Written to medium : 309584 sectors at LBA 48
Writing to 'elemental_modified.iso' completed successfully.
real 0m5.035s
user 0m1.851s
sys 0m2.333sThe new ISO was created with 5seconds on my laptop (not high end box). So I believe the way to proceed to actually release ISOs and adapt our elemental-build-iso script to actually download a release and run the above command to produce a new one adding the custom yaml file.
Closing this card and creating specific follow up issues
scdbackup
commented
2 years ago Hi,
David Cassany Viladomat wrote:
I also removed the partitions from my ISO as I noted they are no longer necessary, I think they just stayed there from a time we attempted to build BIOS & EFI capable images bootable from a block storage.
EFI specs demand an EFI partition with VFAT filesystem for booting from USB stick or hard disk. From CD, DVD, or BD media it is supposed to work via an El Torito catalog entry which advertises a VFAT filesystem image as boot image. Usually in a modern bootable ISO, partition table and El Torito catalog mark the same storage area for EFI. Your shown xorriso run does this by
-append_partition 2 0xef
The virtual machine EFI OVMF is known to be too liberal by booting via El Torito from virtual hard disk and by partition table from virtual DVD. So better do not take success on a virtual machine as reliable prediction of success on real hardware.
Have a nice day :)
Thomas
I am wondering if it is somehow possible to inject a registration yaml to an already existing ISO, so we could release ISOs and prevent users of having to build its own ISO.
Adds the yaml file into the ISO however, it breaks the boot image somehow. Boot image options in xorriso are endless and I am wondering to what extend it is possible to append a fine keep the boot bits alive, so the ISO is still bootable after appending some extra bits.
If we manage to achieve such a thing then it would be simple to relay on released ISOs in order to bootstrap clusters and make the ISO creation process way lighter.