Issue with curtin-hooks not being used in custom MAAS image

I am using ubuntu-lvm.pkr.hcl to create a custom image for MAAS. I have added the following configuration to ubuntu-lvm.pkr.hcl:

$ tar -tvf packages/custom-packages.tar.gz ./
drwxr-xr-x _apt/root         0 2024-07-02 05:47 ./
-rw-r--r-- root/root  73435820 2023-11-17 04:59 ./linux-modules-extra-6.2.0-39-generic_6.2.0-39.40~22.04.1_amd64.deb
-rw-r--r-- root/root      1812 2023-11-17 04:59 ./linux-tools-6.2.0-39-generic_6.2.0-39.40~22.04.1_amd64.deb
-rw-r--r-- root/root  25848900 2023-11-17 04:59 ./linux-modules-6.2.0-39-generic_6.2.0-39.40~22.04.1_amd64.deb
-rw-r--r-- root/root  13078794 2023-11-17 04:59 ./linux-hwe-6.2-headers-6.2.0-39_6.2.0-39.40~22.04.1_all.deb
-rw-r--r-- root/root   8003858 2023-11-17 04:59 ./linux-hwe-6.2-tools-6.2.0-39_6.2.0-39.40~22.04.1_amd64.deb
-rw-r--r-- root/root  13581088 2023-11-17 04:59 ./linux-image-6.2.0-39-generic_6.2.0-39.40~22.04.1_amd64.deb
-rw-r--r-- root/root   3391852 2023-11-17 04:59 ./linux-headers-6.2.0-39-generic_6.2.0-39.40~22.04.1_amd64.deb

build {
  sources = ["source.qemu.lvm"]

  provisioner "file" {
    destination = "/tmp/curtin-hooks"
    source      = "${path.root}/scripts/curtin-hooks"
  }

  provisioner "file" {
    destination = "/tmp/install-custom-packages"
    source      = "${path.root}/scripts/install-custom-packages"
  }

  provisioner "file" {
    destination = "/tmp/setup-bootloader"
    source      = "${path.root}/scripts/setup-bootloader"
  }

  provisioner "file" {
    destination = "/tmp/custom-packages.tar.gz"
    source      = "${path.root}/packages/custom-packages.tar.gz"
  }

  provisioner "shell" {
    environment_vars  = ["HOME_DIR=/home/ubuntu", "http_proxy=${var.http_proxy}", "https_proxy=${var.https_proxy}", "no_proxy=${var.no_proxy}"]
    execute_command   = "echo 'ubuntu' | {{ .Vars }} sudo -S -E sh -eux '{{ .Path }}'"
    expect_disconnect = true
    scripts           = ["${path.root}/scripts/curtin.sh", "${path.root}/scripts/networking.sh", "${path.root}/scripts/cleanup.sh"]
  }

  post-processor "compress" {
    output = "custom-ubuntu-lvm.dd.gz"
  }
}

I have also added a verification step to ensure that the files are being placed under the /curtin directory with the appropriate permissions. However, during deployment, the curtin/curtin-hooks are not being used, and I am not getting the kernel I wanted.

scripts/curtin.sh

# Verification step
echo "Verifying files in /curtin:"
for s in curtin-hooks install-custom-packages setup-bootloader; do
  if [ -f "/curtin/$s" ]; then
    echo "Found: /curtin/$s"
    ls -l "/curtin/$s"
  else
    echo "Missing: /curtin/$s"
  fi
done

if [ -f /curtin/custom-packages.tar.gz ]; then
  echo "Found: /curtin/custom-packages.tar.gz"
  ls -l /curtin/custom-packages.tar.gz
else
  echo "Missing: /curtin/custom-packages.tar.gz"
fi

Log Output:

==> qemu.lvm: + mkdir -p /curtin
==> qemu.lvm: + [ -f /tmp/curtin-hooks ]
==> qemu.lvm: + mv /tmp/curtin-hooks /curtin/
==> qemu.lvm: + chmod 750 /curtin/curtin-hooks
==> qemu.lvm: + [ -f /tmp/install-custom-packages ]
==> qemu.lvm: + mv /tmp/install-custom-packages /curtin/
==> qemu.lvm: + chmod 750 /curtin/install-custom-packages
==> qemu.lvm: + [ -f /tmp/setup-bootloader ]
    qemu.lvm: Verifying files in /curtin:
    qemu.lvm: Found: /curtin/curtin-hooks
    qemu.lvm: -rwxr-x--- 1 ubuntu ubuntu 3910 Jul  2 06:40 /curtin/curtin-hooks
    qemu.lvm: Found: /curtin/install-custom-packages
==> qemu.lvm: + mv /tmp/setup-bootloader /curtin/
    qemu.lvm: -rwxr-x--- 1 ubuntu ubuntu 1390 Jul  2 06:40 /curtin/install-custom-packages
==> qemu.lvm: + chmod 750 /curtin/setup-bootloader
    qemu.lvm: Found: /curtin/setup-bootloader
==> qemu.lvm: + [ -f /tmp/custom-packages.tar.gz ]
==> qemu.lvm: + mv /tmp/custom-packages.tar.gz /curtin/
    qemu.lvm: -rwxr-x--- 1 ubuntu ubuntu 1285 Jul  2 06:40 /curtin/setup-bootloader
    qemu.lvm: Found: /curtin/custom-packages.tar.gz
==> qemu.lvm: + echo Verifying files in /curtin:
==> qemu.lvm: + [ -f /curtin/curtin-hooks ]
==> qemu.lvm: + echo Found: /curtin/curtin-hooks
==> qemu.lvm: + ls -l /curtin/curtin-hooks
==> qemu.lvm: + [ -f /curtin/install-custom-packages ]
==> qemu.lvm: + echo Found: /curtin/install-custom-packages
    qemu.lvm: -rw-r--r-- 1 ubuntu ubuntu 137198245 Jul  2 06:40 /curtin/custom-packages.tar.gz
==> qemu.lvm: + ls -l /curtin/install-custom-packages
==> qemu.lvm: + [ -f /curtin/setup-bootloader ]
==> qemu.lvm: + echo Found: /curtin/setup-bootloader
==> qemu.lvm: + ls -l /curtin/setup-bootloader
==> qemu.lvm: + [ -f /curtin/custom-packages.tar.gz ]
==> qemu.lvm: + echo Found: /curtin/custom-packages.tar.gz
==> qemu.lvm: + ls -l /curtin/custom-packages.tar.gz

Despite the verification showing that the files are present in /curtin with the correct permissions, the curtin-hooks are not being utilized during deployment. Any assistance on this issue would be appreciated.

Additionally, here are some commands and their outputs to provide more context that files are present in the /curtin directory:

$ gunzip custom-ubuntu-lvm.dd.gz
$ sudo parted custom-ubuntu-lvm.dd print
Model:  (file)
Disk /home/jenkins/custom-ubuntu-lvm.dd: 8590MB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name  Flags
 1      1049kB  565MB   564MB   fat32              boot, esp
 2      565MB   2444MB  1879MB  ext4
 3      2444MB  8589MB  6145MB
$ sudo fdisk -l custom-ubuntu-lvm.dd
Disk custom-ubuntu-lvm.dd: 8 GiB, 8589934592 bytes, 16777216 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 42174014-93D7-4034-B2BF-9BB9ED7AB79C

Device                  Start      End  Sectors  Size Type
custom-ubuntu-lvm.dd1    2048  1103871  1101824  538M EFI System
custom-ubuntu-lvm.dd2 1103872  4773887  3670016  1.8G Linux filesystem
custom-ubuntu-lvm.dd3 4773888 16775167 12001280  5.7G Linux filesystem
$ sudo losetup -fP custom-ubuntu-lvm.dd
$ sudo losetup -a
/dev/loop1: [66317]:22810508 (/var/lib/snapd/snaps/core22_1380.snap)
/dev/loop6: [66317]:26873580 (/home/jenkins/custom-ubuntu-lvm.dd)
/dev/loop4: [66317]:22810371 (/var/lib/snapd/snaps/snapd_21759.snap)
/dev/loop2: [66317]:22813936 (/var/lib/snapd/snaps/go_9949.snap)
/dev/loop0: [66317]:22813925 (/var/lib/snapd/snaps/core18_2823.snap)
/dev/loop5: [66317]:22810528 (/var/lib/snapd/snaps/core18_2829.snap)
/dev/loop3: [66317]:22814002 (/var/lib/snapd/snaps/lxd_28460.snap)
$ lsblk /dev/loop6
NAME      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
loop6       7:6    0    8G  0 loop 
├─loop6p1 259:17   0  538M  0 part 
├─loop6p2 259:18   0  1.8G  0 part 
└─loop6p3 259:19   0  5.7G  0 part 
$ sudo vgscan
$ sudo lvdisplay
  --- Logical volume ---
  LV Path                /dev/ubuntu-vg/ubuntu-lv
  LV Name                ubuntu-lv
  VG Name                ubuntu-vg
  LV UUID                G2bQMF-wEHH-Dquq-GmB7-3EgE-92sy-rllFyD
  LV Write Access        read/write
  LV Creation host, time ubuntu-server, 2024-07-01 07:06:40 +0000
  LV Status              NOT available
  LV Size                <5.72 GiB
  Current LE             1464
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
$ sudo lvdisplay
  --- Logical volume ---
  LV Path                /dev/ubuntu-vg/ubuntu-lv
  LV Name                ubuntu-lv
  VG Name                ubuntu-vg
  LV UUID                G2bQMF-wEHH-Dquq-GmB7-3EgE-92sy-rllFyD
  LV Write Access        read/write
  LV Creation host, time ubuntu-server, 2024-07-01 07:06:40 +0000
  LV Status              available
  # open                 0
  LV Size                <5.72 GiB
  Current LE             1464
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           252:0
$ sudo mount /dev/ubuntu-vg/ubuntu-lv /mnt/
$ ls -lrt
total 88
drwxr-xr-x  2 root root  4096 Apr 18  2022 sys
drwxr-xr-x  2 root root  4096 Apr 18  2022 proc
lrwxrwxrwx  1 root root     8 Feb 16 18:37 sbin -> usr/sbin
lrwxrwxrwx  1 root root    10 Feb 16 18:37 libx32 -> usr/libx32
lrwxrwxrwx  1 root root     9 Feb 16 18:37 lib64 -> usr/lib64
lrwxrwxrwx  1 root root     9 Feb 16 18:37 lib32 -> usr/lib32
lrwxrwxrwx  1 root root     7 Feb 16 18:37 lib -> usr/lib
lrwxrwxrwx  1 root root     7 Feb 16 18:37 bin -> usr/bin
drwxr-xr-x 14 root root  4096 Feb 16 18:37 usr
drwxr-xr-x  2 root root  4096 Feb 16 18:37 srv
drwxr-xr-x  2 root root  4096 Feb 16 18:37 opt
drwxr-xr-x  2 root root  4096 Feb 16 18:37 mnt
drwxr-xr-x  2 root root  4096 Feb 16 18:37 media
drwxr-xr-x  4 root root  4096 Feb 16 18:45 dev
drwxr-xr-x 13 root root  4096 Feb 16 18:46 var
drwxr-xr-x  6 root root  4096 Feb 16 18:52 snap
drwxr-xr-x 14 root root  4096 Feb 16 18:52 run
dr-xr-xr-x  2 root root  4096 Feb 16 23:52 cdrom
drwx------  2 root root 16384 Jul  1 07:06 lost+found
drwxr-xr-x  2 root root  4096 Jul  1 07:06 boot
drwxr-xr-x  3 root root  4096 Jul  1 07:09 home
drwx------  4 root root  4096 Jul  1 07:09 root
drwxr-xr-x 96 root root  4096 Jul  1 07:09 etc
drwxr-xr-x  2 root root  4096 Jul  1 07:09 curtin
drwxrwxrwt  9 root root  4096 Jul  1 07:10 tmp
$ cd curtin
$ ls -rlt
total 133996
-rwxr-x--- 1 ubuntu ubuntu      3910 Jul  2 06:40 curtin-hooks
-rwxr-x--- 1 ubuntu ubuntu      1390 Jul  2 06:40 install-custom-packages
-rwxr-x--- 1 ubuntu ubuntu      1285 Jul  2 06:40 setup-bootloader
-rw-r--r-- 1 ubuntu ubuntu 137198245 Jul  2 06:40 custom-packages.tar.gz

Ok I found the issue, here is how I got through

https://github.com/canonical/packer-maas/pull/247

@alexsander-souza I get this error now while deploying using maas while executing /curtin/setup-bootloaded [https://github.com/canonical/packer-maas/blob/main/ubuntu/scripts/setup-bootloader], any hint ?

    + grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=ubuntu --recheck
    Installing for x86_64-efi platform.
    grub-install: error: failed to get canonical path of `/boot/efi'.
    Running command ['udevadm', 'settle'] with allowed return codes [0] (capture=False)
    TIMED subp(['udevadm', 'settle']): 0.020
    Running command ['mount', '--make-private', '/tmp/tmpeh1eok2z/target/sys/firmware/efi/efivars'] with allowed return codes [0] (capture=False)
    Running command ['umount', '/tmp/tmpeh1eok2z/target/sys/firmware/efi/efivars'] with allowed return codes [0] (capture=False)
    Running command ['mount', '--make-private', '/tmp/tmpeh1eok2z/target/sys'] with allowed return codes [0] (capture=False)
    Running command ['umount', '/tmp/tmpeh1eok2z/target/sys'] with allowed return codes [0] (capture=False)
    Running command ['mount', '--make-private', '/tmp/tmpeh1eok2z/target/run'] with allowed return codes [0] (capture=False)
    Running command ['umount', '/tmp/tmpeh1eok2z/target/run'] with allowed return codes [0] (capture=False)
    Running command ['mount', '--make-private', '/tmp/tmpeh1eok2z/target/proc'] with allowed return codes [0] (capture=False)
    Running command ['umount', '/tmp/tmpeh1eok2z/target/proc'] with allowed return codes [0] (capture=False)
    Running command ['mount', '--make-private', '/tmp/tmpeh1eok2z/target/dev'] with allowed return codes [0] (capture=False)
    Running command ['umount', '/tmp/tmpeh1eok2z/target/dev'] with allowed return codes [0] (capture=False)
    Traceback (most recent call last):
      File "/tmp/tmpeh1eok2z/target/curtin/curtin-hooks", line 118, in <module>
        main()
      File "/tmp/tmpeh1eok2z/target/curtin/curtin-hooks", line 113, in main
        curthook(config, target, state)
      File "/tmp/tmpeh1eok2z/target/curtin/curtin-hooks", line 97, in curthook
        run_hook_in_target(target, "setup-bootloader")
      File "/tmp/tmpeh1eok2z/target/curtin/curtin-hooks", line 41, in run_hook_in_target
        in_chroot.subp(["/curtin/" + hook])
      File "/curtin/curtin/util.py", line 792, in subp
        return subp(*args, **kwargs)
      File "/curtin/curtin/util.py", line 280, in subp
        return _subp(*args, **kwargs)
      File "/curtin/curtin/util.py", line 144, in _subp
        raise ProcessExecutionError(stdout=out, stderr=err,
    curtin.util.ProcessExecutionError: Unexpected error while running command.
    Command: ['unshare', '--fork', '--pid', '--', 'chroot', '/tmp/tmpeh1eok2z/target', '/curtin/setup-bootloader']
    Exit code: 1

It appears that system does not have an EFI partition, which is required for UEFI booting. Instead, it seems to have a BIOS/MBR setup.

this is strange, the output of parted you posted before shows a EFI partition:

$ sudo parted custom-ubuntu-lvm.dd print
Model:  (file)
Disk /home/jenkins/custom-ubuntu-lvm.dd: 8590MB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name  Flags
 1      1049kB  565MB   564MB   fat32              boot, esp
 2      565MB   2444MB  1879MB  ext4
 3      2444MB  8589MB  6145MB

is it empty?

we also fixed an issue in the setup-bootloader script recently, please make sure you are running the latest version

j1-atos-5.redlab.datadirectnet.com-.zip

@alexsander-souza I attached Machine output (YAML), curtin logs and Installation output for your reference, I use iso image + lvm to create a maas image and I use latest packer maas code

any hints will be helpful

sda1 is the efi parition, but it not mounted or mounted as a different path as I get below error

grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=ubuntu --recheck
    Installing for x86_64-efi platform.
    grub-install: error: failed to get canonical path of `/boot/efi'.

Below helped but upon auto reboot after installation it didn't come up with network!!

mkdir -p /boot/efi
mount /dev/sda1 /boot/efi
grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=ubuntu --recheck Installing for x86_64-efi platform. Installation finished. No error reported.
update-initramfs -uk all update-initramfs: Generating /boot/initrd.img-6.2.0-39-generic
efibootmgr -v BootCurrent: 0004 Timeout: 5 seconds

@alexsander-souza another issue with lvm image, ubuntu--vg-ubuntu--lv not expanding to full disk size

ubuntu@j1-atos-5:~$ lsblk
NAME                      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
loop0                       7:0    0 63.9M  1 loop /snap/core20/2105
loop1                       7:1    0   87M  1 loop /snap/lxd/27037
loop2                       7:2    0 40.4M  1 loop /snap/snapd/20671
loop3                       7:3    0 38.8M  1 loop /snap/snapd/21759
loop4                       7:4    0 63.9M  1 loop /snap/core20/2318
loop5                       7:5    0   87M  1 loop /snap/lxd/28373
sda                         8:0    1  3.5T  0 disk 
├─sda1                      8:1    1  538M  0 part /boot/efi
├─sda2                      8:2    1  1.8G  0 part /boot
└─sda3                      8:3    1  5.7G  0 part 
  └─ubuntu--vg-ubuntu--lv 253:0    0  5.7G  0 lvm  /
ubuntu@j1-atos-5:~$ 

ubuntu@j1-atos-5:~$ sudo lvextend -l +100%FREE /dev/ubuntu-vg/ubuntu-lv
  New size (1464 extents) matches existing size (1464 extents).

I was able to manually fix it by doing the following but can we handle this during the deployment ?

ubuntu@j1-atos-5:~$ sudo parted /dev/sda
GNU Parted 3.4
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print                                                            
Warning: Not all of the space available to /dev/sda appears to be used, you can fix the GPT to use all of the space (an extra 7484699312 blocks) or continue with the current setting? 
Fix/Ignore? Fix                                                           
Model: ATA Micron_5300_MTFD (scsi)
Disk /dev/sda: 3841GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name  Flags
 1      1049kB  565MB   564MB   fat32              boot, esp
 2      565MB   2444MB  1879MB  ext4
 3      2444MB  8589MB  6145MB

(parted) resizepart 3 100%                                                
(parted) quit                                                             
Information: You may need to update /etc/fstab.

ubuntu@j1-atos-5:~$ sudo pvresize /dev/sda3                               
  Physical volume "/dev/sda3" changed
  1 physical volume(s) resized or updated / 0 physical volume(s) not resized
ubuntu@j1-atos-5:~$ sudo lvextend -l +100%FREE /dev/ubuntu-vg/ubuntu-lv
  Size of logical volume ubuntu-vg/ubuntu-lv changed from <5.72 GiB (1464 extents) to 3.49 TiB (915124 extents).
  Logical volume ubuntu-vg/ubuntu-lv successfully resized.
ubuntu@j1-atos-5:~$ sudo resize2fs /dev/ubuntu-vg/ubuntu-lv
resize2fs 1.46.5 (30-Dec-2021)
Filesystem at /dev/ubuntu-vg/ubuntu-lv is mounted on /; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 447
The filesystem on /dev/ubuntu-vg/ubuntu-lv is now 937086976 (4k) blocks long.

ubuntu@j1-atos-5:~$ lsblk
NAME                      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
loop0                       7:0    0 63.9M  1 loop /snap/core20/2105
loop1                       7:1    0   87M  1 loop /snap/lxd/27037
loop2                       7:2    0 40.4M  1 loop /snap/snapd/20671
loop3                       7:3    0 38.8M  1 loop /snap/snapd/21759
loop4                       7:4    0 63.9M  1 loop /snap/core20/2318
loop5                       7:5    0   87M  1 loop /snap/lxd/28373
sda                         8:0    1  3.5T  0 disk 
├─sda1                      8:1    1  538M  0 part /boot/efi
├─sda2                      8:2    1  1.8G  0 part /boot
└─sda3                      8:3    1  3.5T  0 part 
  └─ubuntu--vg-ubuntu--lv 253:0    0  3.5T  0 lvm  /
ubuntu@j1-atos-5:~$

canonical / packer-maas

Issue with curtin-hooks not being used in custom MAAS image #246