Open viraptor opened 1 year ago
Looking into this a bit deeper:
cloud-localds.iso (working)
hdiutil.iso (failing)
mkisofs.iso (failing)
Edit: Adding the meta-data
file didn't help.
Success! I tried a few flags and this was failing:
/usr/bin/hdiutil makehybrid -o /tmp/packer3596932755.iso -hfs -joliet -iso -default-volume-name cidata /tmp/packer_to_cdrom670900126
But this works:
/usr/bin/hdiutil makehybrid -o $3 -joliet -iso -default-volume-name cidata $9
It looks like adding -hfs
trips up the cloud image... which may make sense in case the module is not loaded at the time? Or some other reason - I didn't dig far enough.
I'll see if I can find the right options for mkisofs
as well, but that's not today. (it may just need a -no-hfs
)
Ping @lbajolet-hashicorp since there's at least one fix available.
Hi @viraptor!
Thanks for the report; from what I can see, hdiutil
is always invoked with -hfs
(the code is from the SDK, SEE: https://github.com/hashicorp/packer-plugin-sdk/blob/main/multistep/commonsteps/step_create_cdrom.go#L164), so the resulting image is indeed HFS+.
If you import it from a Linux image later on, this could be a problem as you mention, the driver likely isn't loaded, therefore making it fail when attempting to mount.
Although according to the hdiutil
docs, even if there's HFS+, there should also be some ISO9660 and Joliet data, so the image should be mountable even without HFS+ support on the guest OS, so this option should theoretically be harmless. As you point out, I suspect there may be a problem with the cloud-utils
here, I do not know what they do, so I'll probably need your help to understand the issue exactly here.
Could you provide some logs to see what's being done with the build here? Preferably with PACKER_LOG=1
so we can have as much information as possible.
Additionally, I see that for the Linux-built ISO, there's Rock Ridge
support, while there isn't for those built by macOS
. This contradicts the documentation for hdiutil
(the -iso
option states Generate an ISO9660 Level 2 filesystem with Rock Ridge extensions.
). This could be a problem since Rock Ridge adds support for longer filenames and UNIX-like permissions among others.
Removing -hfs
from the command-line may not be the solution here, as some users may rely on this, so I'd tread carefully with these options, and since that code lives in the SDK, changes to that code will propagate to all plugins.
If you have other tools like xorriso or mkisofs, they should be used instead of hdiutil
if they're installed on your machine since they have precedence over hdiutil
when generating the command-line to create the data ISO.
To be frank, I won't be able to test/debug this as I don't have a macOS machine on hand, but we do have mac users on the Packer team, so I think we can manage to test and eventually fix this.
Sorry for the wall of text, I'm trying to figure out what the exact problem is. As mentioned though, could you provide logs of a failing build so we can see what the exact error is, and if possible, the template that you are using to build your image, so that we can replicate this behaviour and try to figure out a fix.
This issue may be an SDK one, but we'll have to figure out to understand what is the problem and the fix before we transfer this issue over, so I'll leave this in this repository in the meantime.
Ok, good news, at least xorriso
"just works":
2023/03/28 15:56:11 packer-plugin-qemu_v1.0.9_x5.0_darwin_arm64 plugin: 2023/03/28 15:56:11 Executing: /nix/store/w5n63bk4j68yfpxkfign1wf7sa91m1va-xorriso-1.5.4.pl02/bin/xorriso [-as genisoimage -rock -joliet -volid cidata -output /tmp/packer1815805504.iso /tmp/packer_to_cdrom3991152261]
mkisofs
from cdrtools does too: (cdrkit did not)
2023/03/28 15:59:19 packer-plugin-qemu_v1.0.9_x5.0_darwin_arm64 plugin: 2023/03/28 15:59:19 Executing: /nix/store/79x118zgwxjaslcp5q7bn1vnyf3bl702-cdrtools-3.02a09/bin/mkisofs [-joliet -volid cidata -o /tmp/packer2159155907.iso /tmp/packer_to_cdrom153317239]
hdiutil
call producing the failing hfs/iso:
2023/03/28 16:02:02 packer-plugin-qemu_v1.0.9_x5.0_darwin_arm64 plugin: 2023/03/28 16:02:02 Executing: /usr/bin/hdiutil [makehybrid -o /tmp/packer530833760.iso -hfs -joliet -iso -default-volume-name cidata /tmp/packer_to_cdrom453266334]
This fails on ssh connection - the user I expect simply isn't there (cloud-init never sees the provided cidata).
You should be able to replicate this on an ARM macos with:
source "qemu" "local-build" {
iso_checksum = "none"
iso_url = "https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-arm64.img"
disk_image = true
output_directory = "built-images"
disk_size = "5000M"
format = "qcow2"
qemu_binary = "qemu-system-aarch64"
headless = true
use_default_display = true
firmware = "edk2-aarch64-code.fd"
machine_type = "virt,highmem=off"
cpu_model = "host"
accelerator = "hvf"
ssh_username = "ubuntu"
ssh_password = "ubuntu"
cd_files = [ "meta-data", "user-data" ]
cd_label = "cidata"
}
build {
sources = ["source.qemu.local-build"]
provisioner "shell" {
inline = ["cloud-init status --wait"]
}
}
(+ user-data
like in the description on top)
Overview of the Issue
The options to create the
user-data
cd don't work well on macos. The.iso
s seem to be created just fine and look reasonable, but they're not detected by cloud-init.Reproduction Steps
Try using a user-data generated on macos with either
mkisofs
orhdiutil
. The iso is in place, but gets ignored by the image fromhttps://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-arm64.img
I did not dig into the differences too much.On the other hand, the same user-data packed using
cloud-localds
on linux gets detected and processed just fine. To test, I replacedhdiutil
with:and used the image copied from linux. (update: got the
cloud-utils
built on macos and that one also gets detected just fine, so it's not a mac/linux issue, but rathercloud-utils
vsmkisofs
/hdiutil
)Ok, so these are all external tools, why am I reporting it here? Mainly in case this is a general issue with
hdiutil
and it should be removed from the list. Or maybe there are some options you can use by default to make it work properly. I can't easily tell what the difference is between the different isos yet. I'll attach the a package with the 2 failing and 1 working version. isos.tar.gzPlugin and Packer version
1.8.4
Simplified Packer Buildfile
The
user-data
contains:Operating system and Environment details
Darwin Kernel Version 22.3.0, M2, aarch64 mkisofs - 3.02a09 cloud-utils - 0.32-22-g45fe84a5-0ubuntu1 hdiutil - whatever comes with that macos version.
Log Fragments and crash.log files
Nothing relevant beyond the user never being created.