[Feature Request] - Run Amazon Linux 2023 as a virtual machine on premises

[Manouchehri]

Is your feature request related to a problem? Please describe. Creation of Amazon Linux 2022 virtual machine (VM) images for on-premises development and testing. e.g. the same as: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/amazon-linux-2-virtual-machine.html

Describe the solution you'd like This URL to be valid:

https://cdn.amazonlinux.com/os-images/2022/kvm/

Describe alternatives you've considered I can probably figure out how to make my own bootstrapped install using dnf install --installroot=/mnt, but it would be nice to have an official image to start with.

[chadbrewbaker]

I'm working on this right now! Trying to get a UTM image for my M1, here are the instructions for Arch: https://ktprograms.codeberg.page/blog/posts/2022-03-17_1750_utm-arch-arm/

The first steps I think can be refactored into an Alpine linux container for x86/ARM.

• [ ] Build container
• [ ] Image for wsl2 - upstream to https://github.com/yosukes-dev/AmazonWSL
• [ ] Image for UTM - upstream to https://github.com/utmapp/mac.getutm.app/tree/main/_virtual_machines
• [x] Image for kvm

[Manouchehri]

@chadbrewbaker Hmm, not sure I get how that helps with AL2022?

[liesenml]

Hello, thank you for submitting your request! We review all requests on a weekly basis and will reply to this ticket by with our evaluation.

[chadbrewbaker]

Bump. This is nontrivial. I blew another day on it yesterday - need some official AWS help here.

My notes for inspecting Fedora images on macos:

strings Fedora-Minimal-36-1.5.armhfp.raw | grep RSA "PRIVATE"
xxd -a Fedora-Minimal-36-1.5.armhfp.raw | less
fdisk Fedora-Minimal-36-1.5.armhfp.raw
hdiutil attach -imagekey diskimage-class=CRawDiskImage -nomount Fedora-Minimal-36-1.5.armhfp.raw
hdiutil mount /dev/disk5s1

From what I can tell AmazonLinux2 used effectively the same grub2 VM structure as Fedora. https://arm.fedoraproject.org

@iximiuz has project to take the OSI container and stuff a bootloader etc on it: https://github.com/iximiuz/docker-to-linux/issues/25 This might be the most elegant solution.

[bplessis-swi]

The FAQ still references kvm images as to be provided on the GA release:

Q: Can I use AL2023 outside of AWS?

A: AL2023 images can be used outside of AWS, however, these images are not covered by AWS Support Plans when used outside of AWS. When Amazon Linux 2023 becomes Generally Available, it will be provided as a virtual machine image for on-premises use, enabling you to easily develop, test, and certify applications from a local development environment. This option is not available during the preview.

Look like someone forgot this ^^

[nmeyerhans]

Indeed the FAQ did indicate that VM images would be available at GA for use outside of EC2. Let's consider that FAQ entry to be inaccurate, for now.

We are working on VM images and will note their availability in the release notes and resolve this issue when they're available.

[emboss64]

Any updates on this one? Thanks!

[yoonjs2]

Any updates? Thanks!

[r-vw]

Also very interested in any progress wrt this request

[Rosch]

I am missing AL2023 in a vmware instance desperately! Is there anything known when this comes to life?

[rotan-hanrahan]

According to the Wayback Machine, the following text was present on https://aws.amazon.com/linux/amazon-linux-2023/faqs/ in March 2023:

When Amazon Linux 2023 becomes Generally Available, it will be provided as a virtual machine image for on-premises use, enabling you to easily develop, test, and certify applications from a local development environment. This option is not available during the preview.

The June 3 snapshot of the same page no longer contains this text, so the promise made during the preview regarding the VM image once AL2023 reaches GA appears to be withdrawn. (Note: the revision of the FAQ was suggested by @nmeyerhans in early April, almost three months ago.)

I see no evidence of any outreach to the community to explain this, nor any requests for technical assistance (assuming the issue is technical). If the issue is bureaucratic in nature, we might never see the promised VM image.

Some clarification from Amazon is overdue.

[inflatador]

Red Hat's recent announcement opens a window of opportunity for Amazon Linux to greatly increase on-prem adoption. Here's hoping Amazon will take advantage of that opportunity, starting with the VM images requested here.

[tanvp112]

It's becoming more and more difficult to use AL2 now, eg. https://urllib3.readthedocs.io/en/latest/v2-migration-guide.html#common-upgrading-issues

[stewartsmith]

Apologies for the long time radio-silence here, I understand the frustration this must cause, and there's certainly developers like myself who share that frustration.

While this isn't an image, it may be a useful starting point for building your own. Using the pretty neat kiwi-ng with the following config, I can boot to a login prompt. There's a few other things that are missing that turn this into a base image that we'd release (some cloud-init config at least), but it's a starting point.

You'll need this at root/etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="console=ttyS0,115200n8 console=tty0 rd.emergency=poweroff rd.shell=0 selinux=1 security=selinux quiet"
GRUB_TIMEOUT=0
GRUB_DISABLE_RECOVERY="true"
GRUB_TERMINAL="ec2-console"
GRUB_ENABLE_BLSCFG="true"
GRUB_X86_USE_32BIT="true"
GRUB_DEFAULT=saved
GRUB_UPDATE_DEFAULT_KERNEL=true

and this at root/etc/cloud/cloud.cfg.d/02-onprem.cfg

datasource_list: [ NoCloud, AltCloud, ConfigDrive, OVF, None ]

and then your appliance.kiwi can look like:

<?xml version="1.0" encoding="utf-8"?>

<image schemaversion="7.5" name="amazonlinux-2023">
    <description type="system">
        <author>Stewart Smith</author>
        <contact>trawets@amazon.com</contact>
        <specification>Example Amazon Linux 2023 kiwi config</specification>
    </description>
    <profiles>
        <profile name="Virtual" description="Virtual image of Amazon Linux 2023"/>
    </profiles>
    <preferences>
        <version>2.0.0</version>
        <packagemanager>dnf</packagemanager>
        <keytable>us</keytable>
        <timezone>UTC</timezone>
        <rpm-check-signatures>true</rpm-check-signatures>
    </preferences>
    <preferences profiles="Virtual">
      <type image="oem" filesystem="xfs" kernelcmdline="console=ttyS0" firmware="efi" format="qcow2">
            <oemconfig>
                <oem-resize>false</oem-resize>
            </oemconfig>
            <bootloader name="grub2" console="serial" timeout="10"/>
        </type>
    </preferences>
    <repository imageinclude="false" sourcetype="mirrorlist">
      <source path="https://cdn.amazonlinux.com/al2023/core/mirrors/latest/x86_64/mirror.list"/>
    </repository>
    <packages type="bootstrap" patternType="onlyRequired">
      <package name="dnf"/>
      <package name="curl-minimal"/>
      <package name="gnupg2-minimal"/>
    </packages>
    <packages type="image" patternType="onlyRequired">
      <namedCollection name="onprem"/>
      <namedCollection name="onprem-minimal"/>
      <package name="kernel"/>
      <package name="grub2-pc"/>
      <package name="grub2-pc-modules"/>
      <package name="grub2-efi-x64-modules"/>
      <package name="grub2-efi-x64-ec2"/>
      <package name="zstd"/>
      <package name="systemd-networkd"/>
    </packages>
</image>

Built on a Fedora 38 instance, and tested with qemu-system-x86_64 -boot c -net nic -net user -m 4G -cpu max -smp 4 -hda testboot -bios /usr/share/edk2/ovmf/OVMF_CODE.fd -nographic just to booting to a Login prompt, and not a moment further.

i.e. this is a starting point for someone to continue what that looks like.

[chadbrewbaker]

Apparently you can do this with AWS CLI? https://docs.aws.amazon.com/vm-import/latest/userguide/vmexport.html

al2023-ami is the keyword apparently?

[cpswan]

I've been keeping an eye on this issue for a while, and it seems AWS have really dropped the ball here.

Given the chaos in the Enterprise Linux (EL) space since RedHat announced changes to RHEL source availability there was a real opportunity for Amazon Linux to become the preferred distro for those wanting big company backing without subscribing to RHEL (or dealing with Oracle).

But that opportunity is being squandered, on this very issue. If people can't run AL23 everywhere that they want EL, then they'll probably choose something else as their baseline.

Furthermore, promising images and then failing to deliver makes it look like the project isn't properly resourced, which is another ding on trust.

Choice of distro has been a topic I've been following for a long time (since well before this post almost 7y ago). It seems almost incomprehensible that AWS isn't grabbing hold of the chance to get customers onto AL wherever they run Linux, which of course smoothes the path to migrating stuff to AWS.

Repo maintainers... this isn't aimed at you. I guess you're struggling to keep everything together with insufficient resources. This is for you to show to the higher ups, so they're making an informed decision about where they're placing their bets, and the consequences of ruined expectations and inaction.

[AaronSwearingen]

What OS does AWS recommend for hybrid cloud solutions? AL2 is less then two years away from EOL. AL2023 isn't available as a virtual machine and also presumably isn't supported for use with https://aws.amazon.com/hybrid/ features like ECS/EKS Anywhere.

I appreciate the repo maintainers trying to help out the community, but AWS needs to be made aware this is more than just a community request. There is a real business need to enable companies to leverage (and pay for) AWS hybrid cloud services. Anyone expecting AL2023 to fulfill that need now finds themselves in a position of uncertainty.

[chadbrewbaker]

What OS does AWS recommend for hybrid cloud solutions? AL2 is less then two years away from EOL. AL2023 isn't available as a virtual machine and also presumably isn't supported for use with https://aws.amazon.com/hybrid/ features like ECS/EKS Anywhere.

I appreciate the repo maintainers trying to help out the community, but AWS needs to be made aware this is more than just a community request. There is a real business need to enable companies to leverage (and pay for) AWS hybrid cloud services. Anyone expecting AL2023 to fulfill that need now finds themselves in a position of uncertainty.

Usual suspects - Ubuntu, Debian, Fedora, Suse, Arch. ML practitioners love Ubuntu, Linus loves Fedora.

I would probably use Fedora Desktop as it is closest to AL2023 for laptops.

This Reddit thread is spot on. AWS lost its way firing passionate maintainers like Tim McNamara that would have knocked out this ticket a year ago to get more community test coverage/adoption.

[AaronSwearingen]

I would probably use Fedora Desktop as it is closest to AL2023 for laptops.

ECS Anywhere supports Fedora 33 according to https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-anywhere.html. That's the latest version of Fedora on this list - EOL since 2021-11-30. The rest of the list isn't looking much healthier.

Getting AL2023 on the list would go a long way towards AWS showing it is committed to supporting hybrid cloud.

[hesaurey]

Amazon please release OVA for Vmware it's good for you, good for us!

[cpswan]

The Register referenced this issue with their piece Amazon Linux 2023 virtual machine images still MIA

It closes with the usual:

Amazon did not respond to a request for comment

But let's hope that it helps get some attention onto fixing the problem.

[Rosch]

Hi Amazon

An on premise version of AL 2023 is truly missing. When will it be available? We want to fully support AL2023 with our software but developing for it is only possible when it is available in our intranet. So yet we can only support AL2.

Beside of helping software developers an on premise version of AL 2023 is really a good chance for AL to fill the gap caused by RHELs actions.

Thanks

Roland

[daniejstriata]

I installed a host with AL2023 on-prem using DistroBox. It's adequate for my own limited use case, not the same as a VM running AL2023 though.

[hong539]

Hello! Any updates? Thanks!

[ozbenh]

Unfortunately not yet but we haven't dropped the ball either, stay tuned

[chris-bragg-sift]

Hello, can this be expedited please, I have multiple existing development KVM images that I would like to upgrade to AL2023. Many thanks.

[ferben]

   ~\_  ####_        Amazon Linux 2

  ~~  \_#####\

  ~~     \###|       AL2 End of Life is 2025-06-30.

  ~~       \#/ ___

   ~~       V~' '→

    ~~~         /    A newer version of Amazon Linux is available!

      ~~._.   _/

         _/ _/       Amazon Linux 2023, GA and supported until 2028-03-15.

       _/m/'           https://aws.amazon.com/linux/amazon-linux-2023/

So I would like upgrade to 2023, really, but there no VM ISO 👎

[stewartsmith]

Good news everyone!

We have just released the first KVM and VMware images of Amazon Linux 2023.

The documentation is up at https://docs.aws.amazon.com/linux/al2023/ug/outside-ec2.html and these first images can be found at https://cdn.amazonlinux.com/al2023/os-images/2023.2.20231113.0/ (the https://cdn.amazonlinux.com/al2023/os-images/latest/ URL that is in the User Guide will work shortly, and always lead to the latest release, much like it does for AL2, and I'll update this comment when I've verified it's working).

The documentation covers supported device types and related things.

This version of the images have a few extra development packages installed by default, and we'll make the minor change in an upcoming revision of the images (i.e. AL2023.3) to more closely mirror the package content of the standard AMIs. At that point in time we'll update the docs to show the full package list of the KVM and VMware images (like is present for the AMIs and container image types).

In the meantime, we welcome feedback on these images and documentation on how to use them!

[stewartsmith]

https://cdn.amazonlinux.com/al2023/os-images/latest/ gives a permission denied.

As I noted in my comment, it'll work in a little while, and in the meantime you can grab the first revision of the images at https://cdn.amazonlinux.com/al2023/os-images/2023.2.20231113.0/ .

We're actively working on fixing the latest URL.

[stewartsmith]

FYI, https://cdn.amazonlinux.com/al2023/os-images/latest/ works now.

[chadbrewbaker]

kvm images are giving 404. How about an aarch64 so I can run with @utmapp natively on my Mac M1?

Actually ping @utmapp - I bet they would add AWSLinux2023 to their hosted images. Right now they only have Fedora.

@FlorentRevest has a sweet vscode setup for Linux kernels that uses QEMU. He could add awslinux2023 as an example to make awslinux2023 kernel debugging/tracing stupid simple.

https://lpc.events/event/17/contributions/1614/attachments/1210/2474/VSCode%20for%20kernel%20development.pdf

[ccassidy-maxar]

Thanks for the update! Unfortunately when I try to load the ova into VMWare Workstation (17.0.2) I get the following error:

[azN2]

Ova worked for me. Thanks

[ozbenh]

Thanks for the update! Unfortunately when I try to load the ova into VMWare Workstation (17.0.2) I get the following error:

That is odd... can you try converting it to vmx+vmdk with ovftool first see if that helps ?

[ccassidy-maxar]

Thanks for the update! Unfortunately when I try to load the ova into VMWare Workstation (17.0.2) I get the following error:

That is odd... can you try converting it to vmx+vmdk with ovftool first see if that helps ?

Tried that, and threw a similar error:

Opening OVA source: al2023-vmware_esx-2023.2.20231113.0-kernel-6.1-x86_64.xfs.gpt.ova
The manifest validates
Opening VMX target: al2023-ova/
Writing VMX file: al2023-ova/al2023-vmware_esx-2023.2.20231113.0-kernel-6.1-x86_64.xfs.gpt/al2023-vmware_esx-2023.2.20231113.0-kernel-6.1-x86_64.xfs.gpt.vmx
Transfer Failed
Error: Disk (al2023-vmware_esx-2023.2.20231113.0-kernel-6.1-x86_64.xfs.gpt-disk2.vmdk) could not be found.
Deleting directory tree below: al2023-ova/al2023-vmware_esx-2023.2.20231113.0-kernel-6.1-x86_64.xfs.gpt`

I even tried using tar to extract the files (which does show a disk2 file) and reconstructing the ova and it still failed.

$ ls 
al2023-vmware_esx-2023.2.20231113.0-kernel-6.1-x86_64.xfs.gpt-disk1.vmdk
al2023-vmware_esx-2023.2.20231113.0-kernel-6.1-x86_64.xfs.gpt-disk2.vmdk
al2023-vmware_esx-2023.2.20231113.0-kernel-6.1-x86_64.xfs.gpt.mf
al2023-vmware_esx-2023.2.20231113.0-kernel-6.1-x86_64.xfs.gpt.ovf

$ ovftool al2023-vmware_esx-2023.2.20231113.0-kernel-6.1-x86_64.xfs.gpt.ovf al2023-vmware_esx-2023.2.20231113.0-kernel-6.1-x86_64.xfs.gpt.ova
Opening OVF source: al2023-vmware_esx-2023.2.20231113.0-kernel-6.1-x86_64.xfs.gpt.ovf
The manifest validates
Opening OVA target: al2023-vmware_esx-2023.2.20231113.0-kernel-6.1-x86_64.xfs.gpt.ova
Writing OVA package: al2023-vmware_esx-2023.2.20231113.0-kernel-6.1-x86_64.xfs.gpt.ova
Transfer Failed
Error: Failed to open disk: al2023-vmware_esx-2023.2.20231113.0-kernel-6.1-x86_64.xfs.gpt-disk2.vmdk. Reason: The file specified is not a virtual disk
Completed with errors

[tanvp112]

The links https://cdn.amazonlinux.com/al2023/os-images/latest/ and https://cdn.amazonlinux.com/al2023/os-images/2023.2.20231113.0/ returned 404:

404 Not Found
Code: NoSuchKey
Message: The specified key does not exist.
Key: os-images/2023.2.20231113.0/index.html
RequestId: PM0PRTE9GCANEGW3
HostId: 33/PHIgJZfTLYzV1NDyrTg3M/Vr4K8rh6jJ5VA+qKkbY9XrnHzGNUxhwj8ohNz9E13gLtfUdExY=
An Error Occurred While Attempting to Retrieve a Custom Error Document
Code: NoSuchKey
Message: The specified key does not exist.
Key: error.html

Wrong path (key)?

[stewartsmith]

kvm images are giving 404.

This should have been a temporary issue, I see things are working now.

How about an aarch64 so I can run with @utmapp natively on my Mac M1?

We have aarch64 KVM images available.

I've booted https://cdn.amazonlinux.com/al2023/os-images/2023.2.20231113.0/kvm-arm64/al2023-kvm-2023.2.20231113.0-kernel-6.1-arm64.xfs.gpt.qcow2 with UTM on a M1 MacBook Pro just fine (I did not test anything really past getting to a login: prompt). You will need to enable the serial console, import the disk image, and provide the seed.iso (the macOS instructions for generating it that are in the AL2023 documentation worked for me)

Undoubtedly there could be UX improvements here for running cloud like images in UTM; but it does appear to work.

[stewartsmith]

The links https://cdn.amazonlinux.com/al2023/os-images/latest/ and https://cdn.amazonlinux.com/al2023/os-images/2023.2.20231113.0/ returned 404:

You possibly hit a temporary issue. Releasing a new image type for a new major version of the OS isn't nearly as frequent as releasing a security update, so we get to discover all of the hiccups in that process. Thanks for understanding, and thanks to the AL team for getting this all going.

[stewartsmith]

Undoubtedly there could be UX improvements here for running cloud like images in UTM; but it does appear to work.

I just realized that the drive-resizing UI element that makes this practical was something I worked on a bit back when: https://github.com/utmapp/UTM/pull/4087 - maybe I'll get bored on a weekend again and try and do something for cloud like images.

[ccassidy-maxar]

Follow up to the issue w/ VMWare Workstation and the disk2 error. The following method at least allowed me to boot up to a prompt. Not sure if it will cause any unintended consequences yet.

1. Extract component files from the ova using tar
2. Delete references to the disk2 file from the .ovf file (3 places)
3. Get the SHA512 sum on the updated ovf file
4. Delete the disk2 references from the mf file, and update the ovf checksum
5. Recreated the ova with the following: ovftool al2023-vmware_esx-2023.2.20231113.0-kernel-6.1-x86_64.xfs.gpt.ovf al2023-vmware_esx-2023.2.20231113.0-kernel-6.1-x86_64.xfs.gpt.ova

Also noticed that the al2023-vmware_esx-2023.2.20231113.0-kernel-6.1-x86_64.xfs.gpt-disk2.vmdk file was actually XML, not sure if that is expected or not.

[ozbenh]

Follow up to the issue w/ VMWare Workstation and the disk2 error. The following method at least allowed me to boot up to a prompt. Not sure if it will cause any unintended consequences yet.

1. Extract component files from the ova using tar
2. Delete references to the disk2 file from the .ovf file (3 places)
3. Get the SHA512 sum on the updated ovf file
4. Delete the disk2 references from the mf file, and update the ovf checksum
5. Recreated the ova with the following: ovftool al2023-vmware_esx-2023.2.20231113.0-kernel-6.1-x86_64.xfs.gpt.ovf al2023-vmware_esx-2023.2.20231113.0-kernel-6.1-x86_64.xfs.gpt.ova

Also noticed that the al2023-vmware_esx-2023.2.20231113.0-kernel-6.1-x86_64.xfs.gpt-disk2.vmdk file was actually XML, not sure if that is expected or not.

This is very strange.... we built this ova using the same tooling we've always used for AL2 and tested it with both VMware workstation (17.5) and vSphere... I will take a look as soon as I'm not in a plane

[rotan-hanrahan]

I'm bound to a VirtualBox environment, and have used the VB images from Amazon ( such as here https://cdn.amazonlinux.com/os-images/2.0.20231101.0/ ), so I guess I will wait while the bugs are ironed out of the kvm/vmware versions and see what happens. So glad to see movement on this again, though I did lose one client to the "dark side" on account of no offsite pre-deployment options like we had with the previous images. (I might try to pull them back if the VB image appears.)

I'll take a quick peek once the URLs stop returning 404s.

Thanks @stewartsmith.

[sn-tm]

I'm experiencing the same issue as @ccassidy-maxar. VMWare Workstation Pro 17.5.0 on Windows 11. Open the .ova, give the VM a name and path, then see the same error, "Disk (al2023-vmware_esx-2023.2.20231113.0-kernel-6.1-x86_64.xfs.gpt-disk2.vmdk) could not be found."

I retested the same procedure using the AL2 .ova and had no problems.

By the way, thanks for providing this image. This is what we've been waiting for to help us test our software and migrate to AL2023 (once we resolve this issue of course!)

[ozbenh]

So it does look like the al2023 OVA has two "disks" in it, while previously we had only one. Probably due to its size (which incidently will be reduced in the next release as @stewartsmith mentioned).

I wonder if there's a host type related issue here. What are you using (both the good and base cases) ? We do our testing on a Ubuntu 22.04.3 system with VMWare Workstation for Linux, could it be a Windows specific issue ?

[sn-tm]

Hi @ozbenh, I just tried this on Ubuntu 22.04.3. Downloaded and installed VMWare Workstation 17 Pro, opened al2023-vmware_esx-2023.2.20231113.0-kernel-6.1-x86_64.xfs.gpt.ova, entered a name and location, clicked Import.

[ozbenh]

Ok, something's wrong on our end with the testing :-( We'll get to the bottom of it asap

[ozbenh]

We found the cause of the issue and will be updating the images. A sad story involving different versions of open-vmdk having different (incompatible) syntax for mkova.sh ... ugh. It should have been caught, sorry about that, something slipped in the test validation. Stay tuned for updated images. We'll also be slimming them a bit as mentioned earlier by @stewartsmith

[ozbenh]

A fixed (and slimmed) OVA has been uploaded and replaces the previous one. We haven't changed the KVM ones at all for now

[ccassidy-maxar]

A fixed (and slimmed) OVA has been uploaded and replaces the previous one. We haven't changed the KVM ones at all for now

Tested the new OVA and worked flawlessly on VMWare Workstation 17.0.2 / Windows 10, thanks!

[tanvp112]

I'm bound to a VirtualBox environment, and have used the VB images from Amazon ( such as here https://cdn.amazonlinux.com/os-images/2.0.20231101.0/ ), so I guess I will wait while the bugs are ironed out of the kvm/vmware versions and see what happens. So glad to see movement on this again, though I did lose one client to the "dark side" on account of no offsite pre-deployment options like we had with the previous images. (I might try to pull them back if the VB image appears.)

I'll take a quick peek once the URLs stop returning 404s.

Thanks @stewartsmith.

The OVA can be imported into VB 6.1.48, but after change password there seems to be storage driver incompatibility; either the new password is not recognized upon reboot or error with "Authentication token manipulation". Will need to try 7.x and possibly tune the seed files...