running containerized proxy on aarch64 fails

[jmozd]

Problem description

I'm trying to switch to containerized Uyuni proxy on an aarch64 machine, to replace a successfully running non-containerized version. Installation went smooth, but when starting the uyuni-proxy-pod service, the containers immediately stopped with an error "exec /bin/sh: exec format error".

Looking at the images downloaded to the machine, there's no indication that these are for aarch64, and checking some binaries of these images indeed show them to be x86-64.

Have I missed any special instruction to somehow switch to aarch64 images (a situation that then should have been detected by the startup routines automagically ;) ) or am I out of luck because no images for aarch64 are made available?

Steps to reproduce

1. Install Leap 15.5 on Raspberry Pi 4 8GB
2. zypper install uyuni-proxy-systemd-services
3. systemctl enable --now uyuni-proxy-pod
4. journalctl -f

Uyuni version

Information for package Uyuni-Server-release:
---------------------------------------------
Repository     : opensuse-leap15.5-uyuni_container-aarch64
Name           : Uyuni-Server-release
Version        : 2023.10-230900.209.1.uyuni3
Arch           : aarch64
Vendor         : obs://build.opensuse.org/systemsmanagement:Uyuni
Installed Size : 1.4 KiB
Installed      : Yes
Status         : up-to-date
Source package : Uyuni-Server-release-2023.10-230900.209.1.uyuni3.src
Summary        : Uyuni Server

Uyuni proxy version (if used)

Information for package Uyuni-Proxy-release:
--------------------------------------------
Repository     : opensuse-leap15.5-uyuni_container-aarch64
Name           : Uyuni-Proxy-release
Version        : 2023.10-230900.209.1.uyuni3
Arch           : aarch64
Vendor         : obs://build.opensuse.org/systemsmanagement:Uyuni
Installed Size : 1.4 KiB
Installed      : Yes
Status         : up-to-date
Source package : Uyuni-Proxy-release-2023.10-230900.209.1.uyuni3.src
Summary        : Uyuni Proxy

Images that were downloaded:

REPOSITORY                                            TAG               IMAGE ID      CREATED        SIZE
localhost/podman-pause                                4.7.2-1698840000  fc76462f4a27  2 minutes ago  608 kB
registry.suse.com/suse/manager/4.3/proxy-salt-broker  latest            712a7ff1b7e0  6 days ago     174 MB
registry.suse.com/suse/manager/4.3/proxy-httpd        latest            fdb2d9cc67ae  6 days ago     250 MB
registry.suse.com/suse/manager/4.3/proxy-squid        latest            4dbad6424b54  6 days ago     185 MB
registry.suse.com/suse/manager/4.3/proxy-tftpd        latest            4ae98fbd5167  6 days ago     188 MB
registry.suse.com/suse/manager/4.3/proxy-ssh          latest            d1e68bfab2a8  6 days ago     177 MB

Useful logs

No response

Additional information

No response

[mcalmer]

@raulillo82 @cbosdo what is the status of aarch64 proxy container in uyuni?

[cbosdo]

First issue I see is that you are using SUSE Manager 4.3 images with Uyuni.

Second issue is that for now we don't build the Uyuni container images for non-x86_64 architecture as you can see on https://registry.opensuse.org/cgi-bin/cooverview?srch_term=container%3D%5Euyuni.* . @raulillo82 could we add aarch64 to the Uyuni project repos? I see it's in openSUSE_Leap_15.5_containers repo on Master but not in containers. It's also not in containers on Stable

[jmozd]

First issue I see is that you are using SUSE Manager 4.3 images with Uyuni.

this is a default install from uyuni repository, we never touched /etc/sysconfig/uyuni-proxy-systemd-services:

# This file is expected to be found in `/etc/sysconfig/container-proxy-services.config`,
# the EnvironmentFile services property is pointing there

# Where to get the images from if not defined otherwise in a service-specific configuration
# It should contain the registry FQDN and path to the proxy-* images without trailing slash
NAMESPACE=registry.suse.com/suse/manager/4.3

The file's date matches the installation time frame of the packages and is confirmed by the according entry in /var/log/zypp/history:

# 2024-02-01 20:00:44 uyuni-proxy-systemd-services-4.4.2-230900.1.1.uyuni3.noarch.rpm installed ok
# Additional rpm output:
# Updating /etc/sysconfig/uyuni-proxy-systemd-services ...

While the sysconfig file comment suggests otherwise, the file is /etc/sysconfig/uyuni-proxy-systemd-services and not /etc/sysconfig/container-proxy-services.config:

# ll /etc/sysconfig/container-proxy-services.config
ls: cannot access '/etc/sysconfig/container-proxy-services.config': No such file or directory

So from my point of view, it's the Uyuni proxy install that selected the SUSE Manager images.

Second issue is that for now we don't build the Uyuni container images for non-x86_64 architecture as you can see on https://registry.opensuse.org/cgi-bin/cooverview?srch_term=container%3D%5Euyuni.* . @raulillo82 could we add aarch64 to the Uyuni project repos? I see it's in openSUSE_Leap_15.5_containers repo on Master but not in containers. It's also not in containers on Stable

Thanks for confirming and yes, we'd actually like to use the containers on aarch64, too ;) And I would find it good style to at least check for supported platforms and report when starting the services.

[deneb-alpha]

@jmozd hello, indeed, in the last Uyuni release we still provided only the x86_64 support.

From your report I noticed that both Uyuni proxy and server are using version 2023.10. Do you have by chance the opportunity to use the latest released version (AKA 2024.01)?

Also there the aarch64 support is missing in stable but at least you should also get the other fixes and improvements we made to this technology preview.

I'll keep you posted on the aarch64 enablement on Uyuni

[cbosdo]

With the last release you can also use mgrpxy to install the container proxy. This is the preferred way: the uyuni-proxy-systemd-services package will be phased out.

[cbosdo]

First issue I see is that you are using SUSE Manager 4.3 images with Uyuni.

this is a default install from uyuni repository, we never touched /etc/sysconfig/uyuni-proxy-systemd-services:

Ouch! This should be fixed in mgrpxy for sure.

So from my point of view, it's the Uyuni proxy install that selected the SUSE Manager images.

I don't think I'll fix the uyuni-proxy-systemd-services package as mgrpxy should be preferred.

And I would find it good style to at least check for supported platforms and report when starting the services.

I can implement this in the mgrpxy and mgradm tools, thanks for the hint.

[jmozd]

@deneb-alpha hi,

the central server in question is actually a SUSE Manager v4.3.10, though this does not apply (yet), since the error happens already while starting the proxy containers.

Do you have by chance the opportunity to use the latest released version (AKA 2024.01)?

The packages used for install are from https://download.opensuse.org/repositories/systemsmanagement:/Uyuni:/Master/openSUSE_Leap_15.5_containers/ and according to our reposync logs, the channel is up-to-date. I re-checked the docs and see that this is no longer the correct repo, I'll switch and update this ticket.

@cbosdo Seems I missed the switch to mgrpxy - I'll re-run the installation following the latest docs. Of course this will not get me around the missing aarch64 container builds, but having everything ready up to that should be a better starting point than I have now.

[jmozd]

The instructions at https://www.uyuni-project.org/uyuni-docs/en/uyuni/installation-and-upgrade/proxy-container-installation.html seem ambiguous:

The (non-containerized) proxy installation tells to add a specific Uyuni repository. The container proxy docs only state "Assign Containers Module software channel to the container host in the Uyuni." I know the Container module for SLE, but could find nothing alike for Leap 15.5. Looking around for an according Uyuni channel, I successfully tried https://download.opensuse.org/repositories/systemsmanagement:/Uyuni:/Master:/ContainerUtils/openSUSE_Leap_15.5.

I could install mgrpxy from that repo and use it as documented. This then will download the proper Uyuni (not SUSE Manager) containers:

registry.opensuse.org/uyuni/proxy-salt-broker latest 6bc77acb1525 4 days ago 175 MB registry.opensuse.org/uyuni/proxy-httpd latest 32419d20cca2 4 days ago 250 MB registry.opensuse.org/uyuni/proxy-squid latest 65cb21893363 4 days ago 186 MB registry.opensuse.org/uyuni/proxy-tftpd latest 278e8ec3b673 4 days ago 189 MB registry.opensuse.org/uyuni/proxy-ssh latest c9f731b59013 4 days ago 178 MB

[jmozd]

Found the following in the "Install on k3s" docs:

TODO The mgrpxy package is available in the TBD repository.

Maybe that one might have been added to https://www.uyuni-project.org/uyuni-docs/en/uyuni/installation-and-upgrade/proxy-container-installation.html, too :)

[deneb-alpha]

Hi @jmozd yes, the documentation needs some polishing and it's work in progress as you can see. The container module is indeed something only for SUSE Manager.

Please continue the testing and report the other issues you will spot or if you can, feel free to also provide a PR.

[deneb-alpha]

@jmozd one good news. the server-image on https://build.opensuse.org/package/show/systemsmanagement:Uyuni:Master/server-image doesn't have unresolvables for grub2-x86_64-efi on aarch64 anymore.

Please note that the fix is in Master. for Stable you will need to wait for the next Uyuni release. In the meantime, not sure if you want to give it a try using master. :)

[jmozd]

In the meantime, not sure if you want to give it a try using master. :)

I got an update for mgrpxy which I installed on the Pi4, then ran the following commands:

uyuniyproxy-ue:/etc/systemd/system # mgrpxy --version
mgrpxy version 0.0.0
uyuniyproxy-ue:/etc/systemd/system # rpm -q mgrpxy
mgrpxy-0.1.3-11.1.uyuni.aarch64
uyuniyproxy-ue:/etc/systemd/system # mgrpxy install podman /root/uyuniproxy.config.tar.gz --imagesLocation registry.opensuse.org/systemsmanagement/uyuni/master/containers/uyuni

This started the pods (this time being aarch64, so they were executable), but most of them were stopped almost immediately and the following messages from the httpd container were found in the journal:

Feb 13 17:56:57 uyuniyproxy-ue uyuni-proxy-httpd[30346]: Traceback (most recent call last):
Feb 13 17:56:57 uyuniyproxy-ue uyuni-proxy-httpd[30346]:   File "/usr/bin/uyuni-configure.py", line 83, in <module>
Feb 13 17:56:57 uyuniyproxy-ue uyuni-proxy-httpd[30346]:     logging.critical("Proxy container image version (%s) doesn't match server major version (%s)", container_version, major_version, file=sys.stderr)
Feb 13 17:56:57 uyuniyproxy-ue uyuni-proxy-httpd[30346]:   File "/usr/lib64/python3.6/logging/__init__.py", line 1857, in critical
Feb 13 17:56:57 uyuniyproxy-ue uyuni-proxy-httpd[30346]:     root.critical(msg, *args, **kwargs)
Feb 13 17:56:57 uyuniyproxy-ue uyuni-proxy-httpd[30346]:   File "/usr/lib64/python3.6/logging/__init__.py", line 1355, in critical
Feb 13 17:56:57 uyuniyproxy-ue uyuni-proxy-httpd[30346]:     self._log(CRITICAL, msg, args, **kwargs)
Feb 13 17:56:57 uyuniyproxy-ue uyuni-proxy-httpd[30346]: TypeError: _log() got an unexpected keyword argument 'file'

This is the image used:

uyuniyproxy-ue:/etc/systemd/system # podman images | grep proxy-httpd
registry.opensuse.org/systemsmanagement/uyuni/master/containers/uyuni/proxy-httpd        latest            bf0eca649c9b  2 hours ago  288 MB

[rjmateus]

Which uyuni server version are you running? Inside "/root/uyuniproxy.config.tar.gz" on file "config.yaml" which value do you have for "server_version"?

Engineering note: The version from inside the container is "5.0.1" and we are comparing the major version with equals. We should allow also one major version before. https://github.com/uyuni-project/uyuni/blob/master/containers/proxy-httpd-image/uyuni-configure.py#L77-L83

[jmozd]

Bingo, that's the place:

server_version: 4.3.10

I changed that to "5.0.1", rebuilt the tar ball, redeployed and all is up & running :thumbsup:

(Totally minor suggestion: Could mgrpxy be made more agnostic wrt the tar compression format, like the "tar" command has build-in? I forgot to specity "-z" option when repacking the tar ball (for such small files I tend to ignore this), but used the original suffix (tar.gz). Seems mgrpxy did not like that.)

3:43PM INF Welcome to mgrpxy
3:43PM INF Executing command: podman
3:43PM INF Setting up proxy with configuration /root/uyuniproxy.config-5.0.1.tar.gz
Error: failed to extract proxy config from /root/uyuniproxy.config-5.0.1.tar.gz file: gzip: invalid header

[jmozd]

As the containers are up & running on aarch64 now, "I declare this bazar closed". :grinning:

Thank you all for the excellent and responsive support!