Create/sync repo oraclelinux8-appstream Failed to parse buildorder in component

[pititof]

Problem description

Hello, i update my uyuni system to 2024.07 in open suse 15.6. all is up to date ALl my repo working and sync fine (oraclelinux 7/ ubuntu 22.04/debian 12, and oraclelinux8) but the child oraclelinux-8 appstream have an issue.... i unsubscribe my systeme, delete this channel , retry to resync it always the same issue

2024/07/30 07:42:28 +02:00 Sync of channel started. 2024/07/30 07:42:43 +02:00 2024/07/30 07:42:43 +02:00 Importing modules file f1e8f4f94ea9c5b55639f9205d4eac1790cdb1319da89b664ccc9206d148efc6-modules.yaml.gz. 2024/07/30 07:42:43 +02:00 *** NOTE: Importing modules file for the channel 'oraclelinux8-appstream-x86_64'. Previous modules will be discarded. 2024/07/30 07:42:44 +02:00 An error occurred while reading module metadata: An error occurred while indexing a module entry: 2024/07/30 07:42:44 +02:00 Failed to parse buildorder in component: 18446744073709551615: The integer value is larger than 9223372036854775807 [line 158 col 9] 2024/07/30 07:42:44 +02:00 Failed to parse buildorder in component: 18446744073709551615: The integer value is larger than 9223372036854775807 [line 158 col 9] 2024/07/30 07:42:44 +02:00 Failed to parse buildorder in component: 18446744073709551615: The integer value is larger than 9223372036854775807 [line 170 col 9] 2024/07/30 07:42:44 +02:00 Failed to parse buildorder in component: 18446744073709551615: The integer value is larger than 9223372036854775807 [line 175 col 9] 2024/07/30 07:42:44 +02:00 Failed to parse buildorder in component: 18446744073709551615: The integer value is larger than 9223372036854775807 [line 157 col 9]

i try and check every thing, all repo in oraclinux8 sync evry day, but appstream... impossible to sync, or to recreate it, always the same issue appear :(....

Steps to reproduce

1./usr/bin/spacewalk-repo-sync --channel oraclelinux8-appstream-x86_64 --type yum --non-interactive or spacewalk-common-channels -a x86_64 oraclelinux8-appstream if i delete it and recreate it

2. check the /var/log/rhn/reposync/oraclelinux8-appstream-x86_64.log issue appear

3. ...

Uyuni version

2024.07

Uyuni proxy version (if used)

no proxy

Useful logs

2024/07/30 07:42:28 +02:00 Sync of channel started.
2024/07/30 07:42:43 +02:00
2024/07/30 07:42:43 +02:00   Importing modules file f1e8f4f94ea9c5b55639f9205d4eac1790cdb1319da89b664ccc9206d148efc6-modules.yaml.gz.
2024/07/30 07:42:43 +02:00 *** NOTE: Importing modules file for the channel 'oraclelinux8-appstream-x86_64'. Previous modules will be discarded.
2024/07/30 07:42:44 +02:00 An error occurred while reading module metadata: An error occurred while indexing a module entry:
2024/07/30 07:42:44 +02:00     Failed to parse buildorder in component: 18446744073709551615: The integer value is larger than 9223372036854775807 [line 158 col 9]
2024/07/30 07:42:44 +02:00     Failed to parse buildorder in component: 18446744073709551615: The integer value is larger than 9223372036854775807 [line 158 col 9]
2024/07/30 07:42:44 +02:00     Failed to parse buildorder in component: 18446744073709551615: The integer value is larger than 9223372036854775807 [line 170 col 9]
2024/07/30 07:42:44 +02:00     Failed to parse buildorder in component: 18446744073709551615: The integer value is larger than 9223372036854775807 [line 175 col 9]
2024/07/30 07:42:44 +02:00     Failed to parse buildorder in component: 18446744073709551615: The integer value is larger than 9223372036854775807 [line 157 col 9]

Additional information

No response

[mcalmer]

AFAIK this will be fixed with the next update. CC @cbbayburt

[bamorrow]

any known workaround for now?

[cbbayburt]

AFAIK this will be fixed with the next update. CC @cbbayburt

Not really, we've seen this problem in Liberty Linux repos before, but it was fixed in the repo itself. Other than that, we didn't do anything on our side.

But obviously, this is a more general problem. I'll see how we can patch against this.

[cbbayburt]

I had a deeper look, and it seems that the problem is with the libmodulemd python bindings (python3-libmodulemd) As I understand, some bug in their build service prints those values as 64-bit unsigned instead of 64-bit signed "-1"

And when python3-libmodulemd tries to parse it into 64-bit signed, we see the error.

Unfortunately, there's no direct fix for us, since the parsing happens in python3-libmodulemd via PyGObject.

The only solutions I can think of are:

• Open an upstream bug and track it there
• In reposync, parse the yaml ourselves in a loop before, fixing the offending values and then passing it to libmodulemd

[cbbayburt]

But ultimately, the bug originates from Oracle Linux's build service. Those large values should be "-1" instead. Until we develop a workaround, another repo update from Oracle can already fix the issue.

[bamorrow]

FYI... I had found that bug referenced in RHEL8 in the past. Their fix was to update the get an updated libmoduled that "relaxed" the scanning of the module meteadata. https://bugzilla.redhat.com/show_bug.cgi?id=1984402 https://github.com/fedora-modularity/libmodulemd/commit/37a688cc12d7fbab67fda95c47a4605405d7a154

FYI.. In my case what I do it actually dnf reposync the ULN content to local mirror and then feed UYUNI from the local mirror. The dnf reposync to create the local mirror goes just fine with no issue, so I assume they already have the fixed libraries in it.

[bamorrow]

Just out of curiosity, I did download a new libmodulemd and built it, then moved into place and tried the sync again, but same errors, so it is a bit more than just that. I just thought it was worth a try.

[cbbayburt]

@bamorrow yes, IIUC, our problem appeared just after that patch you linked.

From Bugzilla:

2.13.0 fixed a bug in parsing buildorder value. This version now complains on loading RHEL-8 repository

2.13 is what we're using. Moreover, I think the problem is not in the C part but the python bindings we're using. Dnf itself doesn't complain about it.

Also, I just remembered that Oracle accumulates all the released modules in the metadata so even another repo update won't fix the problem ever.

So that makes this issue more serious for us.

Hopefully we can fix it on our side instead of hoping another upstream fix.

[bamorrow]

I found it funny, so had to post what I found on the actually commit for that library....

+#ifdef HAVE_OVERFLOWED_BUILDORDER

• / A temporary hack. Remove when RHEL 8 goes end of life. /
• if (g_str_equal ((const gchar *)event.data.scalar.value,
• "18446744073709551615"))
• {
• g_debug ("Coercing an invalid signed 64-bit integer to -1: %s",
• (const gchar *)event.data.scalar.value);
• return -1;
• } +#endif

[bamorrow]

FYI.. in the mean time for a workaround I have setup a reposync with the option --setopt=repo_id.module_hotfixes=1 I then do a createrepo on the directory, just in case I deleted my current channel and recreated it with a repository pointed to the new download location.

this does at least let me the create the bootstrap now. Hope other may use this if in the same situation.

[cbbayburt]

I can see that it's been fixed in 2.14.0 upstream: https://github.com/fedora-modularity/libmodulemd/releases/tag/2.14.0

[cbbayburt]

If you need an urgent solution, you can try building 2.14.0 yourself. Getting the new version into Leap/SLES would take some time, so I think it'd be faster if we come up with our own solution.

[agraul]

One detail to note is that libmodulemd-2.14.0 has a new build option to enable the behavior we need. The changelog sounds like it's not the default but opt-in. Just something to keep in mind when building it.

Interpret an invalid buildorder 18446744073709551615 as -1 if the library is built with a new build boolean accept_overflowed_buildorder set to true.

[bamorrow]

I actually downloaded the libmodulemd-2.15.0 and built it, with that option set. Copied the library to /usr/lib64 and changed the symbolic links there. That did fix the issue.

[bamorrow]

In case some need this, I did the following: zypper install git meson git clone https://github.com/fedora-modularity/libmodulemd.git cd libmodulemd edit the file meson_options.txt change the line to read as follows: option('accept_overflowed_buildorder', type : 'boolean', value: 'true', description : 'Accept overflowed 18446744073709551615 buildorder as -1. This breaks a specification, but some RHEL 8 module builds look like that.') [you will need to install a lot of dependencies] meson setup buildir (until it comes up clean that dependencies are resolved) cd buildir ninja ninja install cp /usr/local/lib64/libmodulemd.so.2.15.0 /usr/lib64/ change the symlink for libmodulemd.so.2 to point to the new file (just in case I renamed the libmodulemd.so.2.13.0 to libmodulemd.so.2.13.0.old and cp /usr/local/lib64/libmodulemd.so.2.15.0 /usr/lib64/libmodulemd.so.2.13.0)

[pititof]

hello @bamorrow

I try your git, it's works fine !!! thanks !!!

for all user, before launch the meson setup build, the dependencies you need to install is:

zypper install git meson zypper install cmake zypper install gobject zypper install gobject-introspection zypper install gobject-introspection-devel zypper install libyaml-devel zypper install rpm-devel zypper install gtk-doc zypper install glib2-doc

Sincerely,

[jay2theb]

Thank you @pititof and @bamorrow your workaround worked for me on a rhel8 repo on 2024.07 also.

[bamorrow]

I did manage to build a set rpm to replace the system ones with libmodule 2.15.0

[stdevel]

Second. I also have this on a brand-new Uyuni 2024.08 running in a container. Will try the workaround later.

EDIT: The workaround also worked for me. For the container version, this needs to be repeated everytime the container is restarted as the /usr/lib64 path is volatile. I was using the 2.15.0 tag.

[fritz0011]

@bamorrow for the lazy guys :) can you upload here or somewhere else the newly compiled libmodule ? I suppose you did it for SLES 15 ?

[stdevel]

I built the attached library yesterday on a openSUSE Leap 15.5 machine and successful used it in the Uyuni 2024.08 container image. Should also work with SUMA 5.x aswell, I assume. Simply remove the symlink and copy the library before creating a new one:

# cp libmodulemd.so.2.15.0 /usr/lib64/
# rm /usr/lib64/libmodulemd.so.2
# ln -s /usr/lib64/libmodulemd.so.2.15.0 /usr/lib64/libmodulemd.so.2
# ls -la /usr/lib64/*modulemd*
lrwxrwxrwx. 1 root root      21 Aug 29 13:32 libmodulemd.so.2 -> libmodulemd.so.2.15.0
-rwxr-xr-x. 1 root root  482088 May  8  2022 libmodulemd.so.2.13.0
-rw-r--r--. 1 root root 2273656 Aug 29 13:31 libmodulemd.so.2.15.0

NOTE: Keep in mind that this change is not persistent, once the container is restarted, the old library will be used again.

libmodulemd.so.2.15.zip

[fritz0011]

yes, deployed to a standard opensuse install, so, finally oracle and rhel8 appstream are able to sync :)

[RenownTwiz]

Do we know if a permanent fix for this will be included in 2024.09?

I just ran through the above workaround steps on the container install via "mgrctl term" and it works in v2024.08. The only thing to note is that you have to modify /etc/zypp/zypp.conf with "rpm.install.excludedocs = no" because of a build dependency for a file in glib2-doc.

Error you get without changing rpm.install.excludedocs is: meson.build:94:6: ERROR: Problem encountered: Missing GTK documentation for glib: /usr/share/gtk-doc/html/glib/index.html

[cbbayburt]

Do we know if a permanent fix for this will be included in 2024.09?

Since the bug is in an external library (libmodulemd), it'll be fixed when an update is available in the host OS (Leap, Leap Micro). We'll push for an update there hopefully soon.

[RenownTwiz]

I might be wrong, but it sounds like the thought here is that accept_overflowed_buildorder would be set to "True" in the build of libmodulemd.so.2.15.0 or later in a future openSUSE MicroOS release? It looks like that value is defaulted to "False" when you pull the library from the git repo. What's to say they don't include a new version of libmodulemd in next release of MicroOS but leave accept_overflowed_buildorder set to "False" which wouldn't fix the issue right? This breaks most RHEL/OEL/etc v8 distros from what I can see so its pretty major. Workaround seems okay for VM builds, but a bit harder to support on the new Podman installed version, would it make sense to temporarily bake the fix into the container image itself, then when it's fixed upstream in MicroOS that could be removed?

[stdevel]

Could be possible as openSUSE MicroOS is based openSUSE Tumbleweed (rolling-release) while openSUSE Leap Micro is based on openSUSE Leap. Uyuni requires the latter and thus has the outdated version. I personally like the idea of adding the workaround to the container image but this would require additional testing and keeping track of changes in the upstream project - thrilled to hear what others think.

[oldvladko]

Fix for/etc/zypp/zypp.conf didn't work for me. Had to install old glib2-doc-2.70.5-150400.3.11.1.x86_64 which provides the required /usr/share/gtk-doc/html/glib/index.html . Once it's done the repos started to sync.