wheels on PyPI

[carlwgeorge]

I'm using the createrepoc Python bindings in a project of mine. For the CI test run I'm using the [python:3.7](https://hub.docker.com//python) container image. I'd like to be able to pip install createrepo_c, but this fails due to multiple missing system libraries. I can work around this by installing them with apt first (at a minimum cmake, libmagic-dev, and librpm-dev), but I believe if a wheel package of createrepo_c was uploaded to PyPI it would just work.

Can a createrepo_c wheel be uploaded to PyPI?

[hroncok]

Note that the createrepo-c PyPI package appears to be maintained by pulp maintainers.

[carlwgeorge]

/me checks pulp/pulp_rpm commit history for recently active maintainers...

@goosemania @ipanova @daviddavis @dralley Would y'all be able to help here?

[dralley]

Hello @carlwgeorge

Currently uploading a binary wheel of createrepo_c to PyPI is blocked by some changes in the Python build infrastructure https://github.com/pypa/manylinux/pull/279

Since compiled packages link against system libraries, and the system libraries are different on different systems, Python created a build environment called "manylinux" and then built tooling that wraps up all the dynamically-linked libraries into the wheel so that it doesn't depend on any system-provided libraries. In order to upload a binary wheel to PyPI, it has to have been created in this build environment because otherwise a package built on Ubuntu and uploaded would not work on CentOS.

Unfortunately, that common build environment is based on CentOS 5 and createrepo_c won't build on CentOS 5. The new environment discussed in the PR above is based on CentOS 6 and that will be usable. But it's taking a really, really long time for them to get it ready.

That's why, for now, all we've been able to do is upload a source package and have our CI install all of the build dependencies just like you're doing.

https://github.com/pulp/pulp_rpm/blob/master/.travis/install.sh#L44-L46

[carlwgeorge]

Thanks for the clarification @dralley. I am vaguely familiar with manylinux, but I didn't know that it was still building against EL5. That's understandable that a createrepo_c wheel is blocked on the build environment getting refreshed to EL6.

[dralley]

Yup, sorry I couldn't be more helpful! It was disappointing to us too when we realized that.

[carlwgeorge]

Hey, having a clear understanding of the blocker is absolutely helpful. Thanks!

[hroncok]

but I didn't know that it was still building against EL5

manylinux1 will always build against EL5.

manylinux2010 (imagine "manylinux2", but the naming schema has changed) will always build against EL6.

[dralley]

Also, once that environment does come out, there may be added complications since CentOS 6 doesn't have zchunk or libmodulemd packages available. Hopefully they can at least be compiled on CentOS 6. If not, I'm not really sure what the solution is...

https://pkgs.org/download/zchunk-libs https://pkgs.org/download/libmodulemd

As it stands, these new deps make it (virtually?) impossible to build a fully-featured createrepo_c Python package from source on any Debian-based distro.

[carlwgeorge]

Since those exist in Fedora (and aren't in RHEL) it's just a matter of requesting the respective Fedora maintainers add an el6 branch and build it for EPEL 6. Of course the dependencies will need to be worked out too. Right off the bat I see that meson and gobject-introspection will be needed, and probably a few others.

[sgallagher]

@dralley just pointed me at this ticket.

So, libmodulemd cannot realistically be built on RHEL/EPEL 6. It needs a far newer glib2 and gobject-introspection than will ever be available on that platform. (And we can't ship newer ones in EPEL for a variety of reasons, not least of which being that they would conflict with or replace the official RHEL packages and in so doing cause lots of chaos).

If this is a significant-enough blocker to justify it, we could look into producing a version of libmodulemd that doesn't provide GObject Introspection bindings and statically-links the libyaml and glib2 dependencies, but that's a big undertaking.

[hroncok]

If this was actually desired and some effort put into this, conflicts with RHEL or EPEL wouldn't really matter. A copr repo (for example) would do.

[hroncok]

As an example, with admesh, I just build my own EL5 RPM:

https://github.com/admesh/python-admesh/tree/v0.98.9/travis https://github.com/admesh/python-admesh/blob/027f9fe1922d7aada4e032632fcfcb1a4bde5f76/.travis.yml#L48

This is of course much easier, because it is not a full stack of dependencies but just one tiny library.

[carlwgeorge]

@sgallagher

• gobject-introspection isn't in RHEL6, so why couldn't it be added to EPEL6?
• What is the minimum version of glib2 needed to build libmodulemd? The libmodulemd spec file build requires pkgconfig(gobject-2.0), which is provided in RHEL6 by glib2-devel-2.28.8-10.el6. If a newer version is necessary it would be preferable to build require at least that version.
• For libyaml, the build requirement is pkgconfig(yaml-0.1). That isn't provided by RHEL6's libyaml-devel, but that could just be a packaging oversight as the difference between RHEL6 and RHEL7 here is just 0.1.3 to 0.1.4.

[carlwgeorge]

I don't need libmodulemd or zchunk support for my use case. I just want pip to be able to install a createrepo_c wheel with basic functionality.

[carlwgeorge]

Also related, I tried to just use the system python3-createrepo_c package on Fedora, but it's missing metadata so pip can't see it.

https://bugzilla.redhat.com/show_bug.cgi?id=1695677

[sgallagher]

I don't need libmodulemd or zchunk support for my use case. I just want pip to be able to install a createrepo_c wheel with basic functionality.

Define "basic functionality"? Because I'm working on the libmodulemd support right now with the plan that if you encounter a repo with module metadata but haven't built with libmodulemd support, it's going to refuse to operate on that repo.

[carlwgeorge]

For me, "basic functionality" means being able to create repomd.xml, primary.xml.gz, filelists.xml.gz, and other.xml.gz for a directory of RPMs. Checking my code, these are the attributes I use from the Python bindings:

• package_from_rpm
• Repomd
• RepomdRecord
• PrimaryXmlFile
• FilelistsXmlFile
• OtherXmlFile

[sgallagher]

@sgallagher

• gobject-introspection isn't in RHEL6, so why couldn't it be added to EPEL6?

gobject-introspection is fairly tightly coupled with glib2. (They don't always rev together, but quite often. The current version, 1.6.0 requires glib2 >= 2.58.0.

So for whatever version of GI we want to include, we're going to need a recent-enough glib2 to match. Which means either replacing the system copy, statically linking, or doing a mess of work to get GI and libmodulemd to point at a renamed/relocated library.

• What is the minimum version of glib2 needed to build libmodulemd? The libmodulemd spec file build requires pkgconfig(gobject-2.0), which is provided in RHEL6 by glib2-devel-2.28.8-10.el6. If a newer version is necessary it would be preferable to build require at least that version.

Honestly, I don't know. It builds successfully against glib2-2.54 at least (which was the earliest one I tried). I'm not sure how far back I could go and still have it work properly.

• For libyaml, the build requirement is pkgconfig(yaml-0.1). That isn't provided by RHEL6's libyaml-devel, but that could just be a packaging oversight as the difference between RHEL6 and RHEL7 here is just 0.1.3 to 0.1.4.

Yeah, that's a packaging issue that we could solve in EPEL (create libyaml-pkgconfig whose sole purpose is to provide that file missing from RHEL and Requires: libyaml-devel).

[carlwgeorge]

Thanks for the insight on gobject-introspection/glib2. The earliest reference I can find to a minimum version is when GI 1.29.0 added a requirement of glib2 2.29.7, which is still too new for EL6's 2.28.8.

https://gitlab.gnome.org/GNOME/gobject-introspection/commit/fe01f575b0328107e64fb0c1f2d40d35e6e669cf

That might be a dead end for libmodulemd support, so hopefully createrepo_c will retain a flag to optionally disable that feature. That would allow a createrepo_c wheel (without libmodulemd support) to be uploaded to PyPI once manylinux2010 is finished.

[dralley]

That might be a dead end for libmodulemd support, so hopefully createrepo_c will retain a flag to optionally disable that feature. That would allow a createrepo_c wheel (without libmodulemd support) to be uploaded to PyPI once manylinux2010 is finished.

I don't know, the fact that it would fail on any repository containing modules almost seems as though it's too crippled to ship on PyPI under the same name. In any case, we do need the libmodulemd functionality, so a package like that wouldn't help us, and I don't think we could then justify the effort to set up the (not completely trivial) infrastructure needed to build those wheels (that we can't use) for each release.

I think the ability to produce Python packages is still useful even if they can't be shipped on PyPI or work generically on any kind of distro, though.

@carlwgeorge, Could you build such a package locally, and upload it to some form of file storage, and then just pip install http://..../createrepo_c-12.0.whl or something along those lines inside your CI environment. That would eliminate the need to compile inside your CI environment. And of course, you could keep compiling the source package with any particular combination of command line arguments you need.

See: https://github.com/pulp/pulp_rpm/blob/master/.travis/install.sh#L46

Or: if the issue with the python3-createrepo_c metadata was resolved, that would also solve your problem?

[carlwgeorge]

Yes, if rhbz#1695677 is resolved I'll just use a Fedora container for my CI and and use the system python3-createrepo_c package. Trying to do that now still attempts to install createrepo_c from PyPI because of the missing metadata.

[sgallagher]

So, I got nerd-sniped here and ended up seeing how far I could get towards making libmodulemd build against EPEL 6. Turns out, it was pretty far (thanks, SCLs!), but I've finally discovered the minimum version of glib2 I need: It's 2.44. I'm going to stop here and say that EPEL 6 isn't achievable.

[dralley]

@sgallagher Thanks for providing that confirmation!

[jdieter]

So, I realize you said zchunk wasn't important for your use-case, but I just took a look at what it would take to get it to compile on CentOS 6, and the main missing dependency is meson. zchunk itself has very few dependencies, just libcurl, openssl and zstd, and all are in CentOS 6. On the other hand, I quite like meson as a build system, and I'm not sure that I'm ready to switch to something like autotools just for CentOS 6 compatibility. I don't suppose anyone's worked out a way to get meson to run on CentOS 6?

[hroncok]

You should be able to pip install meson, if you target Python 3.5+.

[carlwgeorge]

It will need ninja-build first, which I requested earlier. rhbz#1695759

Please excuse my accidental markdown in bugzilla. :grimacing:

[sgallagher]

I have a COPR of meson 0.44.1, which is the newest that will build on EPEL 6 (0.45 and later require at least Python 3.5). It might be possible to do something around SCLs though.

[jdieter]

I have a COPR of meson 0.44.1, which is the newest that will build on EPEL 6 (0.45 and later require at least Python 3.5). It might be possible to do something around SCLs though.

Thanks @sgallagher, I installed your COPR and meson and ninja work fine, but now I'm getting loads of compilation errors, I'm assuming because gcc is so old:

../src/lib/zck_private.h:92: error: redefinition of typedef 'zckCtx'
include/zck.h:49: note: previous declaration of 'zckCtx' was here
../src/lib/zck_private.h:106: error: redefinition of typedef 'zck_log_type'
include/zck.h:47: note: previous declaration of 'zck_log_type' was here

I'd be happy to dig into this more.

[sgallagher]

On Wed, Apr 3, 2019 at 5:13 PM Jonathan Dieter notifications@github.com wrote:

I have a COPR of meson 0.44.1, which is the newest that will build on EPEL 6 (0.45 and later require at least Python 3.5). It might be possible to do something around SCLs though.

Thanks @sgallagher https://github.com/sgallagher, I installed your COPR and meson and ninja work fine, but now I'm getting loads of compilation errors, I'm assuming because gcc is so old:

../src/lib/zck_private.h:92: error: redefinition of typedef 'zckCtx' include/zck.h:49: note: previous declaration of 'zckCtx' was here ../src/lib/zck_private.h:106: error: redefinition of typedef 'zck_log_type' include/zck.h:47: note: previous declaration of 'zck_log_type' was here

I'd be happy to dig into this more.

On CentOS 6, you can yum install centos-release-scl and then yum install devtoolset-8 for a recent GCC

[jdieter]

On CentOS 6, you can yum install centos-release-scl and then yum install devtoolset-8 for a recent GCC

That worked and passed the test suite. So, zchunk can be compiled on CentOS 6, as long as you're willing to use a COPR and an SCL.

If meson and ninja get into EPEL 6, I'd be happy to figure out how to make zchunk compile with gcc-4.4.

[sgallagher]

On CentOS 6, you can yum install centos-release-scl and then yum install devtoolset-8 for a recent GCC

That worked and passed the test suite. So, zchunk can be compiled on CentOS 6, as long as you're willing to use a COPR and an SCL.

If meson and ninja get into EPEL 6, I'd be happy to figure out how to make zchunk compile with gcc-4.4.

Apparently, EPEL is permitted to use SCLs at build time (but not runtime). Among others, chromium uses the devtoolset-8 package to build on EPEL 7. So presumably it's okay for libzchunk to do the same if we go ahead and land ninja and meson 0.44.1

I'm willing to package them for EPEL 6 with the understanding that they will be effectively unmaintained since they're beyond upstream support.

[hroncok]

What went quite ignored here is that it is not needed to prep this in any sustainable way. A docker container that downloads and compiles glib2 >= 2.44, pip installs meson etc. would do the job.

[dralley]

@hroncok I assume you're talking about doing that inside the manylinux environment (which is itself a docker container)?

Manylinux 2010 comes with SCL support, EPEL, and devtoolset 8 baked in, but recompiling something like glib2 that it provides might be a bit hairy. They use patchelf to inject all the dynamic libs inside the package and I'm not sure how that would work out if we start swapping the versions of fundamental libraries.

Also, what's the build story with gobject-introspection now? Fixing the zchunk support is excellent but libmodulemd is probably the larger issue.

[sgallagher]

@hroncok I assume you're talking about doing that inside the manylinux environment (which is itself a docker container)?

Manylinux 2010 comes with SCL support, EPEL, and devtoolset 8 baked in, but recompiling something like glib2 that it provides might be a bit hairy. They use patchelf to inject all the dynamic libs inside the package and I'm not sure how that would work out if we start swapping the versions of fundamental libraries.

Also, what's the build story with gobject-introspection now? Fixing the zchunk support is excellent but libmodulemd is probably the larger issue.

Building libmodulemd on EL 6 is probably not realistic. We might be able to manage a fully statically-linked version, but I'm not convinced that it's worth the effort. The current plan is to just make sure that createrepo_c gracefully refuses to operate on repos where modulemd is present if it wasn't built with libmodulemd. I'll make sure it doesn't break compatibility with non-modular repos.

[dralley]

Cool, I just wasn't sure whether that situation had changed. My thought was that without being able to support modules, an EPEL 6 version of createrepo_c would not be so useful and not necessarily worth the effort of trying to make Zchunk work.

It seems like the best path forwards for @carlwgeorge is to have the metadata of the RPM package fixed, and the best path forwards for us is to find a plan for distributing pulp_rpm that does not involve providing createrepo_c via precompiled binary Python package from PyPI.

[carlwgeorge]

I was added as a co-maintainer for ninja-build, so it's done now, pending release to epel-testing. I've opened a similar request for meson.

[jdieter]

Apparently, EPEL is permitted to use SCLs at build time (but not runtime). Among others, chromium uses the devtoolset-8 package to build on EPEL 7. So presumably it's okay for libzchunk to do the same if we go ahead and land ninja and meson 0.44.1

That would be perfect! I'd rather not try to figure out how to compile zchunk on an ancient compiler.

[carlwgeorge]

I was added as a co-maintainer for meson, and have submitted meson 0.44.1 to EPEL6. I've created overrides for ninja-build and meson so you can start using them right away.

[jdieter]

Apparently, EPEL is permitted to use SCLs at build time (but not runtime). Among others, chromium uses the devtoolset-8 package to build on EPEL 7. So presumably it's okay for libzchunk to do the same if we go ahead and land ninja and meson 0.44.1

Do you know where the documentation is for this? I've followed the chromium example as best I can, and it builds properly on my CentOS 6 instance, but I can't seem to get a scratch build to find the devtoolset-8-toolchain.

See https://koji.fedoraproject.org/koji/taskinfo?taskID=34036474

[carlwgeorge]

@dralley The quay.io/pypa/manylinux2010_x86_64 container image is now available. PyPA isn't claiming that it's done per say, but it's ready for people to start hammering on it and reporting issues.

[jdieter]

For those who care, zchunk has now been built for EPEL 6 and is on its way to epel-testing.

https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2019-149a9e1b96

[Conan-Kudo]

@sgallagher So... I've done terrible things for $DAYJOB to backport a newer glib2 to EL6 without overriding the system one. However, I've only gone up to glib2 2.36.3. Would this be any help?

[sgallagher]

@sgallagher So... I've done terrible things for $DAYJOB to backport a newer glib2 to EL6 without overriding the system one. However, I've only gone up to glib2 2.36.3. Would this be any help?

@sgallagher wrote in https://github.com/rpm-software-management/createrepo_c/issues/145#issuecomment-479629150

So, I got nerd-sniped here and ended up seeing how far I could get towards making libmodulemd build against EPEL 6. Turns out, it was pretty far (thanks, SCLs!), but I've finally discovered the minimum version of glib2 I need: It's 2.44. I'm going to stop here and say that EPEL 6 isn't achievable.

[Conan-Kudo]

I haven't tried to see if the terrible things I did for glib2 2.36.3 will work for glib2 2.44... That would actually help with other project of mine, so I may take a crack at it...

[sgallagher]

@Conan-Kudo For the record, 2.44 is the oldest version that I think will work. (We require some features that definitely aren't present before that). I haven't actually tested to make certain. It's been a month since I did that digging. If you were going to go through the effort of grabbing a newer version, I'd try to go for the gold and match 2.56 from EL7, which we know for certain will work. But if that's infeasible, let me know what the newest version you could manage is and I'll try to see if I can make it work.

[hroncok]

JFYI the manylinux2014 wheel platform tag is supported by pip 19.3+. It is based on CentOS 7 and also supports a wide scale of architectures.

[dralley]

@hroncok , thanks, great news!

[dralley]

This is now blocked on an ecosystem issue https://github.com/pypa/auditwheel/issues/230 but can move forwards once that is resolved.

[wohali]

@dralley Hello, it appears https://github.com/pypa/auditwheel/issues/230 has been fixed. Does that unblock merging your fixes for this?

[dralley]

In the sense that it builds, yes, in the sense that it works as intended, no.

https://github.com/pypa/manylinux/issues/823

tl;dr createrepo_c uses the "magic number" database located at /usr/share/misc/magic to determine the types of various kinds of files. It uses libmagic to parse that database. When you build a cross-platform Python wheel, it bundles all the libraries that it links against into the package, including libmagic.

(libmagic) https://github.com/file/file

The old version that gets bundled (from CentOS 7) can't parse the /usr/share/misc/magic databases for more recent linux distros, so the createrepo_c binary wheel works on older distros, but it doesn't work on newer distros.