Add support for fetching RPMs from a registry

[cgwalters]

This project is making a potential transition to being "dnf image" where we default to fetching the base OS from a registry, and we have opinionated support for things like dnf image build that creates a container image in a sane reproducible way.

But there's another really interesting bridge we can make - there's recently an effort called "OCI artifacts" https://github.com/opencontainers/artifacts and people are already experimenting with storing things like debs/RPMs natively in a registry.

Now that we're linking to and using https://github.com/containers/containers-image-proxy-rs - it seems very, very natural to:

• extend skopeo and the proxy to support fetching OCI artifacts
• Design an equivalent of createrepo_c that instead stores data into a registry
• Teach the client how to use this

For example, we might have:

$ cat /etc/yum.repos.d/fedora.repo
[fedora]
name=Fedora $releasever
registry=quay.io/fedora/everything:$releasever
enabled=1
countme=1
metadata_expire=7d
repo_signed=1
type=rpm-registry
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch
skip_if_unavailable=False

The biggest thing here is to notice registry=quay.io/fedora/everything:$releasever instead of the current metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-$releasever&arch=$basearch.

Notice that things like basearch are now handled natively via manifest listing. I also put in here a strawman of repo_signed=1 to require that the remote container image is e.g. signed with cosign/sigstore or another mechanism known by containers/image.

If we do this, we could just stick the existing XML as-is into the registry, but since we're doing this transition it's also probably time to define a saner metadata format...that's its own topic, and xref https://github.com/coreos/rpm-ostree/issues/1127

Benefits

• RPM repositories can be versioned in the same way as container images! This is huge. Today for example Fedora derivatives have this idea of a "compose" which is just a set of hardlinked RPMs served by a webserver. But yum/dnf have no idea about that. But suddenly it'd be possible to do e.g. change the value in the repo file to e.g. registry=quay.io/fedora/everything:20221118.n.0 (note a tagged version number) and that would just work.
• Authenticating to RPM repositories can use a container pull secret! (Instead of e.g. client certificates)
• RPM repositories can be mirrored via OCI artifact aware tooling, which can also mirror other things
• (As noted above) RPM repositories can be signed via sigstore/cosign and not GPG
• Registries like quay.io have nice support for things like viewing tag history that applies here too - operating system builds don't just look like stock Apache directory listings!
• Support for disconnected installations can actually just reuse the containers/image support for this! Instead of editing .repo files on the client, one can use remapping and mirroring registries

[j-mracek]

Please don't do that. It breaks DNF. PLEASE DON'T!

Error: Repository 'fedora-registry' has unsupported type: 'type=rpm-registry', skipping.

Please do not add any new type of repository definition in /etc/yum.repos.d, /etc/yum/repos.d, or /etc/distro.repos.d. The registry repository type is not yum/dnf type of repository!

I also do not recommend to use .repo file. .repo file are not defined and adding new staff there makes the situation even worse.

[j-mracek]

This project is making a potential transition to being "dnf image" where we default to fetching the base OS from a registry, and we have opinionated support for things like dnf image build that creates a container image in a sane reproducible way.

The proposal can be used as an another argument why using dnf image syntax with RPM-OSTREE is bad idea. DNF image is used as an argument to add new and incompatible content in paths that is used by DNF. dnf image idea is a logic but it will create a confusion, problems, incompatibility and so on. Users uses our tooling by the way we did not planned, therefore we can expect variety of problems that we cannot predict.

[cgwalters]

Please don't do that. It breaks DNF. PLEASE DON'T!

Sure, but we probably wouldn't be using "traditional dnf" in places where this is used. That said, nothing stops us from also shipping this functionality there...and that leads into a big topic of code sharing.

[cgwalters]

I should have said: first, thank you for replying here! I think it's quite important to have ongoing conversations in this area. That said, this particular issue is not an immediate priority; it's speculative/aspirational and clearly subject to change. So we don't need to have any kind of nuanced debate I think.

This all said on a higher level, I think your feedback has rather consistently been that everything we're trying to do is basically "something different, not dnf". But the problem I see with that is that we're doing some of these things because they provide real value. We can't assume we'll continue to ship software in RPMs with metadata stored in XML served by httpd for the next 20 years. We do need to maintain compatibility with what we did in the past. But we also need to evolve - but again in a way which allows users and scripts to retain at least some compatibility.

In that vein, I think "traditional dnf would error out on this modification to the repo file" is absolutely valid feedback.

[cgwalters]

One more different way to say this: I think I agree that were we to actually consider spending some engineering time on this, it'd probably make sense to transfer this issue to e.g. rpm-software-management and discuss with a wider group; it doesn't just impact rpm-ostree/dnf-image.

But...the flip side is: we are definitely going in the direction of fetching the base operating system as a container image, not as a set of RPMs. And when we do that, the motivation for this is significantly stronger, because it allows mirroring and managing the base OS in the same way as "extensions".

But again, I think the benefits of rpms-in-registries are pretty significant even outside of that...recently the fedora mirrormanager server has been giving fairly frequent 503 errors which cause flakes in CI and OS updates. If we were storing in a registry like quay.io, we suddenly have only one infrastructure to maintain, not two.

[j-mracek]

Please don't do that. It breaks DNF. PLEASE DON'T!

Sure, but we probably wouldn't be using "traditional dnf" in places where this is used. That said, nothing stops us from also shipping this functionality there...and that leads into a big topic of code sharing.

I am sorry, may be my comment was misunderstand. I don't want to say that the proposal for additional distribution channel is wrong. In DNF we have very strong requirements for legacy user cases. In general we expect that people are creative therefore we always expect that things appears in a way that it was not planned. I say not if something happen but only how often or when. Even if DNF will support registry type of repository in future, we cannot use path /etc/yum.repos.d to store the definition for them, because it will break access from older software management. This is something that is not rare in standard deployment. May be I sound paranoiac but with any new feature in DNF I start to think whether it can break something. And when I discover a potential issue I try to change the design to avoid any risk. I also remember that someone created a repository with unknown type of repository to DNF (unknown name of supported type) and such an issue is not nice. People cannot upgrade at all and fire is on the DNF roof.

I am happy to see new ideas around and new opportunity for cooperation. On the other side we cannot close any current user cases as minor, not modern or obsoleted.

[cgwalters]

xref https://github.com/reproducible-containers/repro-get