search

theforeman / foreman-infra

Puppet modules and scripts to manage Foreman project infrastructure
https://theforeman.github.io/foreman-infra
Apache License 2.0
23 stars 51 forks source link

Proposal: Update yum repository layout to be per repository #1937

Open ehelms opened 11 months ago

ehelms commented 11 months ago

Background

Currently the yum.theforeman.org repository layout follows an almost common model with Foreman being the exception:

client/<version>
client/nightly
client/latest
katello/<version>/katello
katello/<version>/candlepin
katello/nightly/katello
katello/nightly/candlepin
latest/
nightly/
plugins/<version>
plugins/nightly
plugins/latest
pulpcore/<version>
releases/<version>

Where:

Proposal

That the repositories get aligned under a common naming and structure to make them more predictable, easier to code against and allow better matching with stagingyum and our layout that is coming from Copr.

client/<version>
client/nightly
client/latest
foreman/<version>
foreman/nightly
foreman/latest
katello/<version>
katello/nightly
katello/latest
candlepin/<version>
candlepin/nightly
candlepin/latest
plugins/<version>
plugins/nightly
plugins/latest
pulpcore/<version>

In addition to this being the structure, we would retain (for some period of time) legacy symlinks:

releases/ -> foreman/
latest/ -> foreman/latest
ekohl commented 11 months ago

I'm wondering if we should instead go for versions as top level, then the version specific repos. So:

$ mkdir -p {3,8,3.9,nightly}/{client,foreman,plugins,katello}
$ ln -s 3.8 latest
$ tree
.
├── 3.8
│   ├── client
│   ├── foreman
│   ├── katello
│   └── plugins
├── 3.9
│   ├── client
│   ├── foreman
│   ├── katello
│   └── plugins
├── latest -> 3.8
└── nightly
    ├── client
    ├── foreman
    ├── katello
    └── plugins

We can also store the per-release GPG key in the directory (which would be missing in nightly). That way you immediately know which Katello release belongs to which Foreman release.

ehelms commented 11 months ago

What would you do with Pulpcore and Candlepin? For consistency, should we do by project and then version? e.g.

├── candlepin
│   ├── 4.1
│   └── 4.2
├── foreman
│   ├── 3.8
│   │   ├── client
│   │   ├── foreman
│   │   ├── katello
│   │   └── plugins
│   ├── 3.9
│   │   ├── client
│   │   ├── foreman
│   │   ├── katello
│   │   └── plugins
│   └── nightly
│       ├── client
│       ├── foreman
│       ├── katello
│       └── plugins
└── pulpcore
    ├── 3.22
    ├── 3.28
    └── nightly
ekohl commented 11 months ago

Yes, that would be best. Though for candlepin aren't they essentially also tied to a Foreman release? Today they're for a Katello release and Katello is essentially for a Foreman release. Or do you plan to maintain it more like pulpcore? Which would also imply a pulpcore-packaging equivalent.

Maybe that's a good rule: -packaging repository gets a top level directory. Each rpm/ branch in that repository is the second level directory, where rpm/develop maps to rpm/nightly. Each packages/* directory within that branch gets a third level directory.

Pulpcore doesn't have rpm/develop so there wouldn't be a pulpcore/nightly. It also doesn't split packages into multiple sub-directories so it doesn't get a third level directory.

ehelms commented 11 months ago

Yes, that would be best. Though for candlepin aren't they essentially also tied to a Foreman release? Today they're for a Katello release and Katello is essentially for a Foreman release. Or do you plan to maintain it more like pulpcore? Which would also imply a pulpcore-packaging equivalent.

Given it's one package I don't intend to management in the same way as our packaging repository. But I do think it would be convenient for us to treat Candlepin as stand-alone and released based on it's version to allow updates to Candlepin that can then be tested without the risk of breaking nightly (or a release branch). We could push updates to Candlepin, run for example our puppet test suite or something equivalent and know the package changes are "good" before we promote them into Katello.

Pulpcore doesn't have rpm/develop so there wouldn't be a pulpcore/nightly. It also doesn't split packages into multiple sub-directories so it doesn't get a third level directory.

Doesn't change things, just an FYI, as of today, this is no longer true -- https://github.com/theforeman/pulpcore-packaging/tree/rpm/develop @Odilhao is working on nightly builds for Pulpcore

ekohl commented 11 months ago

I'm mostly trying to find a good way to describe a "policy".

Given it's one package I don't intend to management in the same way as our packaging repository. But I do think it would be convenient for us to treat Candlepin as stand-alone and released based on it's version to allow updates to Candlepin that can then be tested without the risk of breaking nightly (or a release branch). We could push updates to Candlepin, run for example our puppet test suite or something equivalent and know the package changes are "good" before we promote them into Katello.

Candlepin has always been an exception. It has never had its own specs in our packaging repository. In that sense, it's not surprising it needs its own policy. I'm not opposed to this. Taking it to puppet-candlepin, we could easily have a policy very similar to puppet-pulpcore. If the repo can be consumed standalone, that would simplify testing.

Doesn't change things, just an FYI, as of today, this is no longer true -- https://github.com/theforeman/pulpcore-packaging/tree/rpm/develop

Ah, I missed that. Good to know.

ehelms commented 11 months ago

Candlepin has always been an exception. It has never had its own specs in our packaging repository. In that sense, it's not surprising it needs its own policy. I'm not opposed to this. Taking it to puppet-candlepin, we could easily have a policy very similar to puppet-pulpcore. If the repo can be consumed standalone, that would simplify testing.

That is the tactic I am taking starting from the Copr side as there will be a dedicated build repo for it -- https://copr.fedorainfracloud.org/coprs/g/theforeman/foreman-candlepin-nightly-staging/

ehelms commented 11 months ago

I thought more about, and started testing the Candlepin part and realized it needed it's own repository even if that repo was realllly small and didn't actually have packages. To that end, I created https://github.com/theforeman/candlepin-packaging to allow me to work through this concept.

evgeni commented 6 months ago