search

theforeman / foreman-infra

Puppet modules and scripts to manage Foreman project infrastructure
https://theforeman.github.io/foreman-infra
Apache License 2.0
23 stars 51 forks source link

Investigate Copr as the build system #1795

Closed evgeni closed 9 months ago

evgeni commented 1 year ago

Open questions:

ehelms commented 1 year ago

After some refreshing of Copr knowledge, digging into current availability, I have put together an initial draft of what our usage of Copr would look like. As acknowledged, there are some open questions around how we configure it, and tooling choices. In order to migrate to Copr, the biggest initial undertaking would be to get the tooling in place. Our previous Copr tooling relied on tito and that would need to be re-vamped.

Copr Usage Design

Copr provides the idea of a project that represents one or more repositories which are defined by the chroots (e.g. RHEL, Centos stream) included in the project.

The Foreman project would use Copr as a build system and locationg for staging repositories that are tested before being pulled to yum.theforeman.org.

Managing Configuration

In order to manage the configuration, this can be stored within the foreman-packaging repository package_manifest.yaml or a new file and propagated to Copr either through Obal functionality or a stand-alone script. This will allow storing the configuration alongside the packages and facilitate keeping them in sync when branching.

Repositories

Nightly Server

Client

Modules

Copr can generate modulemd during repository generation based upon configuration through Copr or by uploading an existing module metadata. This will allow continuity of current modules.

https://docs.fedoraproject.org/en-US/modularity/building-modules/copr/building-modules-in-copr/#_submit_an_existing_modulemd_yaml

Branching

Scratch Builds

Copr does not have the notion of scratch builds as Koji does, instead a new project that is a copy of the primary project must be created. The process, using foreman-nightly example:

There would be a single scratch build repo per pull request, thus all RPM builds in a PR would end up in the same temporary project. Repoclosure would be run against the split project, in theory giving a better repoclosure result.

Benefits from what we have today:

Tooling

Copr provides a few interfaces:

The python-copr library would allow easy integration and building of modules into Obal to support our current workflows through the same interface.

Comps

Comps can be uploaded to the project configuration and are scoped per chroot. This would mean any updates to comps will need to be identified and uploaded to update the chroot after merge.

Open Questions

evgeni commented 1 year ago

One thing that come up somewhere else:

Today we have a very crude way to build the foreman-discovery-image and don't ship it as an RPM upstream. Downstream does ship it as an RPM, but builds stuff differently (using koji/brew) and we see a variety of problems due to the differences. Copr does not support building ISOs and the recommended "solution" is image-builder, but I am not even sure image-builder would work for our usecase (my understanding is it's customizing the RHEL installer, while we actually want a live cd with own software).

This by no means should mean we shouldn't use Copr, just that we should also go and unify fdi builds, and it won't be on the current downstream solution if we use Copr.

evgeni commented 1 year ago

And to answer your open question: I think I prefer a layout as close to the current repo layout as possible, so no dedicated per OS projects. As you say, target chroots can be configured if needed and otherwise I think it's a fair thing to say that if one OS fails, the build as such is faulty.

If you're cautious about bootstrapping net-new OSes (like EL9), I think this can be either done in a separate bootstrap project and then copied over or by using the config to limit chroots more granularly.

evgeni commented 1 year ago

Last infra-sig you asked for explicit YAY/NAY on the design.

I'd have one question: the client repo currently has no nighty/release differentiation, which matches what we have downstream, but not upstream (right now). I'd be happy to change that, but also don't think that's something we should piggy-back on the copr change?

Overall I'm "YAY" tho :)

@ekohl @pcreech @Odilhao @zjhuntin any opinions here?

ekohl commented 1 year ago

I'm mentally going through the tasks that need to be done.

First of all: I think we said we want to maintain our yum.theforeman.org host so we'll need to figure out how we sync from COPR to our repositories. Today we use rsync, but not sure if COPR supports that (and if it's the best choice).

That also implies we consider COPR as staging repositories. How do we make it clear for users that they're not intended as real repositories? Do we include -staging- in the repo names?

Speaking of naming: should we use the chance to rename Katello to use the Foreman version numbering? So foreman-katello-3.6 instead of katello-4.8? That makes branching procedures easier since you can just replace nightly with the version number instead of figuring out which is which.

That brings us to branching. We'll need to replace the tooling we have to branch and update the procedure. Given we just branched, I do think it's the best time since I'd rather not do it under time pressure. One implication is that (at least for Foreman) https://github.com/theforeman/tool_belt becomes obsolete, which I think is a good thing.

During releasing we also sign RPMs with GPG, which relies on Koji. How do we replace this?

This is not a YAY/NAY on the design: I think it's incomplete and needs a bit of refinement.

  • foreman-nightly-el8 vs. foreman-nightly

My vote would be foreman-nightly.

ehelms commented 1 year ago

That also implies we consider COPR as staging repositories. How do we make it clear for users that they're not intended as real repositories? Do we include -staging- in the repo names?

Sounds like a good approach.

First of all: I think we said we want to maintain our yum.theforeman.org host so we'll need to figure out how we sync from COPR to our repositories. Today we use rsync, but not sure if COPR supports that (and if it's the best choice).

I'll work on tracking down this answer.

Speaking of naming: should we use the chance to rename Katello to use the Foreman version numbering? So foreman-katello-3.6 instead of katello-4.8? That makes branching procedures easier since you can just replace nightly with the version number instead of figuring out which is which.

+1

During releasing we also sign RPMs with GPG, which relies on Koji. How do we replace this?

I am asking about this one.

ehelms commented 1 year ago

During releasing we also sign RPMs with GPG, which relies on Koji. How do we replace this?

I am asking about this one.

The answer to this is that Copr can generate a key and sign packages with but does not have the ability to allow a key generated outside Copr to be used to sign. We can request this as a feature and see where we get. Generally, this seems like it should work for us, except for the fact that we also sign the tarballs.

ekohl commented 1 year ago

Technically I think we can maintain our current process where we sign manually for a release. That they're signed with another key in copr isn't really a problem.

Perhaps then the question becomes what we want to signal to users with it. If you say that it's about integrity and we've verified the bits then I'm not sure how true that really is.

And for Debian we sign automatically. Would the same be good enough for RPMs?

ehelms commented 1 year ago

To close this issue out, we'll establish the next steps here, open follow on issues and close this investigation.

ehelms commented 1 year ago

Making a note of a something we will have to handle:

ehelms commented 1 year ago

@evgeni @ekohl Could y'all read over https://github.com/fedora-copr/copr/issues/2782 and let me know your thoughts on the tradeoffs of the work the Copr team would have to do just for us versus us implementing the necessary steps ourselves until we get off EL8?

ehelms commented 1 year ago

Here is a proposal for how we handle modularity with Copr given modularity is deprecated and asking the Copr team to address this is a lot of work (see: https://github.com/fedora-copr/copr/issues/2782).

Process proposal:

  1. Builds are created in Copr for the respective project which is 1:1 to a repository we publish (e.g. https://copr.fedorainfracloud.org/coprs/g/theforeman/foreman-nightly/)
  2. The repository is copied to a new location: https://yum.stage.theforeman.org
  3. A script is ran on the stage repository to add modularity metadata
  4. Tests are ran against https://yum.stage.theforeman.org in our pipelines
  5. If tests pass, the stage repository is promoted to production https://yum.theforeman.org

This proposal will require us to create a new vhost and ensure we have enough storage on web02 to support the stage repositories.

Alternatively, we could continue to use http://koji.katello.org/releases/, however, this will mean it takes us longer to get off of the Koji infrastructure after we have migrated to Copr.

ekohl commented 1 year ago

I think a new name makes a lot of sense. More than trying to shoe horn it into Koji.

The repository is copied to a new location: https://yum.stage.theforeman.org

I think we should look at consistency with debs, so deriving from stagingdeb.theforeman.org I'm proposing stagingyum. But this is a minor detail.

A script is ran on the stage repository to add modularity metadata

AFAIK this is something we don't have on Koji now, but do have such a script on the real repos. Having those match would be a good thing for testing quality.

ensure we have enough storage on web02 to support the stage repositories.

Keeping old releases on Koji it doesn't matter much since they're hard links and we need to keep them anyway, but on web02 they'll have costs. I think we can also adopt cleaning of it. For example, we remove repos for old (unsupported) releases. May need to consider n-2 testing support, though I'd prefer to have the n-2 and n-1 consume the release repos.

Overall :+1: for the plan.

Odilhao commented 1 year ago

Sorry for the delay, with copr team providing the chroot, that means that we don't have control over the buildroot?

I'm trying to find the best design for our necessary packages for python, specially the effort to enable PEP-517 on EL8, my plan was to later create on branch or one project with all the spec files necessary to build the buildroot.

FrostyX commented 1 year ago

with copr team providing the chroot, that means that we don't have control over the buildroot?

It depends on what control you have in mind. Copr allows you to specify what packages and modules should be installed in the buildroot and what repositories should be enabled.

FrostyX commented 1 year ago

Add chroots: rhel-7-x86_64 (to be added by Copr team still) rhel-8-x86_64 (to be added by Copr team still) rhel-9-x86_64 (to be added by Copr team still)

Just for the record, those are already available in Copr.

ehelms commented 1 year ago

@Odilhao Here is the working example of the configuration for Foreman: https://github.com/theforeman/foreman-packaging/pull/9290/files

Initially, I have provided support for defining builldroot packages, modules and external repos in Obal: https://github.com/theforeman/obal/blob/master/obal/data/roles/copr_project/tasks/main.yaml#L18C14-L20

ehelms commented 1 year ago

Add chroots: rhel-7-x86_64 (to be added by Copr team still) rhel-8-x86_64 (to be added by Copr team still) rhel-9-x86_64 (to be added by Copr team still)

Just for the record, those are already available in Copr.

@FrostyX So we know in the future, are the buildroots based off of latest or .0 release? (e.g. 8.8 or 8.0)

FrostyX commented 1 year ago

@FrostyX So we know in the future, are the buildroots based off of latest or .0 release? (e.g. 8.8 or 8.0)

When I do this on our builders:

$ mock -r rhel-8-x86_64 --shell
<mock-chroot> sh-4.4# cat /etc/os-release

it says

VERSION="8.8 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.8 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.8
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.8"

Update: You can see the mock config and what repositories are used here: https://github.com/rpm-software-management/mock/blob/main/mock-core-configs/etc/mock/templates/rhel-8.tpl

ehelms commented 1 year ago

Follow up to the staging repository, we will need to decide where the work happens based on storage and parallelization concerns. I've outlined this in the PR that adds the script to do the generation for feedback:

https://github.com/theforeman/foreman-packaging/pull/9596#issuecomment-1662707704

ehelms commented 1 year ago

Let's talk about RPM signing with Copr. At this point we have a workflow that looks like:

Copy repository from Copr -> Add modularity -> Copy repository to stagingyum.theforeman.org -> Test

We have three options for packaging signing:

  1. All packages are automatically signed by Copr and we can choose to use the GPG key created for a given repository as our signing key for packages (we would need a separate one for any non-RPM assests to be signed)
  2. Generate stage repository, then halt the pipeline. Download RPMs locally from the staging repository, sign them, createrepo and then rsync them back to stagingyum and continue testing.
  3. Via a pipeline generate stage repository, copy to stagingyum on web01, sign packages on web01 automatically, createrepo update and then run tests.
ekohl commented 1 year ago

All packages are automatically signed by Copr and we can choose to use the GPG key created for a given repository as our signing key for packages (we would need a separate one for any non-RPM assests to be signed)

We have foreman-release, which stores the GPG key, allowing users to simply dnf install https://yum.theforeman.org/..../foreman-release.rpm. I like that workflow. That means we should set up a process that makes sure we have the correct key in foreman-packaging. Not too different from today.

One thing to keep in mind is that copr may have one GPG key per repo. So katello might be different from foreman and yet different from plugins then client is yet another key. That would argue for option 2 from an end user perspective.

Generate stage repository, then halt the pipeline. Download RPMs locally from the staging repository, sign them, createrepo and then rsync them back to stagingyum and continue testing.

This feels error prone. Because if you have:

Then the next time you sync in new files from copr, you will overwrite the already signed files with copr signed files. Today koji takes care of that for us and I'm not sure how I'd rewrite the logic.

Via a pipeline generate stage repository, copy to stagingyum on web01, sign packages on web01 automatically, createrepo update and then run tests.

I'm not sure I like this approach, because in my mind GPG signing is a fixed point where you "bless" the files that they went through the proper channels. If it's automated, there's no guarantee at all anymore and supply chain attacks may be easier. On the other hand, you could argue that today we barely inspect the RPM files either.

Overall I'm still torn. I was slightly leaning to option 1, but having multiple GPG keys for a single release is tedious. We'll need to expand our process to publish them so users can securely sync them (to systems like Pulp). Perhaps the copr team can say if it's technically possible to use a key between multiple repositories.

ehelms commented 1 year ago

having multiple GPG keys for a single release is tedious. We'll need to expand our process to publish them so users can securely sync them (to systems like Pulp). Perhaps the copr team can say if it's technically possible to use a key between multiple repositories.

@FrostyX what do you think? Any related experiences you have seen for Copr users?

ehelms commented 1 year ago

From some initial testing, it does not appear that comps support in Copr does what we expect of comps in our traditional method so I've reached out to ask if there is a bug here or a native way to exclude packages. This may be another case where we need to handle this filtering during stage repository creation similar to modularity.

Tracking issue for Copr -- https://github.com/fedora-copr/copr/issues/2901

ehelms commented 1 year ago

For the time being I have proposed doing the comps filtering in our stage generation script -- https://github.com/theforeman/theforeman-rel-eng/pull/275

Another Option

By splitting out repositories based on comps during stage generation, this does open up the possibility to have a single build repository that is then split out into end-user repositories. This would solve the single GPG key issue as well since all packages would build into a single repository signed by a single Copr GPG key.

Proposal for using our own key

If we stick with our own key, I'd propose the following workflow for releases process:

Stage 1 Generate stage repository -> push to stagingyum -> Run installation tests -> STOP

Stage 2 Release engineer reposync's repository from stagingyum -> signs packages -> rsyncs back to stagingyum -> update repo -> run signature verification -> push RPMs to production

FrostyX commented 1 year ago

having multiple GPG keys for a single release is tedious. We'll need to expand our process to publish them so users can securely sync them (to systems like Pulp). Perhaps the copr team can say if it's technically possible to use a key between multiple repositories.

@FrostyX what do you think? Any related experiences you have seen for Copr users?

Hello @ehelm, I brought the topic up at a Copr team meeting and found out that there were some discussions in the past about the scope for which GPG keys are used. But the conclusion always was that having a different GPG key for every project might not be ideal in some use cases but the decision has already been made and the change would be too expensive to make it worth it.

One of the discussed options was uploading someone's GPG keys to be used for their Copr namespace. We don't want this - I can elaborate if necessary.

The other option was to have a checkbox in user/group settings that would make all projects in that namespace use the same GPG key (which would be generated the same way other Copr GPG keys are but it would be assigned to the user/group namespace instead of a project). We can implement this feature if it is a hard requirement for you but it will probably take some time.

Tagging @xsuchy and @praiskup in case they want to add something.

ehelms commented 1 year ago

One of the discussed options was uploading someone's GPG keys to be used for their Copr namespace. We don't want this - I can elaborate if necessary.

Nah, we understand this one, managing the security of this would be painful at best.

The other option was to have a checkbox in user/group settings that would make all projects in that namespace use the same GPG key (which would be generated the same way other Copr GPG keys are but it would be assigned to the user/group namespace instead of a project). We can implement this feature if it is a hard requirement for you but it will probably take some time.

I think for our use case this would be too broad. I would think of it having to be more like:

  1. Interface to tell Copr to generate a GPG key
  2. See all of my GPG keys in a list
  3. Be able to assign a GPG key to a project selectively

I would equate this to creating tokens in Github where I can assign different permissions to different tokens and manage them.

We are a bigger, possibly more unique in this respect project so I understand not undertaking implementing all of this if you do not have other stakeholders asking for this.

ehelms commented 1 year ago

After some further thought, I am leaning towards this workflow.

For Releases

  1. Release engineer generates stage repositories locally.
  2. Release engineer signs the RPMs locally and generates the repository.
  3. Release engineer runs a script to rsync local repo to stagingyum.
  4. Release engineer kicks off pipeline.

For Nightly

  1. Jenkins generates stage repository.
  2. Jenkins rsyncs to stagingyum.
  3. Jenkins runs tests.

@ekohl @evgeni

ekohl commented 1 year ago

I also thought about this and it does make sense, but we don't want to overwrite files which were already signed. I wonder how we prevent that.

Practical use case: when we release Foreman 3.9.0-rc2, how do we prevent rewriting all the packages which were already signed when we released RC1?

With koji we have download_rpms where we list all tagged packages, including their sigs and any packages not signed with the correct key is downloaded.

Can we download the production repo metadata, list all RPMs and then do the same for the staging repo. All files on staging that aren't in production should be downloaded and signed. Is that feasible?

ehelms commented 1 year ago

An example using repodiff to identify only changed packages:

$ repodiff --repofrompath old,https://stagingyum.theforeman.org/foreman/nightly/el8/x86_64/ --repofrompath new,https://download.copr.fedorainfracloud.org/results/@theforeman/foreman-nightly-staging/rhel-8-x86_64/ --repo-old old --repo-new new --refresh --archlist x86_64,noarch,src --simple
Added old repo from https://stagingyum.theforeman.org/foreman/nightly/el8/x86_64/
Added new repo from https://download.copr.fedorainfracloud.org/results/@theforeman/foreman-nightly-staging/rhel-8-x86_64/
old                                                                                                          24 kB/s | 3.5 kB     00:00    
new                                                                                                          17 kB/s | 2.3 kB     00:00    
Added package  : nodejs-tslib-2.6.2-1.el8
Added package  : python-websockify-0.10.0-3.el8

Modified packages
nodejs-babel-core-7.22.10-1.el8 -> nodejs-babel-core-7.22.17-1.el8
nodejs-graphql-tag-2.11.0-1.el8 -> nodejs-graphql-tag-2.12.6-1.el8
nodejs-node-gyp-6.1.0-9.el8 -> nodejs-node-gyp-6.1.0-10.el8
nodejs-node-sass-4.13.1-1.el8 -> nodejs-node-sass-4.14.1-2.el8
nodejs-react-ellipsis-with-tooltip-1.0.8-4.el8 -> nodejs-react-ellipsis-with-tooltip-1.1.1-1.el8
nodejs-theforeman-vendor-12.0.1-1.el8 -> nodejs-theforeman-vendor-12.2.0-1.el8
nodejs-uuid-3.3.2-4.el8 -> nodejs-uuid-3.4.0-1.el8
rubygem-autoprefixer-rails-10.4.13.0-1.el8 -> rubygem-autoprefixer-rails-10.4.15.0-1.el8
rubygem-autoprefixer-rails-doc-10.4.13.0-1.el8 -> rubygem-autoprefixer-rails-doc-10.4.15.0-1.el8
rubygem-css_parser-1.14.0-1.el8 -> rubygem-css_parser-1.16.0-1.el8
rubygem-css_parser-doc-1.14.0-1.el8 -> rubygem-css_parser-doc-1.16.0-1.el8
rubygem-excon-0.100.0-1.el8 -> rubygem-excon-0.102.0-1.el8
rubygem-excon-doc-0.100.0-1.el8 -> rubygem-excon-doc-0.102.0-1.el8
rubygem-execjs-2.8.1-1.el8 -> rubygem-execjs-2.9.0-1.el8
rubygem-execjs-doc-2.8.1-1.el8 -> rubygem-execjs-doc-2.9.0-1.el8
rubygem-foreman_maintain-1:1.3.5-1.el8 -> rubygem-foreman_maintain-1:1.4.0-1.el8
rubygem-foreman_maintain-doc-1:1.3.5-1.el8 -> rubygem-foreman_maintain-doc-1:1.4.0-1.el8
rubygem-globalid-1.1.0-1.el8 -> rubygem-globalid-1.2.1-1.el8
rubygem-globalid-doc-1.1.0-1.el8 -> rubygem-globalid-doc-1.2.1-1.el8
rubygem-kafo-7.0.0-1.el8 -> rubygem-kafo-7.1.0-1.el8
rubygem-kafo-doc-7.0.0-1.el8 -> rubygem-kafo-doc-7.1.0-1.el8
rubygem-mime-types-3.5.0-1.el8 -> rubygem-mime-types-3.5.1-1.el8
rubygem-mime-types-doc-3.5.0-1.el8 -> rubygem-mime-types-doc-3.5.1-1.el8
rubygem-pg-1.5.3-1.el8 -> rubygem-pg-1.5.4-1.el8
rubygem-pg-debuginfo-1.5.3-1.el8 -> rubygem-pg-debuginfo-1.5.4-1.el8
rubygem-pg-debugsource-1.5.3-1.el8 -> rubygem-pg-debugsource-1.5.4-1.el8
rubygem-pg-doc-1.5.3-1.el8 -> rubygem-pg-doc-1.5.4-1.el8
rubygem-sequel-5.71.0-1.el8 -> rubygem-sequel-5.72.0-1.el8
rubygem-sequel-doc-5.71.0-1.el8 -> rubygem-sequel-doc-5.72.0-1.el8
rubygem-sprockets-4.2.0-1.el8 -> rubygem-sprockets-4.2.1-1.el8
rubygem-sprockets-doc-4.2.0-1.el8 -> rubygem-sprockets-doc-4.2.1-1.el8

Summary
Added packages: 2
Removed packages: 0
Modified packages: 31
ekohl commented 1 year ago

That looks hard to parse and you would need to glob to get exact paths, but it is generally the idea that would work.

ehelms commented 1 year ago
  • Release engineer runs a script to rsync local repo to stagingyum.

How would we handle this? Right now only the yumrepostage user has the rsync script available. Would we deploy something for each user on web01 to be able to rsync signed RPMs to it and then that script puts them in the right place? And I suppose add a createrepo step there.

praiskup commented 1 year ago

One of the discussed options was uploading someone's GPG keys to be used for their Copr namespace. We don't want this - I can elaborate if necessary.

This feature would be relatively expensive for implementation, and the question is whether it is worth it. We could, IMO, manually upload keys if a few teams really needed it.

I think for our use case this would be too broad. I would think of it having to be more like:

  1. Interface to tell Copr to generate a GPG key
  2. See all of my GPG keys in a list
  3. Be able to assign a GPG key to a project selectively

This is an interesting idea.

After some further thought, I am leaning towards this workflow.

Seems like you propose GPG signatures outside of Copr, while the building stays in Copr. Correct? How much effort we would save you if we implemented the GPG key handling mechanism, or at least uploaded a custom signature upon your request in the meantime? (this is somewhat needed before we start working on something non-trivial).

Odilhao commented 11 months ago

One question, do we have any quota/limit rate while building on copr? I'm building one projectwith 100+ packages, I'm releasing locally to control the build order, looks like one build got stuck and never started, it's on the same status for 4 hours and counting.

FrostyX commented 11 months ago

looks like one build got stuck and never started, it's on the same status for 4 hours and counting.

We had issues with spawning new builders because subscription-manager was timeouting. @xsuchy already resolved it. There is a bit of a queue but builds are running again.

ekohl commented 11 months ago

https://github.com/orgs/theforeman/projects/6 lists various tasks.

ehelms commented 10 months ago

Two things i am noticing that I don't have a clear understanding of:

  1. Even if rsync does not push an updated file, it appears to trigger a change to the last modified date (e.g. https://stagingyum.theforeman.org/plugins/nightly/el8/x86_64/)

  2. Some files are getting cached I assume by Fastly and if those packages change, they are not getting updated from the webservers perspective. This results in the RPM metadata having one checksum and the file being downloaded having another. Example:

ehelms commented 9 months ago

This is the build system now! This is active for nightly and 3.9, so I will close this issue.

Odilhao commented 9 months ago

Investigation completed

evgeni commented 9 months ago

No idea why, but I've read "This is the build system now!" in @zjhuntin's voice.