Fixes #36902 - Support Pulpcore 3.39, drop older versions

[ianballou]

Assumes that https://github.com/pulp/pulpcore/issues/4679 is in Pulpcore 3.39.

Also assumes that https://github.com/pulp/pulpcore/issues/4681 is going to be fixed for Pulpcore 3.39 in time. This doesn't affect the installer but we shouldn't release Katello with that Pulpcore bug.

I had to assume that the entry point will live in /usr/bin since, when installed via pip, it is installed to /usr/local/bin.

I verified these work (minus the jitter for now):

ExecStart=/usr/local/bin/pulpcore-api --preload --timeout 90 --workers 5 --max-requests 800 --access-logfile - --access-logformat 'pulp [%({correlation-id}o)s]: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"'

and

ExecStart=/usr/local/bin/pulpcore-content  --preload --timeout 90  --workers 5 --access-logfile -

[ianballou]

I haven't touched the tests yet.

[ianballou]

I suppose this is going to break all of the older Pulp tests?

[ianballou]

@ekohl I updated all of those files including the README.

[ianballou]

[test]

[ianballou]

Actually I don't think we can test just yet, the packages are available but as nightly https://yum.theforeman.org/pulpcore/nightly/el8/

[evgeni]

I've just merged https://github.com/theforeman/puppet-pulpcore/pull/318, if you rebase and add "nightly" to the matrix, at least that we should see working

[ianballou]

@evgeni I've added nightly

[evgeni]

  missing groups or modules: pulpcore:el8

this seems odd… cc @ehelms @odilhao

[evgeni]

and on el9 it tries to fetch gpg keys, fails (obviously) and bails out

  Error: /Stage[main]/Pulpcore::Install/Package[pulpcore]/ensure: change from 'purged' to 'present' failed: Execution of '/usr/bin/dnf -d 0 -e 1 -y install pulpcore' returned 1: Status code: 404 for https://yum.theforeman.org/pulpcore/nightly/GPG-RPM-KEY-pulpcore (IP: 151.101.42.49)

[ianballou]

I just found out that Python 11 doesn't work with module streams, so there will be no modules. This might cause some bigger issues, at the very least our docs will need updating...

[evgeni]

Oooh, you're saying Pulpcore is also demodularized now?!

[ianballou]

Oooh, you're saying Pulpcore is also demodularized now?!

Seems like it, I just found out a couple minutes ago. Sounds like it's not by choice, but by how Python 3.11 works. Sounds like we could still ship the modular metadata if removing it is too troublesome.

[evgeni]

Well, just because Python is not shipped in a module anymore doesn't mean we can't ship (no Android, not shit, I mean sure, in a way, but no) Pulpcore as a module. That being said, there is no reason for us to now. I think this is fine and we can just drop the module stuff from repo.pp.

[ekohl]

I didn't review the last version, but I opened the question whether pulpcore needs modularity given Python 3.11 is just a regular package. That would needs updates, which makes the whole Pulp upgrade from a platform perspective an even larger risk than it already was, but cleaner

[ianballou]

Would it only be the docs and this puppet-pulpcore repo that will need updates to remove the modular expectation?

[evgeni]

That would be my naive assumption.

Maybe foreman operations collection, don't recall how we set up Pulpcore there.

[ianballou]

I've pushed a change to drop the modules, let's see if I did it right.

[evgeni]

https://github.com/search?q=org%3Atheforeman%20%22pulpcore%3Ael8%22&type=code Agrees

https://github.com/search?q=org%3Atheforeman%20%22pulpcore%3A%22&type=code adds foreman maintain

So not a super crazy list.

The most interesting question will be: what does dnf think when a package (pulpcore-selinux, e.g.) came from a module (in 3.28) and now does not. I recall dnf having some very odd way of memorizing module metadata for repositories that are gone. Things to try out.

[ianballou]

Also the nightly gpg keys live somewhere new: https://download.copr.fedorainfracloud.org/results/@theforeman/pulpcore-nightly-staging/pubkey.gpg

I guess we need a different repo spec for nightly since the GPG key is elsewhere?

[Odilhao]

The GPG key and the repo name for 3.39 will be same as usual

[ianballou]

Tests are now passing for EL8 nightly :)

[evgeni]

It's something.jpg

Seems pulp cli is broken on el9. But it's a start.

[evgeni]

3.39 is obviously failing right now, but other than that this looks fine to me!

[Odilhao]

3.39 is obviously failing right now, but other than that this looks fine to me!

It should be able to get released after theforeman/forklift#1731

[evgeni]

3.39 is obviously failing right now, but other than that this looks fine to me!

It should be able to get released after theforeman/forklift#1731

I merged that, but centos ci is broken right now, so you won't be able to do a release pipe :(

[ianballou]

Nightly GPG key is an issue again: Error: /Stage[main]/Pulpcore::Install/Package[pulpcore]/ensure: change from 'purged' to 'present' failed: Execution of '/usr/bin/dnf -d 0 -e 1 -y install pulpcore' returned 1: Status code: 404 for https://yum.theforeman.org/pulpcore/nightly/GPG-RPM-KEY-pulpcore (IP: 146.75.94.49)

[ianballou]

I'm attempting to remove the GPG key requirement for nightly...

[ianballou]

Hmm still erroring Error: /Stage[main]/Pulpcore::Install/Package[pulpcore]/ensure: change from 'purged' to 'present' failed: Execution of '/usr/bin/dnf -d 0 -e 1 -y install pulpcore' returned 1: Status code: 404 for https://yum.theforeman.org/pulpcore/nightly/GPG-RPM-KEY-pulpcore

[ianballou]

Test-wise I think this is good to go -- the only failing bits are Pulp-CLI on EL9 with the 3.39 repo. Nightly is working on EL9, and everything EL8 works.

[evgeni]

So I tried an upgrade by installing 3.28 with the module currently on Forge and then upgrading to 3.39 with this PR here.

Generally, things seem to work (well, a pulp status is happy, this was my sole test :D) after applying pulpcore::repo from this PR, running dnf upgrade and applying pulpcore and pulcore::cli afterwards.

[ekohl]

I realize now that I only merged this part, which will end up in nightly but I can't merge https://github.com/Katello/katello/pull/10791 because I don't have permissions.

[ekohl]

Everything related is now merged

[Odilhao]

The Katello release failed for nightly, and it's related to the installer, probably it was not released yet on the nightly package:

[2023-11-24T20:42:14.474Z]     Error 1: Puppet Service resource 'pulpcore-api.service' failed. Logs:

[2023-11-24T20:42:14.474Z]       /Service[pulpcore-api.service]

[2023-11-24T20:42:14.474Z]         Starting to evaluate the resource (1386 of 1453)

[2023-11-24T20:42:14.474Z]         Skipping restart; service is not running

[2023-11-24T20:42:14.474Z]         Triggered 'refresh' from 3 events

[2023-11-24T20:42:14.474Z]         The container Systemd::Unit_file[pulpcore-api.service] will propagate my refresh event

[2023-11-24T20:42:14.474Z]         Evaluated in 1.14 seconds

[2023-11-24T20:42:14.474Z]       /Stage[main]/Pulpcore::Service/Pulpcore::Socket_service[pulpcore-api]/Systemd::Unit_file[pulpcore-api.service]/Service[pulpcore-api.service]/ensure

[2023-11-24T20:42:14.474Z]         change from 'stopped' to 'running' failed: Systemd start for pulpcore-api.service failed!

[2023-11-24T20:42:14.474Z]     journalctl log for pulpcore-api.service:

[2023-11-24T20:42:14.474Z]     -- Logs begin at Fri 2023-11-24 20:25:45 UTC, end at Fri 2023-11-24 20:42:05 UTC. --

[2023-11-24T20:42:14.474Z]     Nov 24 20:42:04 pipe-katello-server-nightly-centos8-stream.example.com systemd[1]: Starting Pulp API Server...

[2023-11-24T20:42:14.474Z]     Nov 24 20:42:05 pipe-katello-server-nightly-centos8-stream.example.com pulpcore-api[44109]: Error: This app must be executed using pulpcore-api entrypoint.

[2023-11-24T20:42:14.474Z]     Nov 24 20:42:05 pipe-katello-server-nightly-centos8-stream.example.com systemd[1]: pulpcore-api.service: Main process exited, code=exited, status=1/FAILURE

[2023-11-24T20:42:14.474Z]     Nov 24 20:42:05 pipe-katello-server-nightly-centos8-stream.example.com systemd[1]: pulpcore-api.service: Failed with result 'exit-code'.

[2023-11-24T20:42:14.474Z]     Nov 24 20:42:05 pipe-katello-server-nightly-centos8-stream.example.com systemd[1]: Failed to start Pulp API Server.

[ekohl]

I investigated this and wrote it down in https://github.com/theforeman/jenkins-jobs/pull/382#issuecomment-1826124215. My initial theory was wrong, and it appears things aren't synced from staging to release.

[evgeni]

But the pipes should be using staging anyway. So it shouldn't matter if staging>prod sync is broken?

And https://ci.theforeman.org/job/foreman-nightly-rpm-pipeline/2099/ was ree, which explains not synced prod?

[evgeni]

https://ci.theforeman.org/job/foreman-nightly-rpm-pipeline/2099/consoleFull ran at 9:50PM and pushed the new installer but https://ci.theforeman.org/job/katello-nightly-rpm-pipeline/1837/ ran at 8:18PM and thus just didn't have the new installer (katello doesn't generate foreman staging, but the installer is in foreman).

but when you looked, staging was updated already and confused you :)

[evgeni]

and now installs passed fine.

tests are failing tho, see https://community.theforeman.org/t/katello-nightly-rpm-pipeline-1838-failed/35929/2