Add support for pulp for RPM repository management

[cottsay]

Here are some examples of what the corresponding config changes look like:

hiera/hieradata/buildfarm_role/master.yaml:

@@ -5,6 +5,10 @@ credentials::jenkins-slave::username: jenkins-agent
 credentials::jenkins-slave::id: 1e7d4696-7fd4-4bc6-8c87-ebc7b6ce16e5
 credentials::jenkins-slave::passphrase: 4lRsx/NwfEndwUlcWOOnYg==

+credentials::pulp-api::username: admin
+credentials::pulp-api::id: 0ddb9953-40d3-4922-80be-298086e30370
+credentials::pulp-api::passphrase: asdf1234
+
 # Uncomment below to increase the memory available for Jenkins
 # This is highly recommended for larger deployments
 # Don't use more than 75% of your RAM for Jenkins

hiera/hieradata/buildfarm_role/repo.yaml:

@@ -131,3 +131,12 @@ jenkins-agent::reprepro_config:
             # needed until we deploy the gpg key
             verify_release: blindtrust

+jenkins-agent::pulp_config:
+    admin_passphrase: asdf1234
+    rpm:
+        el:
+            architectures: [aarch64, x86_64]
+            versions: ['7', '8']
+        fedora:
+            architectures: [aarch64, armhf, i386, x86_64]
+            versions: ['30', '31']

The approach is to create a CentOS 8 container that installs and provisions pulpcore, and run that container using a systemd service. There is also a python script for creating the initial repositories and endpoints in pulp once it is up-and-running.

I'm pretty new to Puppet, Hiera and Ruby. If something looks wonky, it's probably a mistake, so let me know.

[nuclearsandwich]

The approach is to create a CentOS 7 container that installs and provisions pulpcore, and run that container using a systemd service

Does pulp itself need to run on CentOS in order to serve RPMs?

[cottsay]

Does pulp itself need to run on CentOS in order to serve RPMs?

It might not. At first, I firmly believed that it did, but after many iterations I realized that the python module for createrepo was re-building createrepo from source as part of the pip package (the version of createrepo in Ubuntu is insufficient).

Maybe we can run pulp on Xenial. I'll give it a shot, at least.

[cottsay]

After digging into this, it doesn't look like running pulp natively on Ubuntu is a good idea right now. There are missing dependencies that would need to be built from source, and some dependencies that might be too old.

I changed this PR to move the non-pulp components (psql and redis) out of the container, simplified the Dockerfile, and split out a service for each of the pulp components.

[cottsay]

So I don't think my credentials::pulp-api::password_secret trick is working. If I understand correctly, the Jenkins secrets are salted in some way, and aren't portable values.

It looks like the Jenkins puppet module actually talks to the Jenkins API to add credentials.

@nuclearsandwich, do you have any insights here?

[nuclearsandwich]

So I don't think my credentials::pulp-api::password_secret trick is working. If I understand correctly, the Jenkins secrets are salted in some way, and aren't portable values.

Jenkins secrets are AES encrypted with a master key generated for each deployment. I was under the impression that unencrypted secrets would be noticed and encrypted when the config was read and written back out but maybe that's not correct. I'll look into it and get back to you.

[ros-discourse]

This pull request has been mentioned on ROS Discourse. There might be relevant details there:

https://discourse.ros.org/t/rpm-packaging-support-in-ros-buildfarm/15672/1