[pkg.jenkins.io] migrate the pkg.origin.jenkins.io service from AWS VM to Azure `publick8s`

Service(s)

pkg.jenkins.io

Summary

The service pkg.jenkins.io serves the Jenkins distribution packages: Linux, Windows, WAR files.

It's merely a webserver serving the package manager index (or HTML index files which are listing directories):

It's cached by the Fastly CDN (pkg.origin.jenkins.io is the original service)
Most of the "heavy" files are served by the get.jenkins.io or mirrors.jenkins.io services (which are 2 DNS pointing to the same system)

We want to move the service pkg.origin.jenkins.io out from AWS:

It will be decorelated from the Update Center (updates.jenkins.io) webservice (as both are running on the same VM today)
No more pinning to an old Apache server version due to Ubuntu 18.04

The obvious choice is to set up the webservice in Azure:

Generation of the packages could be done directly in release.ci.jenkins.io with the data stored in an Azure file share
The cluster publick8s would host the webservice with the azure file mounted in read-only => it would allows adding HA to the service

Reproduction steps

No response

First pass of analysis shows that we might want to keep Apache (httpd) for this webservice to avoid introducing too much changes:

The puppet profile pkgrepo describes a classic Apache setup (virtualhost, log with rotation, TLS termination with HTTPS redirect) => no strict pinning to Apache over Nginx
But this pkgrepo puppet profiles calls 3 other profiles which are generating custom .htaccess files (Apache only) with custom rules:
- (Debian with custom htaccess for APT) https://github.com/jenkins-infra/jenkins-infra/blob/production/dist/profile/manifests/debian_repo.pp => https://github.com/jenkins-infra/jenkins-infra/blob/production/dist/profile/templates/pkgrepo/debian_htaccess.erb
- (Opensuse, which reuse the Redhat htaccess) https://github.com/jenkins-infra/jenkins-infra/blob/production/dist/profile/manifests/opensuse_repo.pp => https://github.com/jenkins-infra/jenkins-infra/blob/production/dist/profile/templates/pkgrepo/redhat_htaccess.erb
- (Redhat with custom redhat htaccess) https://github.com/jenkins-infra/jenkins-infra/blob/production/dist/profile/manifests/redhat_repo.pp => https://github.com/jenkins-infra/jenkins-infra/blob/production/dist/profile/templates/pkgrepo/redhat_htaccess.erb

=> It means we'll need an Apache helm chart, not sure if we can reuse the existing "files" service in our custom mirrorbits helm chart (chart inheritance/split seems quite needed here)

Continuing prerequisites analysis to determine:

"Where to store the persistent data served by the webserver"?
"What kind of persistent system to use"?

The Core release of Jenkins have a packaging pipeline defined here: https://github.com/jenkins-infra/release/blob/master/Jenkinsfile.d/core/package

Each kind of package is generated by a call to release.bash script with the flag --packaging set to the kind of expected package to generate.
- It's running inside a pod template agent executed in the privatek8s cluster, as part of the release.ci.jenkins.io controller (ref. https://github.com/jenkins-infra/kubernetes-management/blob/774040c2f139be0a60f6c2f8b84761ddacf1d3fb/clusters/privatek8s.yaml#L79-L104 => 3 Helm Releases for release.ci.jenkins.io: 1 for controller, 1 for agents and 1 for the persistent volumes used to store packages)
The release.bash script in jenkins-infra/release is only a pass-trough to jenkinsci/packaging's Makefile as per https://github.com/jenkins-infra/release/blob/5afd723749659bf38ac3f5fcfb760eae55877587/utils/release.bash#L327-L330 which calls each "kind of packaging"'s build and publich shell scripts
The publish.sh script all have the same pattern: they generate the package as part of their "Build" stage and then their "Release" stage publishes the generated package into a "staging" directory
- Each generated package (e.g. the binary package .deb, .rpm, etc.) is copied to 2 destinations:
- Locally to the agent, in the Azure File storage referenced as binary-core-packages which is not used by the webservice pkg.jenkins.io (Wild guess: it's for get.jenkins.io). Example with debian: https://github.com/jenkinsci/packaging/blob/2da9b53b83c878d82b39ea8f753a3c0edd0cc3ac/deb/publish/publish.sh#L93.
- In the pkg.origin.jenkins.io VM referenced as the $PKGSERVER SSH/Rsync URL. Example for debian: https://github.com/jenkinsci/packaging/blob/2da9b53b83c878d82b39ea8f753a3c0edd0cc3ac/deb/publish/publish.sh#L102.
- Each generated package index (for each new package there is a package index update referenced as "sitepackage") is copied to 2 destinations:
- Locally to the agent, in another Azure File storage referenced as website-core-packages. Example with Debian: https://github.com/jenkinsci/packaging/blob/2da9b53b83c878d82b39ea8f753a3c0edd0cc3ac/deb/publish/publish.sh#L118.
- In the pkg.origin.jenkins.io VM referenced as the $PKGSERVER SSH/Rsync URL. Example for debian: https://github.com/jenkinsci/packaging/blob/2da9b53b83c878d82b39ea8f753a3c0edd0cc3ac/deb/publish/publish.sh#L126
- Some package managers also generates additional HTML files (such as Debian) which are uploaded to the PKGSERVER's webdir and the binary Azurefile (as it is served by get.jenkins.io) but not to the website-core-packages (minor issue that can be easily fixed)
Once the whole package generation process is run with success, the last stage is to promote the content of the "staging directory" to the "production directory".
- It seems to be done remotely by https://github.com/jenkins-infra/release/blob/5afd723749659bf38ac3f5fcfb760eae55877587/utils/release.bash#L483 which is a remote (in the PKGSERVER VM) execution of the following script:
- In particular the promotion is done by this instruction (a rsync command from /var/www/pkg.jenkins.io.staging to /var/www/pkg.jenkins.io/): https://github.com/jenkins-infra/mirror-scripts/blob/bebd7675616a2073173b241ddf5021d098c5f2b4/sync.sh#L64-L69

Considering https://github.com/jenkins-infra/helpdesk/issues/3636, https://github.com/jenkins-infra/helpdesk/issues/3338 and https://github.com/jenkins-infra/helpdesk/issues/3183, it looks like that the strategy of splitting index and package creates unexpected problems:

Package Indexes are only served from one location in the world. => We might want to use a mirror service to project the indexes to other regions such as China or Asia (which have quite a network latency when readhing an us-east based webservice).
- Same pattern as for the Update Center (updates.jenkins.io) with a set of mirrors that we do manage. That would means changing the current Fastly CDN configuration to act as a mirror instead of a front CDN
We want a "reference" mirror for download to act as a fallback for the binary files of pkg.jenkins.io. As archives.jenkins.io is designed for this, we could use the webservice pkg.origin.jenkins.io for this.

jenkins-infra / helpdesk