AKS: add cluster `publick8s` and migrate `prodpublick8s` public services on it

[dduportal]

This issue tracks the work for spawning a new "public" AKS cluster for production to replace the former prodpublick8s.

Goals:

• Get rid of the network overlap from the current AKS prodpublick8s
• Ensure that the cluster is managed as code
• Support IPv6 for services hosted in this cluster (requires NOT using CNI for this cluster, forbidding us to use Windows node pools)
  • Ref. https://learn.microsoft.com/en-us/azure/aks/configure-kubenet-dual-stack?tabs=azure-cli%2Ckubectl
• Ensure a new "home" for https://github.com/jenkins-infra/helpdesk/issues/2752 in Azure environment

This issue is the "twin" of https://github.com/jenkins-infra/helpdesk/issues/2844 but for public network.

---

Some notes:

• Node pools:
  • No Windows node pool is expected (only used by release.ci.jenkins.io which will be migrated to privatek8s)
  • No Highmem node pool is expected (only used by release.ci.jenkins.io which will be migrated to privatek8s - Only reference to highmem string in the context of Kubernete in https://github.com/search?q=org%3Ajenkins-infra+highmem&type=code is https://github.com/jenkins-infra/release/blob/902df23d5657c074f184d8e08d1d00d7e3e67c95/PodTemplates.d/release-linux.yaml#L44)
  • The current agentpool for prodpublick8s should have an equivalent (in term of VM size, disk size and autoscaling limits) in the new privatek8s
  • A "default system node pool" is expected, like for privatek8s, as a good AKS practise (ref. https://learn.microsoft.com/en-us/azure/aks/use-system-pools?tabs=azure-cli and https://learn.microsoft.com/en-us/azure/aks/use-multiple-node-pools)
• The #3306 is NOT needed (we can reach the private-ingress endpoint using the private's VPN network peering to public)
• [ ] We'll have to decide which can of "egress" (attribute outboundType) type to use: https://learn.microsoft.com/en-us/azure/aks/egress-outboundtype
• We'll have to ensure that the CSI storage classe are set to retain for the non-default, and that any "custom" CSI classe is installed
• We'll have to install in this cluster the following (usual) side services:
  • [x] datadog
  • [x] certmanager+acme
  • [x] Nginx (private) ingress controller (with an internal load balancer that should be reachable from the peered private network), and should be the default
  • [x] Nginx (public) ingress controller (with a public LB and public IP)
  • [x] Falco (to ensure some policies are enforced on workloads)
• We'll have to migrate the following services from prodpublick8s to this cluster:
  • [x] plugin-site (plugins.jenkins.?io)
  • [x] plugin-site-issues
  • [x] reports (reports.jenkins.io)
  • [x] jenkinsio (⚠️ when changing the DNS, we'll have to watch fastly) (origin.jenkins.io)
  • [x] javadoc (javadoc.jenkins.io)
  • [x] accountapp (accounts.jenkins.io)
  • [x] ldap (ldap.jenkins.io with its own public LB)
  • [x] uplink (uplink.jenkins.io)
  • [x] mirrorbits (get.jenkins.io)
  • [x] keycloak
  • [x] incrementals-publisher
  • [x] jenkins-weekly
  • [x] jenkinsisthewayio-redirect
  • [x] artifact-caching-proxy
  • [x] plugin-health-scoring
  • [x] wiki
  • [x] rating

    ### Tasks

[lemeurherve]

With this migration we'll be able to close https://github.com/jenkins-infra/helpdesk/issues/3209

[dduportal]

Putting on hold: #2844 tracks the migration of release.ci from prodpublick8s to privatek8s, before proceeding forward here.

[lemeurherve]

weekly.jenkins.io migrated:

• https://github.com/jenkins-infra/kubernetes-management/pull/3930
• https://github.com/jenkins-infra/azure-net/pull/67
• https://github.com/jenkins-infra/kubernetes-management/pull/3931

We noticed that while the LDAP wasn't accessible for Jenkins, it doesn't rendered the HTML set as welcome message.

[dduportal]

Next steps: migrating the services relying on a postgreSQL database.

To avoid any propagation of the network overlap, we need a flexible Postgres instance that the public-vnet network can access.

A new instance has to be created:

• Delegated subnet: https://github.com/jenkins-infra/azure-net/pull/78
• New instance: https://github.com/jenkins-infra/azure/pull/356

Alas, Azure Flexible servers do not support IPv6 virtual nets, so we'll have to find another solution. Current scenario is to create a dedicated virtual net, IPv4 only, and study the methods to access it privately.

• Rollback: https://github.com/jenkins-infra/azure/pull/359

[dduportal]

As per https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-networking:

• A given flexible Postgres instance cannot be in a virtual network and have public endpoint as the same time
• The only way to access a Flexible Postgres instance in a virtual network from another network is to use network peering
• The "private link" concept is only for the DNS resolution

=> as we want a managed database with no public endpoint, it means we need a dedicated vnet + (delegated) subnet for the new Postgres flexbible instance. Only unknown is the peering behavior with IPv6 / IPv4.

[dduportal]

• The public-db instance is now working, reachable from both private and public vnets
• Database for keycloak is now managed in public-db by Terraform (and the postgres provider)
• Keycloak migration in progress:
  • Not managed anymore from prodpublick8s (but not removed yet)
  • Issues with Dualstask network mode, should be fixed but no HA available (we did not use it so it's ok)
  • WiP on the VPN routing to allow reaching "private" IPs (such as private-ingress LB) of the public network throug the private VPN

[dduportal]

Keycloak is migrated with success to the new cluster:

• DNS record
• Network routes (through the VPN)

Removed the former ingress to confirm and looks good for both @smerle33 and I with the private VPN connected

Next step: gotta remove the migrated resources from prodpublick8s to be really sure that nothing runs on the former cluster (starting with keycloak only as it would only iumpact infra team: gotta wait for annoucement for public services)

[dduportal]

Update:

Keycloak cleanup:

• Namespace delete from prodpublick8s with success
• Database deleted from public PgSQL instance with success
• https://admin.accounts.jenkins.io works well

Removed the jenkins-weekly, javadoc, jenkinsisthewayio-redirector and wiki namespaces from `prodpublick8s (already migrated)

Removed the (empty) namespace archives from prodpublick8s

Next candidates: Plugin Helath Score and rating

[dduportal]

Migration of plugin-health score done, in mob-programming with the help of @smerle33 and @alecharp

• Annonce status.jenkins.io - https://github.com/jenkins-infra/status/pull/289
• Opportunity to verify the impact on plugins.jenkins.io when PHS goes down
  • Manually: backup of the ingress in prodpublick8s, delete ingress, and check: no impact on plugins.jenkins.io (only on the static generation)
  • Restored the ingress
• Disabled jenkins-infra/kubernetes-management
• Deleted (manually) namespace in prodpublick8s
• Backup BDD -> restore to new instance public-db
• Migrated Helm release to publick8s - https://github.com/jenkins-infra/kubernetes-management/pull/4000
• Change the cron to validate deployment - https://github.com/jenkins-infra/kubernetes-management/commit/ea7a257612c7143322f52076dba13c55868bdb65
• Enabled jenkins-infra/kubernetes-management => deployment of PHS success
• Checking logs: both cron tasks (Update Center and plugins) went well
• Migrated DNS: https://github.com/jenkins-infra/azure-net/pull/85
  • => homepage visible at https://plugin-health.jenkins.io/probes after 2-3 minutes
• Rollback cron - https://github.com/jenkins-infra/kubernetes-management/pull/4001
• Deleted the old postgres database from public instance
• Closes status - https://github.com/jenkins-infra/status/pull/290
• Check plugins.jenkins.io generation in infra.ci:
  • Triggered a build - https://infra.ci.jenkins.io/job/website-jobs/job/plugin-site/job/master/2989
  • https://plugins.jenkins.io/ui/search/?query=kube works without error

✅ 🚀

[dduportal]

Update:

• Fixed the updatecli manifest for plugin-health-score (we have to do this for each migration): https://github.com/jenkins-infra/kubernetes-management/pull/4003
• Manage (and fix) as code the jenkins-ci.org records for javadoc and wiki: https://github.com/jenkins-infra/azure-net/pull/86
• Removed the (unused) grafan DNS records on both jenkins.io and jenkins-ci.org domains:

[dduportal]

Migration of the Incremental Publisher service (https://github.com/jenkins-infra/iep/tree/master/iep-009):

• Open Status - https://github.com/jenkins-infra/status/pull/291
• Send email to developer mailing list - https://groups.google.com/g/jenkinsci-dev/c/hUjdo21dAOA
• No database, no persistence: can be migrated without downtime
  • Checked configuration and secrets: no need to update any environment-related element ✅
  • No whitelist.denylist on artifactory ✅
• Migrate Helm release management to publick8s - https://github.com/jenkins-infra/kubernetes-management/pull/4004
• Migrate DNS - https://github.com/jenkins-infra/azure-net/pull/88
• Test a build on ci.jenkins.io that should publish with success - https://ci.jenkins.io/job/Plugins/job/jenkins-infra-test-plugin/job/master/192/ ✅
• Delete namespace in prodpublick8s
• Test another build on ci.jenkins.io that should publish with success - https://ci.jenkins.io/job/Plugins/job/jenkins-infra-test-plugin/job/master/192/ ✅
• Close Status + email - https://github.com/jenkins-infra/status/pull/292

[dduportal]

Migration of the Rating service

• [x] Open Status - https://github.com/jenkins-infra/status/pull/294
• No need for email communication
• Database
  • [x] Creating database in Terraform Azure - https://github.com/jenkins-infra/azure/pull/368
  • [x] Initial backup/restore to evaluate the amount of time (< 1 min)
• [x] Prepare to migrate Helm release management to publick8s (both at same time, PR to prepare prior to the operation) - https://github.com/jenkins-infra/kubernetes-management/pull/4012
• Shut down service on prodpublick8s (removing ingress manually to fail incoming requests)
  • [x] Backup ingress
  • [x] Remove ingress
• Deploy on Kubernetes:
  • [x] Push the new credentials to jenkins-infra/chart-secrets - https://github.com/jenkins-infra/charts-secrets/commit/f7c0b1f145814bd4bca9de8c97a9cda83803c7ea
  • [x] Merge the jenkins-infra/kubernetes-management PR
• [x] Migrate DNS - https://github.com/jenkins-infra/azure-net/pull/89 (deletes the former A record in favor of a CNAME)
• [x] Sanity Checks: On jenkins.io, test to rate a release
• [x] Delete namespace in prodpublick8s
• [x] Delete former database
• [x] Close Status - https://github.com/jenkins-infra/status/pull/295

[dduportal]

Migration of the Uplink service

Main challenge is the database size: 88% storage used of the available 85 Gb for the current PostgreSQL Single Server:

• We need the disk space on the new PostgreSQL flexible server
• The service interruption is constrained by the backup/restore time

[dduportal]

Update:

• https://github.com/jenkins-infra/azure/pull/372 to cleanup resources after rating migration
• https://github.com/jenkins-infra/azure/pull/373 to increase the storage size of the public-db instance to sustain the uplink database

[dduportal]

Update "Uplink":

on the (new.)ci.jenkins.io:

• Installed postgres-client
• Checked size of /var/lib/jenkins/: 478 Gb available
• OLD DB:
  • Connection OK with psql --host=produplink.postgres.database.azure.com --username='uplinkadmin@produplink' --password uplink
  • Ran a VACUUM VERBOSE in a screen named uplink_vacuum
• NEW DB:
  • Connection OK with psql --host=public-db.postgres.database.azure.com --username=uplink --password uplink
• Backup:
  • Idea is to disable compression, use "directory" (recursive for better I/Os) and all vCPUs
  • time pg_dump --host=produplink.postgres.database.azure.com --username='uplinkadmin@produplink' --password --dbname=uplink --compress=0 --jobs=8 --format=d --file=/var/lib/jenkins/uplink_$(date +%Y%m%d_%H%M%S) in a screen named uplink_dump

[dduportal]

Update in uplink:

• Vacuum went fine but did not decrease the storage size (expected as the event table is huge).
• First successful dump took 523 min and weight 105Gb
  • Limitation was due to not enough jobs and vacuum competing for resources on server side
• Second try with 64 jobs: the I/O disk write is up from 200 Kb/s to 8 Mb/s, took 482 min
• A restore took ~ 4h on the new DB

[dduportal]

Update with uplink: as the "old" PostgreSQL is reachabale from the new cluster and has too much data, I've opened https://github.com/jenkins-infra/helpdesk/issues/3609 for the database migration.

Also, it seems that the "old" database is a Postgres 10 instance while our public-db is version 13.

Gotta migrate uplink today, keeping the same database as before to avoid too many changes.

[dduportal]

(updated) Plan for migrating Uplink:

• [x] Open Status - https://github.com/jenkins-infra/status/pull/296
• [-] No need for email communication
• [-] Database: we keep the same database as before
• [x] Deploy on the new Kubernetes cluster
  • [x] Migrate Helm release management to publick8s - https://github.com/jenkins-infra/kubernetes-management/pull/4019
  • [x] Fix to ensure each pod of every service runs on the same x86medium publick8s node pool - https://github.com/jenkins-infra/kubernetes-management/pull/4025
  • [x] Verify the new application is up and running (testing its ingress and checking logs)
• [x] Migrate DNS - https://github.com/jenkins-infra/azure-net/pull/90
• [x] Sanity Checks:
  • [x] Check the main webpage
  • [x] Check Sentry: no event since May 6th 😅
• [x] Delete namespaces in prodpublick8s
• [x] Sanity Checks:
  • [x] Check the main webpage
  • [x] Check Sentry
• [x] Close Status - https://github.com/jenkins-infra/status/pull/297

[lemeurherve]

Migration of uplink.jenkins.io completed, no service interruption.

[lemeurherve]

Migration of the Reports service:

• [x] Open status - https://github.com/jenkins-infra/status/pull/299
• [-] No need for email communication
• [-] No database, only a storage account on Azure, accessible to the new cluster
• [x] Deploy on the new Kubernetes cluster
  • [x] Migrate Helm release management to publick8s (and add nodeSelector) -https://github.com/jenkins-infra/kubernetes-management/pull/4028
  • [x] Verify the new application is up and running (testing its ingress and checking logs)
• [x] Migrate DNS (with moved block) - https://github.com/jenkins-infra/azure-net/pull/93
• [x] Sanity checks
• [x] Delete reports namespace in prodpublick8s cluster
• [x] Sanity checks
• [x] Close status - https://github.com/jenkins-infra/status/pull/300

[lemeurherve]

Migration of https://reports.jenkins.io completed, no service interruption.

[lemeurherve]

Migration of accountapp service:

• [x] Open status - https://github.com/jenkins-infra/status/pull/301
• [-] No need for email communication
• [-] No data migration: stateless service, LDAP accessibility from publick8s already taken care of while migrating Keycloak service
• [x] Deploy on the new Kubernetes cluster
  • [x] Migrate Helm release management to publick8s (and add nodeSelector) - https://github.com/jenkins-infra/kubernetes-management/pull/4030
  • [x] Verify the new application is up and running (testing its ingress and checking logs)
• [x] Migrate DNS (with moved block)
  • [x] Import accounts.jenkins-ci.org CNAME repo in terraform
  • [x] Disable azure-net job on infra.ci.jenkins.io
  • [x] terraform import 'azurerm_dns_cname_record.jenkinsciorg_target_public_publick8s["accounts"]' /subscriptions/<subscription-id>/resourceGroups/proddns_jenkinsci/providers/Microsoft.Network/dnsZones/jenkins-ci.org/CNAME/accounts
  • [x] Change CNAME target - https://github.com/jenkins-infra/azure-net/pull/95
  • [x] Enable azure-net job on infra.ci.jenkins.io
• [x] Sanity checks: successfully logged in
• [x] Delete accountapp namespace in prodpublick8s cluster
• [x] Sanity checks
• [x] Close status - https://github.com/jenkins-infra/status/pull/303

[lemeurherve]

Migration of https://accounts.jenkins.io completed, no service interruption.

[lemeurherve]

Migration of the LDAP service:

• [x] Create a new storage account accessible to the new cluster dedicated to LDIF dumps from LDAP - https://github.com/jenkins-infra/azure/pull/385 > https://github.com/jenkins-infra/azure/pull/392
  • [x] Fix: add Storage endpoint to publick8s-tier subnet - https://github.com/jenkins-infra/azure-net/pull/96
  • [x] File Share manually created, issue opened to recreate it later as code instead - https://github.com/jenkins-infra/azure/issues/401
• [x] Deploy on the new Kubernetes cluster
  • [x] Add Helm release management to publick8s (and add nodeSelector) and the new storage account credentials, but keeping the release on prodpublick8s until the end of the migration - https://github.com/jenkins-infra/kubernetes-management/pull/4042
  • [x] Public IPv4 created as code - https://github.com/jenkins-infra/azure/pull/402
  • [x] Set the LDAP load balancer (TCP) to use this determined IP instead of using a random new IP to facilitate DNS setup and change (no need to wait for the new IP address)
  • [x] Verify the new application is up and running
  • [x] Restore a complete LDIF dump for testing purpose
  • [x] From the LDAP slapd container on publick8s, took a bit more than 5 minutes to complete service slapd stop && ./entrypoint/restore
  • [x] Sanity checks
• [x] Open status - https://github.com/jenkins-infra/status/pull/312
• [x] Send email to the mailing lists - https://groups.google.com/g/jenkinsci-dev/c/0sXeaS0Yeng & https://groups.google.com/g/jenkins-infra/c/dLNgQJQXrJk
• [x] Import ldap A record as code (and manually reduced its TLL to 60s) - https://github.com/jenkins-infra/azure-net/pull/97
• [x] Switch accountapp DNS (with moved block) from publick8s to status.jenkins.io as we need to stop writes on LDAP while performing the LDIF dump - https://github.com/jenkins-infra/azure-net/pull/98
  • [x] Fix to redirect to the status.jenkins.io Netlify origin site in order to avoid SAN cert error - https://github.com/jenkins-infra/azure-net/pull/102
• [x] Dump and retrieve LDIF from prodpublick8s and transfer it to the new storage account
• [x] Restore the dump in publick8s
• [x] Sanity checks via slapd container logs OK
• [x] Migrate LDAP DNS (no moved block as it's only an A record modification, not a transfer of block like for the CNAME ones) from prodpublick8s to publick8s - https://github.com/jenkins-infra/azure-net/pull/99
• [x] Sanity checks via a login in admin.accounts.jenkins.io (Keycloak) & requests proceeded in slapd container logs in publick8s
• [x] Remove ldap release from prodpublick8s - https://github.com/jenkins-infra/kubernetes-management/pull/4053
• [x] Disable ldap service in prodpublick8s cluster (scaled to 0)
• [x] Sanity checks via a login in admin.accounts.jenkins.io (Keycloak)
• [x] Switch back accountapp DNS (with moved block) from status.jenkins.io to publick8s - https://github.com/jenkins-infra/azure-net/pull/102
• [x] Sanity checks via a login in accounts.jenkins.io
• [x] Delete ldap namespace in prodpublick8s cluster
• [x] Close status - https://github.com/jenkins-infra/status/pull/313

[dduportal]

Update on the LDAP: https://github.com/jenkins-infra/azure/pull/385#issuecomment-1577251779

=> there are missing elements to allow managing storage accounts in some of the publick8s networks

[lemeurherve]

All preliminary steps for the LDAP migration are completed, I'll proceed to the switch tomorrow.

[lemeurherve]

Although the intended redirections from accounts.jenkins.io & accounts.jenkins-ci.org to status.jenkins.io didn’t work as expected (SAN cert issue?), the LDAP migration has been successfully completed, no service interruption.

[lemeurherve]

Migration of mirrorbits (https://get.jenkins.io, https://mirrors.jenkins.io, https://mirrors.jenkins-ci.org, https://fallback.get.jenkins.io) service:

• [x] Open status - https://github.com/jenkins-infra/status/pull/316
• [-] No need for email communication
• [-] No data migration: the existing redis database is accessible from publick8s

• [x] Deploy on the new Kubernetes cluster

  • [x] Migrate Helm release management to publick8s (and add nodeSelector) - https://github.com/jenkins-infra/kubernetes-management/pull/4062

  • [x] Verify the new application is up and running (testing its ingress and checking logs)

    logs:

    $ kubectl config use-context publick8s-endless-ghoul $ kubectl logs -n mirrorbits -f -l app.kubernetes.io/name=mirrorbits -c mirrorbits _______ __ __ __ __ | | |__|.----.----.-----.----.| |--.|__| |_.-----. | | || _| _| _ | _|| _ || | _|__ --| |__|_|__|__||__| |__| |_____|__| |_____||__|____|_____| v0.5.1 2023/06/12 12:28:24.648 UTC Loading GeoLite2-City.mmdb database (built on 2023-06-08 18:44:53 +0000 UTC) 2023/06/12 12:28:24.648 UTC Loading GeoLite2-ASN.mmdb database (built on 2023-06-08 16:46:47 +0000 UTC) 2023/06/12 12:28:24.649 UTC Service listening on :8080 2023/06/12 12:28:24.668 UTC Connected to redis master jenkins-mirrorbits.redis.cache.windows.net:6379 2023/06/12 12:28:24.764 UTC [source] Scanning the filesystem... _______ __ __ __ __ | | |__|.----.----.-----.----.| |--.|__| |_.-----. | | || _| _| _ | _|| _ || | _|__ --| |__|_|__|__||__| |__| |_____|__| |_____||__|____|_____| v0.5.1 2023/06/12 12:28:24.480 UTC Loading GeoLite2-City.mmdb database (built on 2023-06-08 18:44:53 +0000 UTC) 2023/06/12 12:28:24.480 UTC Loading GeoLite2-ASN.mmdb database (built on 2023-06-08 16:46:47 +0000 UTC) 2023/06/12 12:28:24.481 UTC Service listening on :8080 2023/06/12 12:28:24.501 UTC Connected to redis master jenkins-mirrorbits.redis.cache.windows.net:6379 2023/06/12 12:28:24.605 UTC [source] Scanning the filesystem... 2023/06/12 12:43:58.139 UTC [source] Indexing the files... 2023/06/12 12:43:58.401 UTC [source] Indexing the files... 2023/06/12 12:44:00.192 UTC [source] Scanned 46261 files 2023/06/12 12:44:00.859 UTC -> Node mirrorbits-55d8647848-dwk85-30145 joined the cluster 2023/06/12 12:44:00.859 UTC -> Node mirrorbits-55d8647848-whfl6-28588 joined the cluster 2023/06/12 12:44:01.786 UTC archives.jenkins.io Up! (400ms) 2023/06/12 12:44:02.412 UTC servanamanaged.com Up! (1027ms) 2023/06/12 12:44:03.011 UTC [source] Scanned 46261 files 2023/06/12 12:44:03.561 UTC -> Node mirrorbits-55d8647848-whfl6-28588 joined the cluster 2023/06/12 12:44:03.561 UTC -> Node mirrorbits-55d8647848-dwk85-30145 joined the cluster 2023/06/12 12:44:03.561 UTC -> Node mirrorbits-776858c54f-w2shk-21783 joined the cluster 2023/06/12 12:44:04.252 UTC -> Node mirrorbits-776858c54f-pkff6-15361 joined the cluster 2023/06/12 12:44:04.743 UTC rwth-aachen.de Up! (490ms) 2023/06/12 12:45:02.694 UTC archives.jenkins.io Up! (309ms) 2023/06/12 12:45:04.484 UTC servanamanaged.com Up! (1099ms) 2023/06/12 12:45:05.653 UTC rwth-aachen.de Up! (399ms)
• [x] Migrate DNS (with moved block) - https://github.com/jenkins-infra/azure-net/pull/104
• [x] Sanity checks
• [x] Disable ingress in prodpublick8s cluster
• [x] Sanity checks
• [x] Delete mirrorbits namespace in prodpublick8s cluster
• [x] Sanity checks
• [x] Close status - https://github.com/jenkins-infra/status/pull/319

[lemeurherve]

Migration of plugin-site-issues service:

• [x] Open status - https://github.com/jenkins-infra/status/pull/317
• [-] No need for email communication
• [-] No data migration: stateless service
• [x] Deploy on the new Kubernetes cluster
  • [x] Migrate Helm release management to publick8s (and add nodeSelector) - https://github.com/jenkins-infra/kubernetes-management/pull/4064
  • [x] Verify the new application is up and running (testing its ingress and checking logs)
• [x] Migrate DNS (with moved block) - https://github.com/jenkins-infra/azure-net/pull/105
• [x] Sanity checks
• [x] Disable ingress in prodpublick8s cluster
• [x] Sanity checks
• [-] ~Delete plugin-site-issues namespace in prodpublick8s cluster~ plugin-site-issues is deployed in the plugin-site namespace, will be deleted during plugin-site migration later.
• [x] Close status - https://github.com/jenkins-infra/status/pull/320

[lemeurherve]

Migration of mirrrorbits and plugin-site-issues completed, no service interruption.

As a precaution, we'll delete the mirrorbits namespace from prodpublick8s tomorrow or Wednesday.

[lemeurherve]

Migration of plugin-site service:

• [x] Open status - https://github.com/jenkins-infra/status/pull/321
• [-] No need for email communication
• [-] No data migration: stateless service
• [x] Disable https://infra.ci.jenkins.io/job/website-jobs/job/plugin-site/ job
• [x] Deploy on the new Kubernetes cluster
  • [x] Migrate Helm release management to publick8s (and add nodeSelector) - https://github.com/jenkins-infra/kubernetes-management/pull/4067
  • [x] Verify the new application is up and running (testing its ingress and checking logs)
• [x] Migrate plugins.origin.jenkins.io (with moved block) - https://github.com/jenkins-infra/azure-net/pull/107
• [x] Sanity checks
• [x] Disable ingress in prodpublick8s cluster
• [x] Sanity checks
• [x] Refresh/purge Fastly cache
  • [x] Test a purge of https://plugins.jenkins.io/jenkins-infra-test/ with "soft delete" option as first test
  • [x] Purge all
• [x] Sanity checks
• [x] Delete plugin-site namespace in prodpublick8s cluster
• [x] Enable https://infra.ci.jenkins.io/job/website-jobs/job/plugin-site/ job
• [x] Close status - https://github.com/jenkins-infra/status/pull/323

[lemeurherve]

Migration of jenkins.io service:

• [x] Open status - https://github.com/jenkins-infra/status/pull/322
• [-] No need for email communication
• [-] No data migration: stateless service
• [x] Disable jenkins.io job on trusted.ci.jenkins.io
• [x] Deploy on the new Kubernetes cluster
  • [x] Migrate Helm release management to publick8s (and add nodeSelector) - https://github.com/jenkins-infra/kubernetes-management/pull/4068
  • [x] Verify the new application is up and running (testing its ingress and checking logs)
• [x] Migrate www.origin.jenkins.io (with moved block) - https://github.com/jenkins-infra/azure-net/pull/108
• [x] Sanity checks
• [x] Disable ingress in prodpublick8s cluster
• [x] Sanity checks
• [x] Refresh/purge Fastly cache
• [x] Sanity checks
• [x] Delete namespace in prodpublick8s cluster
• [x] Enable jenkins.io job on trusted.ci.jenkins.io
• [x] Close status - https://github.com/jenkins-infra/status/pull/324

[lemeurherve]

plugin-site migration completed, no service interruption.

[lemeurherve]

jenkins.io migration completed, no service interruption.

[lemeurherve]

Cleanup of unused DNS records found while working on this issue

A records pointing to 52.167.253.43 (prodpublick8s public IP)

• [x] public.aks.jenkins.io
• [x] jenkins-ci.org (apex record), to be imported in terraform and changed to 20.119.232.75 (publick8s public IP)
  • Import: https://github.com/jenkins-infra/azure-net/pull/110
  • IP change: https://github.com/jenkins-infra/azure-net/pull/111

CNAME records pointing to publick.aks.jenkins.io (ie prodpublick8s cluster):

• [x] archives.azure.jenkins.io
• [x] mirror.azure.jenkins.io^*
• [x] polls.jenkins.io
• [x] beta.accounts.jenkins.io
• [x] customize.jenkins.io

A records pointing to 10.0.2.5 (prodpublick8s private IP)

• [x] private.aks.jenkins.io

CNAME records pointing to private.aks.jenkins.io (ie prodpublick8s cluster):

• [x] release.pkg.jenkins.io
• [x] admin.polls.jenkins.io
• [x] release.repo.jenkins.io

Miscellaneous

• [x] archives.azure.jenkins.io
• [x] mirrors.azure.jenkins.io
• [x] mirrors2.jenkins-ci.org

^*: need additional cleanup in:

• [x] mirrorbits helm chart README
• [x] runbook
• [x] datadog checks.

[lemeurherve]

As we've noticed quite a lot of remaining requests still send to mirrorbits on prodpublick8s, we'll postpone the cluster deletion to next week, and @dduportal will see for the publication of a blogpost indicating the migration of this service to the new cluster.

[dduportal]

Namespaces removal: @lemeurherve and I paired and removed the following namespaces from prodpublick8s:

• datadog (causing https://github.com/jenkins-infra/datadog/pull/193 😅)
• cert-manager
• private-nginx-ingress.

Remaining namespaces are required until https://github.com/jenkins-infra/helpdesk/issues/3351#issuecomment-1591672053 is fixed.

[dduportal]

As we've noticed quite a lot of remaining requests still send to mirrorbits on prodpublick8s, we'll postpone the cluster deletion to next week, and @dduportal will see for the publication of a blogpost indicating the migration of this service to the new cluster.

As discussed with the last infrastructure meeting:

• We are planning to decommission the former prodpublick8s cluster the Tuesday 27 June 2023, including the former mirror service and the former IPv4
• Community thread in: https://community.jenkins.io/t/new-public-ipv4-for-jenkins-mirrors/8137
• Jenkins Blog post:
  • PR: https://github.com/jenkins-infra/jenkins.io/pull/6483
  • Direct link:

[dduportal]

Update:

• Cluster prodpublick8s destroyed with its resource group
• https://github.com/jenkins-infra/helpdesk/issues/3639 should be fixed

[lemeurherve]

Additional monitors added: https://github.com/jenkins-infra/datadog/pull/195

Potential improvements for later:

• [ ] use a PV/PVC instead of azurefile for plugin-site storage
• [ ] evaluate if plugin-site needs the same GitHub and Jira credentials as plugin-site-api (optional)
• [ ] create an issue for a mechanism to display a maintenance page on services while operating on them
• [ ] activate the "secure transfer required" for javadoc storage account
• [ ] add cluster outbound IPs as terraform output

After 3 years and 27 days of good and faithful service, prodpublick8s is not anymore, closing this issue 🤗