search

jenkins-infra / helpdesk

Open your Infrastructure related issues here for the Jenkins project
https://github.com/jenkins-infra/helpdesk/issues/new/choose
16 stars 10 forks source link

[INFRA-3100] Migrate updates.jenkins.io to another Cloud #2649

Open jenkins-infra-bot opened 2 years ago

jenkins-infra-bot commented 2 years ago

Why

Read the EPIC (aws cost decrease: https://github.com/jenkins-infra/helpdesk/issues/2646)

What

How

Different paths can be taken here: to be discussed in infra meeting + validated by the board as it's an important service.

Originally reported by dduportal, imported from: Migrate updates.ci.jenkins.io to another Cloud
  • status: Open
  • priority: Major
  • resolution: Unresolved
  • imported: 2022/01/10

[note]

dduportal commented 2 years ago

Requires setting up the Oracle Terraform project, like #2682

Todo list:

dduportal commented 2 years ago

Blocked by #2973

smerle33 commented 2 years ago

actual machine size is : 32Gb RAM 8cpu 1,2Tb data disk (372Gb free)

about half of the power is used currently (checked with the local SAR probe)

smerle33 commented 2 years ago

Infra to specify :

smerle33 commented 2 years ago

VM specifications :

dduportal commented 2 years ago

Side note: once you'll be working on the terraform project for these resource, please note that the network and sub-network would have to be datasources while the other would be resources (at least until we work on #2682 that will manage the networks)

timja commented 1 year ago

https://github.com/jenkinsci/acceptance-test-harness/pull/973/checks?check_run_id=11028552907 failed because of the reported error Error in the HTTP2 framing layer

ekohl commented 1 year ago

The Foreman project is sponsored by Fastly, I believe via the Fast Forward program (though someone else arranged it, so not sure). Given Jenkins is an open source project, it may qualify as well. See https://www.fastly.com/fast-forward

dduportal commented 1 year ago

The Foreman project is sponsored by Fastly, I believe via the Fast Forward program (though someone else arranged it, so not sure). Given Jenkins is an open source project, it may qualify as well. See https://www.fastly.com/fast-forward

Thanks for sharing! Fastly is already sponsoring us and used for existing jenkins.io websites, including the package distribution (see our setup here: https://github.com/jenkins-infra/fastly).

At first sight (from memory: I could be wrong or missing something), the main issue to put the update center distribution in Fastly was the uncaching challenge (time and steps): am I correct @daniel-beck @olblak @timja ?

Seconde concern is the amount of data served could make our sponsored program to cross the allowed threshold. For this one, we could discuss with Fastly.

Anyway, we have both Oracle and DigitalOcean cloud which provides an outbound bandwidth really cheap. As DigitalOcean is already used, the partnership is really dynamic (see #3487 for instance), we are thinking moving the update center from AWS to DigitalOcean (better infra as code API, better support, and already used by the infra).

timja commented 1 year ago

I think the issue is that fastly doesn’t sponsor the project enough credit to make it work although a CDN would be great.

lemeurherveCB commented 1 year ago

Another alternative could be https://www.cloudflare.com/en-gb/products/r2/

dduportal commented 1 year ago

Another alternative could be https://www.cloudflare.com/en-gb/products/r2/

Good point!

Only for the updates.jenkins.io / updates.jenkins-ci.org service (which serves the Update Center JSON files):

For pkg.jenkins.io: already cached using fastly and pricing depends on amount of storage (ever growing) so not sure if it is that easy (but should be checked)

dduportal commented 1 year ago

Another alternative could be https://www.cloudflare.com/en-gb/products/r2/

Good point!

Only for the updates.jenkins.io / updates.jenkins-ci.org service (which serves the Update Center JSON files):

* https://developers.cloudflare.com/r2/buckets/public-buckets/ shows we can expose their buckets publicly with a custom domain

* Pricing is really appealing: https://developers.cloudflare.com/r2/pricing/ as no egress bandwidth costs, and 4,50 $ per millions requests => nice catch @lemeurherve !

* Fine tuning:

  * We can use CORS: https://developers.cloudflare.com/r2/buckets/cors/#use-cors-with-a-public-bucket
  * We might have to tune the lifecycle of the objects though: https://developers.cloudflare.com/r2/buckets/object-lifecycles/ (given we update the JSON once a week after a weekly release)
  * We should be able to manage partially or totally the service using Terraform (ref. https://developers.cloudflare.com/r2/examples/terraform-aws/)

For pkg.jenkins.io: already cached using fastly and pricing depends on amount of storage (ever growing) so not sure if it is that easy (but should be checked)

Ping @dbeck @olblak @Wadeck if you have thought on this (e.g. using a bucket storage system such as AWS, Azure or CloudFlare R2 in this case to host the Update Center site, instead of using a not highly-available VM with Apache)

lemeurherve commented 1 year ago

Here is what I've came with for updates.jenkins.io:

Note: we can have R2 buckets in China with CloudFlare 🎉

we can expose their buckets publicly with a custom domain

Unfortunately this requires the custom domain to be registered by CloudFlare.

lemeurherve commented 1 year ago

We'll setup this HTTP redirector service using our mirrorbits chart to get the HTTP server and the mirrorring system included.

lemeurherve commented 1 year ago

Good news, we can use delegated domains for public R2 buckets 🎉 (I didn't wait long enough in my initial tests with a delegated personal domain)

I'll create a sub DNS zone cloudflare.jenkins.io then delegate this sub zone to CloudFlare so we could use it for R2.

lemeurherve commented 1 year ago

Too bad we can't use a sub domain for CloudFlare delegation:

image

At least we'll be able to register a new domain name with the same provider we're already using instead of registering it with another provider.

lemeurherve commented 1 year ago

Update: the base of the service has been deployed to publick8s cluster: https://github.com/jenkins-infra/kubernetes-management/pull/4211, I've tested adding a mirror with success.

Currently working on adding R2 bucket as mirror, then on .htacess redirections.

lemeurherve commented 1 year ago

The R2 dev bucket address has been added as a mirror, with the same files and folder as the ones on the azure file share.

Currently mirrorbits has a lot of errors in its logs:

UTC [r2dev] parsing trace file failed: "<!DOCTYPE" is not a valid timestamp

lemeurherve commented 1 year ago

Currently mirrorbits has a lot of errors in its logs:

UTC [r2dev] parsing trace file failed: "<!DOCTYPE" is not a valid timestamp

Found the culprit, mirrors need to be able to respond either to FTP or RSYNC requests: https://github.com/etix/mirrorbits//blob/24df0bc7ca40d397290f7713254973a04bb48d99/rpc/rpc.go#L550-L569

See also the corresponding error variable at https://github.com/etix/mirrorbits/blob/master/scan/scan.go#L31

Hopping mirrorbits accepts different domains for mirror URL and mirror rsync address, I'm currently transplanting the rsyncd part of the deleted mirror chart which was used for azure.mirror.jenkins.io decomissionned mirror service (https://github.com/jenkins-infra/helm-charts/pull/607) in the mirrorbits chart, see https://github.com/jenkins-infra/helm-charts/pull/609

With this rsyncd service using the same local reference volume of mirrorbits, we should then be able to add the R2 bucket mirror, associated with this rsyncd service address.

It implies updating both the azure storage account and the R2 bucket content.

FTR, I've also tried to find a way to mount the R2 bucket content as volume for a rsyncd service running in EKS, hopping there would be a CSI driver for that by AWS. Unfortunately, their FSx file system solution seems to require permission access to the bucket, forbidding the use of S3 compatible buckets from other providers: https://docs.aws.amazon.com/fsx/latest/LustreGuide/setting-up.html#fsx-adding-permissions-s3 Every other solution seems to resolve around s3fs solution or equivalent, which are a no-go as their required privileged context execution, too risky.

lemeurherve commented 1 year ago

After splitting mirrorbits helm chart into 3 distinct charts (mirrorbits-lite, httpd and rsyncd) regrouped as subchart in mirrorbits-parent for greater velocity and configurability, our updates.jenkins.io POC has been deployed on publick8s with this new chart in https://github.com/jenkins-infra/kubernetes-management/pull/4320

Adding a mirror with (for example) the R2 bucket URL with the local kubernetes service URL of the rsyncd service is working, the next steps are:

lemeurherve commented 12 months ago

Excellent news, CloudFlare is sponsoring us 🎉 🥳

Refocusing on R2 buckets and the availability of a new updates.jenkins.io as soon as possible, we'll work on mirrors in parallel.

I've created a cloudflare.jenkins.io NS record, allowing us to delegate it to CloudFlare ✅

The next steps are the last two points of my previous comment.

daniel-beck commented 12 months ago

Copy generated .htaccess files into the storages

Do these support mod_rewrite rules? I would expect only Apache to support this enough…

dduportal commented 12 months ago

Copy generated .htaccess files into the storages

Do these support mod_rewrite rules? I would expect only Apache to support this enough…

In the new architecture, the Apache rewrite rules are handled by the Apache in Azure which receives all requests by default. Only the exact requests to JSON files will be sent to the mirror redirector.

The storage mentionned by @lemeurherve is the storage shared by both mirror redirector and Apache in azure.

lemeurherve commented 12 months ago

I've created a (for now empty) terraform state dedicated to CloudFlare, stored in Azure: https://github.com/jenkins-infra/terraform-states/tree/main/cloudflare (private repository)

I've also initialized the https://github.com/jenkins-infra/cloudflare repository, and before merging https://github.com/jenkins-infra/cloudflare/pull/1 we need to:

Then we'll start an initial job on empty state from the main branch.

We'll finally be able to add CloudFlare resources via terraform to continue our work on updates.jenkins.io:

lemeurherve commented 11 months ago

[...] we need to:

* Create staging (read-only) an prod (write) API tokens from the jenkins-infra-team member via CloudFlare UI
* Add these tokens and the terraform backend config to infra.ci.jenkins.io credentials
* Create a corresponding job in infra.ci.jenkins.io
* Configure repository team and branch protection

Then we'll start an initial job on empty state from the main branch.

We'll finally be able to add CloudFlare resources via terraform to continue our work on updates.jenkins.io:

* R2 buckets in several regions (Western North America, Eastern North America, Western Europe, Eastern Europe & Asia-Pacific)
* The corresponding (CloudFlare) "zones" (ie NS records for each of these regions, ex: `westeurope.cloudflare.jenkins.io`)

All of this has been done, currently recreating the westeurope zone as its edge SSL certificate seems stuck in "pending validation", probably due to a collusion with cloudflare.jenkins.io own certificate validation for *.cloudflare.jenkins.io.

This cloudflare.jenkins.io has been since deleted, and we'll store westeurope.cloudflare.jenkins.io NS records inside Azure instead of CloudFlare.

smerle33 commented 11 months ago

to have a better view on the impact of the code (aka parallelization) here is a graph of the cpu and memory as for now (with hervé PR tries during those last 24h on a specific folder on the current VM pkg.origin.jenkins.io):

Capture d’écran 2023-10-18 à 14 26 03 Capture d’écran 2023-10-18 à 14 29 17
dduportal commented 11 months ago

to have a better view on the impact of the code (aka parallelization) here is a graph of the cpu and memory as for now (with hervé PR tries during those last 24h on a specific folder on the current VM pkg.origin.jenkins.io): Capture d’écran 2023-10-18 à 14 26 03 Capture d’écran 2023-10-18 à 14 29 17

thanks! do you mind also checking the other CPU usages (I/O, sys and free)? CPU contention is not always at the user process level so better be safe than sorry ;)

smerle33 commented 11 months ago
Capture d’écran 2023-10-18 à 15 55 53 Capture d’écran 2023-10-18 à 15 56 21 Capture d’écran 2023-10-18 à 15 59 08
dduportal commented 10 months ago

Update after pairing session (and async work) with @smerle33 and @lemeurherve during the past 10 days:

TODO:

dduportal commented 10 months ago

Update:

TODO:

dduportal commented 10 months ago

Update:

dduportal commented 10 months ago

Update on the "Setup the command to trigger mirrorbits scan -all through kubernetes on the trusted agent":

# With $HOME/.kube/config setup with the same content as the credential
$ dduportal@agent:~$ kubectl --namespace=updates-jenkins-io exec updates-jenkins-io-mirrorbits-lite-67577f5bd4-fffwt --container=mirrorbits-lite -- echo OK
OK

TODO:

dduportal commented 10 months ago

Update:

TODO:

dduportal commented 10 months ago

Update:

TODO:

dduportal commented 10 months ago

Update:

Result: AWS S3 sync work: we can proceed to remove it from puppet (along with rclone)

Note: New error to investigate: Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:updates-jenkins-io:mirrorbits" cannot list resource "pods" in API group "" in the namespace "updates-jenkins-io" (on the helm chart size I believe)

dduportal commented 10 months ago

Update:

dduportal commented 10 months ago

WiP:

TODO:

Then

dduportal commented 10 months ago

Update:

$ for f in $(find /var/www/updates.jenkins.io -type f); do filename="$(basename "$f")"; echo "${filename##*.}" ;done  | sort -u
htaccess
html
json
txt
dduportal commented 10 months ago

Update:

  * Deployment to trusted.ci.jenkins.io: [Bump Packer Agent Templates (All-In-One) Version to 1.39.0 jenkins-infra#3177](https://github.com/jenkins-infra/jenkins-infra/pull/3177)

* Fine tune ingress:

  * The current update center site (updates.jenkins.io on the PKG VM) has the following file extensions:
$ for f in $(find /var/www/updates.jenkins.io -type f); do filename="$(basename "$f")"; echo "${filename##*.}" ;done  | sort -u
htaccess
html
json
txt
* Per type analysis for the ingress rules:

  * `htaccess` => Do not map to an URL, served by httpd, no mirror
  * `html` => Map to an URL (such as index pages). Either the ingress send to httpd (outbound bandwidth on Azure but reference) or we copy HTML files to mirror
  * `txt` => same as HTML. Either httpd or we copy to mirrors
  * `json` => **always** send to mirrorbits

Opened https://github.com/jenkins-infra/kubernetes-management/pull/4672 to fine tune the ingress. All requests ending in .html, .json and .txt are sent to mirrorbits

lemeurherve commented 10 months ago

WiP:

  • crawler update by @lemeurherve

The crawler is now uploading the update/ folder content to the file share and R2 buckets from trusted.ci.jenkins.io: https://github.com/jenkins-infra/crawler/blob/5b0906afe4850efe8ab3f7d9b5562829ba74d6ef/Jenkinsfile#L71-L93

As this job is using a linux agent and not the permanent agent like the update-center2 job, we had to add azcopy to the provisioning of the Linux packer images used by these agents. (aws-cli was already provisioned)

lemeurherve commented 10 months ago

Update:

The content on azure.updates.jenkins.io has been restored from a copy of updates.jenkins.io content.

We had to ensure that the content was synchronized in the correct destination folder, extracting the SAS token query string from the existing credentials to its own credentials so we can set the destination folder in the R2 bucket URL:

I've updated UC2 publish.sh script in the pull request and in the test branch pr-745 accordingly. As the updates-jenkins-io-file-share-url-with-sas-token credentials is still set on update-center2 job on trusted.ci.jenkins.io, I haven't deleted it yet.

As noted by @daniel-beck in https://github.com/jenkins-infra/crawler/pull/130#issuecomment-1812350610 (thanks btw, we totally missed that!):

Some crawlers don't work, but the existing files will ensure continued operation of the plugins relying of them. They'll just slowly become outdated.

The content from the FlywayInstaller and PerlInstaller crawlers isn't generated and thus synchronized anymore. We have to ensure we copy their JSONs from from pkg.origin.jenkins.io into the File Share and the R2 bucket.

Note: I've experimentally replaced updates.jenkins.io by azure.updates.jenkins.io in every .htaccess and *.html files of this copy with sed so all links and htaccess redirections point to azure.updates.jenkins.io This won't have any use in production as there won't be any azure.updates.jenkins.io anymore so it won't be carried over.

I'm now testing azure.updates.jenkins with a controller and https://github.com/jenkinsci/plugin-installation-manager-tool/ in Docker.

lemeurherve commented 10 months ago

We'll have to ensure .hpi links on plugin download pages like https://updates.jenkins.io/download/plugins/zos-connector/ are correctly redirected by Apache to get.jenkins.io:

$ curl --silent --show-error --output /dev/null --fail --write-out '%{redirect_url}' https://updates.jenkins.io/download/plugins/zos-connector/3.257.v41e144167971/zos-connector.hpi 
https://get.jenkins.io/plugins/zos-connector/3.257.v41e144167971/zos-connector.hpi

We should add/adapt datadog monitor on this kind of redirection.

lemeurherve commented 10 months ago

Local testing of azure.updates.jenkins.io with Docker and plugin-installation-manager-tool:

Running plugin-installation-manager-tool with default settings (updates.jenkins.io) ```console $ docker run -it jenkins/jenkins jenkins-plugin-cli --verbose -p ansicolor No .txt or .yaml file containing list of plugins to be downloaded entered. No directory to download plugins entered. Will use default of /usr/share/jenkins/ref/plugins Using update center https://updates.jenkins.io/update-center.json from JENKINS_UC environment variable Using experimental update center https://updates.jenkins.io/experimental/update-center.json from JENKINS_UC_EXPERIMENTAL environment variable Using incrementals mirror https://repo.jenkins-ci.org/incrementals from JENKINS_INCREMENTALS_REPO_MIRROR environment variable No CLI option or environment variable set for plugin info, using default of https://updates.jenkins.io/plugin-versions.json No war entered. Will use default of /usr/share/jenkins/jenkins.war Retrieving update center information Created cache at: /var/jenkins_home/.cache/jenkins-plugin-management-cli Update center URL: https://updates.jenkins.io/update-center.json?version=2.432 Cache miss for: update-center-2.432 Downloaded update-center-2.432 from https://updates.jenkins.io/dynamic-2.427/update-center.json (attempt 1 of 3) Cache miss for: experimental-update-center-2.432 Cache miss for: plugin-versions Downloaded plugin-versions from https://updates.jenkins.io/current/plugin-versions.json (attempt 1 of 3) Couldn't find checksum for ansicolor at version: latest ansicolor depends on: workflow-api 1215.v2b_ee3e1b_dd39 workflow-step-api 639.v6eca_cd8c04a_a_ Setting checksum for: ansicolor to bUHwcLjm/CLLqmHVugbg3zT6xyG6XvRVzBSN9wKUal8= Setting checksum for: ansicolor to bUHwcLjm/CLLqmHVugbg3zT6xyG6XvRVzBSN9wKUal8= Setting checksum for: ansicolor to bUHwcLjm/CLLqmHVugbg3zT6xyG6XvRVzBSN9wKUal8= Will install new plugin ansicolor 1.0.4 Will use url: https://updates.jenkins.io/download/plugins/ansicolor/1.0.4/ansicolor.hpi to download ansicolor plugin Downloaded ansicolor from https://ftp.belnet.be/mirror/jenkins/plugins/ansicolor/1.0.4/ansicolor.hpi (attempt 1 of 3) Downloaded and validated plugin ansicolor Checksum valid for: ansicolor Done ```
Running plugin-installation-manager-tool with custom settings (azure.updates.jenkins.io) ```console $ docker run -it \ --env JENKINS_UC="https://azure.updates.jenkins.io" \ --env JENKINS_UC_EXPERIMENTAL="https://azure.updates.jenkins.io/experimental" \ --env JENKINS_PLUGIN_INFO="https://azure.updates.jenkins.io/update-center.json" \ jenkins/jenkins jenkins-plugin-cli --verbose -p ansicolor No .txt or .yaml file containing list of plugins to be downloaded entered. No directory to download plugins entered. Will use default of /usr/share/jenkins/ref/plugins Using update center https://azure.updates.jenkins.io/update-center.json from JENKINS_UC environment variable Using experimental update center https://azure.updates.jenkins.io/experimental/update-center.json from JENKINS_UC_EXPERIMENTAL environment variable Using incrementals mirror https://repo.jenkins-ci.org/incrementals from JENKINS_INCREMENTALS_REPO_MIRROR environment variable Using plugin info https://azure.updates.jenkins.io/update-center.json from JENKINS_PLUGIN_INFO environment variable No war entered. Will use default of /usr/share/jenkins/jenkins.war Retrieving update center information Created cache at: /var/jenkins_home/.cache/jenkins-plugin-management-cli Update center URL: https://azure.updates.jenkins.io/update-center.json?version=2.432 Cache miss for: update-center-2.432 Downloaded update-center-2.432 from https://westeurope.cloudflare.jenkins.io/update-center.json (attempt 1 of 3) Cache miss for: experimental-update-center-2.432 Downloaded experimental-update-center-2.432 from https://westeurope.cloudflare.jenkins.io/experimental/update-center.json (attempt 1 of 3) Cache miss for: plugin-versions Downloaded plugin-versions from https://westeurope.cloudflare.jenkins.io/update-center.json (attempt 1 of 3) Couldn't find checksum for ansicolor at version: latest ansicolor depends on: workflow-api 1215.v2b_ee3e1b_dd39 workflow-step-api 639.v6eca_cd8c04a_a_ Setting checksum for: ansicolor to bUHwcLjm/CLLqmHVugbg3zT6xyG6XvRVzBSN9wKUal8= Setting checksum for: ansicolor to bUHwcLjm/CLLqmHVugbg3zT6xyG6XvRVzBSN9wKUal8= Setting checksum for: ansicolor to bUHwcLjm/CLLqmHVugbg3zT6xyG6XvRVzBSN9wKUal8= Will install new plugin ansicolor 1.0.4 Will use url: https://azure.updates.jenkins.io/download/plugins/ansicolor/1.0.4/ansicolor.hpi to download ansicolor plugin Downloaded ansicolor from https://ftp.belnet.be/mirror/jenkins/plugins/ansicolor/1.0.4/ansicolor.hpi (attempt 1 of 3) Downloaded and validated plugin ansicolor Checksum valid for: ansicolor Done ```

Diff

--- default settings (updates.jenkins.io)
+++ custom settings (azure.updates.jenkins.io)
@@ -1,19 +1,20 @@
 No .txt or .yaml file containing list of plugins to be downloaded entered.
 No directory to download plugins entered. Will use default of /usr/share/jenkins/ref/plugins
-Using update center https://updates.jenkins.io/update-center.json from JENKINS_UC environment variable
+Using update center https://azure.updates.jenkins.io/update-center.json from JENKINS_UC environment variable
-Using experimental update center https://updates.jenkins.io/experimental/update-center.json from JENKINS_UC_EXPERIMENTAL environment variable
+Using experimental update center https://azure.updates.jenkins.io/experimental/update-center.json from JENKINS_UC_EXPERIMENTAL environment variable
 Using incrementals mirror https://repo.jenkins-ci.org/incrementals from JENKINS_INCREMENTALS_REPO_MIRROR environment variable
-No CLI option or environment variable set for plugin info, using default of https://updates.jenkins.io/plugin-versions.json
+Using plugin info https://azure.updates.jenkins.io/update-center.json from JENKINS_PLUGIN_INFO environment variable
 No war entered. Will use default of /usr/share/jenkins/jenkins.war

 Retrieving update center information
 Created cache at: /var/jenkins_home/.cache/jenkins-plugin-management-cli
-Update center URL: https://updates.jenkins.io/update-center.json?version=2.432
+Update center URL: https://azure.updates.jenkins.io/update-center.json?version=2.432
 Cache miss for: update-center-2.432
-Downloaded update-center-2.432 from https://updates.jenkins.io/dynamic-2.427/update-center.json (attempt 1 of 3)
+Downloaded update-center-2.432 from https://westeurope.cloudflare.jenkins.io/update-center.json (attempt 1 of 3)
 Cache miss for: experimental-update-center-2.432
+Downloaded experimental-update-center-2.432 from https://westeurope.cloudflare.jenkins.io/experimental/update-center.json (attempt 1 of 3)
 Cache miss for: plugin-versions
-Downloaded plugin-versions from https://updates.jenkins.io/current/plugin-versions.json (attempt 1 of 3)
+Downloaded plugin-versions from https://westeurope.cloudflare.jenkins.io/update-center.json (attempt 1 of 3)
 Couldn't find checksum for ansicolor at version: latest

 ansicolor depends on: 
@@ -23,10 +24,9 @@
 Setting checksum for: ansicolor to bUHwcLjm/CLLqmHVugbg3zT6xyG6XvRVzBSN9wKUal8=
 Setting checksum for: ansicolor to bUHwcLjm/CLLqmHVugbg3zT6xyG6XvRVzBSN9wKUal8=
 Will install new plugin ansicolor 1.0.4
-Will use url: https://updates.jenkins.io/download/plugins/ansicolor/1.0.4/ansicolor.hpi to download ansicolor plugin
+Will use url: https://azure.updates.jenkins.io/download/plugins/ansicolor/1.0.4/ansicolor.hpi to download ansicolor plugin
 Downloaded ansicolor from https://ftp.belnet.be/mirror/jenkins/plugins/ansicolor/1.0.4/ansicolor.hpi (attempt 1 of 3)
 Downloaded and validated plugin ansicolor
 Checksum valid for: ansicolor
 Done
lemeurherve commented 10 months ago

To check Jenkins itself with azure.jenkins.io, I've blackholed updates.jenkins.io by adding 127.0.0.1 updates.jenkins.io to my /etc/host.

I then ran the controller image with docker run -it jenkins/jenkins.

Note that env vars like the ones I've passed to plugin-installation-manager-tool aren't taken in account by Jenkins.

Next, I've set https://azure.updates.jenkins.io in "Manage Jenkins > Plugins > Advanced settings > Update Site > URL"

Then, I've went to "Manage Jenkins > Plugins > Updates" and clicked on the refresh button to retrieve the update-center.json from azure.jenkins.io.

I've finally confirmed that the list of plugins was available at "Manage Jenkins > Plugins > Available plugins".

What I've also understood by getting an (expected) error when trying to install a plugin from there is that Jenkins doesn't use the same mechanism as plugin-installation-manager-tool to retrieve the plugin download URL:

lemeurherve commented 10 months ago

From these elements local tests of azure.updates.jenkins.io seems conclusive.

It also means that we'll probably have to plan a brownout to test our prototype in situ.

lemeurherve commented 9 months ago
  • "Setup the command to trigger mirrorbits scan -all through kubernetes on the trusted agent"
    • mirrorbits scan works very well from the script using the embeded credential
    • [...]
    • Please note that, the script fails if another scan is already in progress: not a nominal case for now though

Coming back to this initial hypothesis: as the update-center2 job will run every few minutes and trigger a scan, we don't need and have to disable the 5 min cron scan configured by default on mirrorbits via its secrets (RepositoryScanInterval: 5) to avoid failing the synchronization job.

lemeurherve commented 9 months ago

Note: we can have R2 buckets in China with CloudFlare 🎉

Related open help desk issue that could be resolved with it in the future:

dduportal commented 8 months ago

Update:

dduportal commented 8 months ago

Todo: