get.jenkins.io cannot geolocate my IP address on 50% of requests

[MarkEWaite]

Service(s)

get.jenkins.io

Summary

Alternating requests to get.jenkins.io show that half the time it cannot resolve my geographic location

Reproduction steps

1. Open https://get.jenkins.io/plugins/git/5.3.0/git.hpi?mirrorlist multiple times in succession. Sometimes it will locate the IP address and look like this:

2. Sometimes it will not geo-locate the IP address and will look like this:

[dduportal]

Multiple possible causes, need to investigate:

• The pod introduced in https://github.com/jenkins-infra/helpdesk/issues/4240 to manage the GeoIP database is in CrashLoopback off since yesterday's Kubernetes upgrade
• Errors logs for get-jenkins-io are polluted by errors such as:
  • 2024/08/24 07:37:33.902 UTC HTTP error: write tcp 10.100.8.14:8080->10.100.8.1:34408: write: broken pipe
  • http: superfluous response.WriteHeader call from github.com/etix/mirrorbits/http.(*HTTP).mirrorStatsHandler (http.go:607)

[dduportal]

First set of actions:

• Disabled the GeoIP data container (set deployment manually to zero replicas)
• Deleting the 2 get.jenkins.io pods one after the other

[dduportal]

First set of actions:

* Disabled the GeoIP data container (set deployment manually to zero replicas)

* Deleting the 2 get.jenkins.io pods one after the other

No results: I can still see the error :|

http: superfluous response.WriteHeader call from github.com/etix/mirrorbits/http.(*HTTP).mirrorStatsHandler (http.go:607)

This is an internal HTTP error :'(Thrown by https://github.com/etix/mirrorbits/blob/e83e56ac6496a3643d18a731324bd266f75f7b32/http/http.go#L607

[dduportal]

Could be related to https://github.com/jenkins-infra/docker-mirrorbits/pull/18. Gotta try with @timja 's proposal (using the Debian package instead of our custom build of mirrorbits)

[dduportal]

• Root cause described in https://github.com/jenkins-infra/helpdesk/issues/4240#issuecomment-2308222117 (Azure File share)

  • Could not even delete the FileHandle (as pointed out by Microsoft Support pages with their -yikes- PowerShell commands)
  • Triggered by the cluster 1.29 upgrade yesterday

• Fixed the problem (at least I cannot reproduce it on my machine after the operation while I was hitting it on 80% requests) by using the "Nuclear option" (3 min downtime on get.jenkins.io):

  • manually deleting the Azure File Share
  • Deleted the geoipdata PVC
  • Scaled down all the mirrorbits instances
  • Ran the Terraform Azure project to recreate it empty
  • Downloaded the GeoIP databases manually once from my personal account (as we are currently rate limited...)
  • Uploaded the files to the new (empty) file share
  • Re-applied the get.jio and updates.jio kubernetes helmfiles

[dduportal]

Added the following 2 hotfixes to ensure automation does NOT try to re-create the geoipdata stuff:

• https://github.com/jenkins-infra/kubernetes-management/commit/0e863251438cfb99de4447f5eb5afddcf2dcf2ae
• https://github.com/jenkins-infra/azure/commit/4058d7f9bf85db67eb9d1d27be0a105c45a75e9b

Reopened https://github.com/jenkins-infra/helpdesk/issues/4240 so that we can implement another solution which does not involve using ReadWriteMany PV.

@MarkEWaite @daniel-beck can you confirm the problem is gone for you?

[MarkEWaite]

Thanks very much @dduportal . I confirm that the problem is gone for me.

[dduportal]

Thanks!