Closed dduportal closed 16 hours ago
First set of options after initial discussions:
mirrorbits
to force this mirror to be picked when requests comes from West Russia (instead of fall backing to Hostico in Romania)
I tried to contact the mirror.yandex team via the official email address, but got no reply. Maybe @w-e-g could directly contact the engineer he mentioned.
My friend pinged a man who worked on mirror.yandex.ru, but no answer at all.
We've had a response from yandex to the jenkins infra team FYI, they would like to make their mirror an official one.
We're going to add this mirror to get.jenkins.io today (Monday 24 June 2024) between 01:35pm UTC and 02:00pm UTC (along with https://github.com/jenkins-infra/helpdesk/issues/4145)
Yandex mirror has been added with success 🥳
https://get.jenkins.io/debian/jenkins_2.463_all.deb?mirrorstats
Note: we had the same issue as for Hostico (mirror marked as "down", cf https://github.com/jenkins-infra/helpdesk/issues/3976#issuecomment-2158466326), resolved after some times.
Not sure what worked, mirrorbits edit
wasn't at first, we then deleted and readded the mirror but it was only after seeing files being individually scanned in mirrorbits logs that the mirror appeared as "up".
Is this the cause now for:
java.io.IOException: Unable to tunnel through proxy. Proxy returns "HTTP/1.1 403 Forbidden"
at java.base/sun.net.www.protocol.http.HttpURLConnection.doTunneling0(HttpURLConnection.java:2271)
at java.base/sun.net.www.protocol.http.HttpURLConnection.doTunneling(HttpURLConnection.java:2143)
at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
at java.base/sun.net.www.protocol.http.HttpURLConnection.followRedirect0(HttpURLConnection.java:2909)
at java.base/sun.net.www.protocol.http.HttpURLConnection.followRedirect(HttpURLConnection.java:2818)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1929)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1599)
at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:223)
at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1323)
Caused: java.io.IOException: Failed to load https://updates.jenkins.io/download/plugins/build-timeout/1.33/build-timeout.hpi to /var/lib/jenkins/plugins/build-timeout.jpi.tmp
at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1334)
Caused: java.io.IOException: Failed to download from https://updates.jenkins.io/download/plugins/build-timeout/1.33/build-timeout.hpi (redirected to: https://mirror.yandex.ru/mirrors/jenkins/plugins/build-timeout/1.33/build-timeout.hpi)
at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1368)
at hudson.model.UpdateCenter$DownloadJob._run(UpdateCenter.java:1925)
at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:2237)
at hudson.model.UpdateCenter$DownloadJob.run(UpdateCenter.java:1899)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:121)
at java.base/java.lang.Thread.run(Thread.java:1583)
yandex is blocked in a lot of corporate environments.
yandex is blocked in a lot of corporate environments.
Hi @Unknown-Guy , there isn't a lot we can do on the infra, unless one of these corporations helps by providing a (public) download mirror in the Russia area. As a reminder, the Jenkins project is living on sponsorships: we do not have the financial resource to cover all cases and quality of service unless users helps by sponsoring.
Other alternative is to contact companies such as CloudBees (and pay their product) to ensure this kind of constraint is solved for your company.
I don't need the yandex mirror at all, we are in EU, it seems, after this change, closest mirror is yandex one which is blocked, some countries even block yandex (Finland and Norway come to mind). So this is breaking change.
Looks like this is vice versa situation.
@dduportal It is time to add mirror selection preferences on Jenkins end, what do you think?
Maybe Jenkins team just could use CountryOnly
option on mirrorbits, to limit Yandex to RU?
That would be (IMO) the easiest solution.
Ref:
@lemeurherve I've seen you comment come and go, but in case the CountryOnly
code *is set*, yet EU users are routed to Yandex, maybe the CountryCodes
were incorrectly detected during "add" operation?
Something along the following code-path - https://github.com/etix/mirrorbits/blob/master/rpc/rpc.go#L329-L358
@QMiqTx6DHn1bA9yaNaAbsD3CLG8gTmd4 This addition will fix current problem (for some EU countries), but will lead for new problems:
CountryCodes
blockingI think we need another issue for this discussion, what do you think?
I've seen you comment come and go
Yes sorry I deleted my comment which was wrong, this CountryOnly
setting is currently not set.
I should have edited my comment instead of deleting it.
Hi folks, thanks for the pointers and details.
is time to add mirror selection preferences on Jenkins end, what do you think?
If this is something which look interesting to you, you can raise a feature request on mirrorbits open source software we use: if they implement it then we could benefit from it. However, I don't think it is a wise choice as it would allow anyone to bypass the redirections, which defeat the whole purpose (ref. https://www.sonatype.com/blog/maven-central-and-the-tragedy-of-the-commons). If the goal is to solve local optimums and break the other, then it start to be a problem. If you have a solution which does not involve weeks of works for the infra contributors, please share it with us we are all hears!
Reopening as this is a problem to be solved
Proposal based on the above comments from different users + internal discussions:
Any objection of thing to consider if we set up this change next week?
@dduportal Can you describe what Jenkins will do if first mirror picked is not reachable for some reason? Will it fail or pick second mirror? If it can pick second mirror - it will fix all these problems, don't you think so? Or it is hard to implement? (I'm not familiar with mirrorbits, sorry)
@dduportal Can you describe what Jenkins will do if first mirror picked is not reachable for some reason? Will it fail or pick second mirror? If it can pick second mirror - it will fix all these problems, don't you think so? Or it is hard to implement? (I'm not familiar with mirrorbits, sorry)
No problem, this is a good question. Mirrorbits only emits an HTTP 3xx redirection, with the Location header set to the mirror URL closest to you. There is no dynamic reverse proxying so we cannot do such feature.
the goal for it is to be simple to defer traffic to mirrors (handling only http/3xx request does not cost much). So it can handle a lot of traffic.
Proposal based on the above comments from different users + internal discussions:
* Set Yandex mirror to Russian country only => this mirror would only be selected for Russian users * All East-Europe traffic will get to Hostico due to this one => if is not sustainable for them, then we'll also restrict them to Romania only * It means, in the worst case scenario, we'll back to the situation one month ago with traffic going to Germany and/or Belgium (except for Russian and Romania) so still better than before
Any objection of thing to consider if we set up this change next week?
We are starting the operation
Proposal based on the above comments from different users + internal discussions:
* Set Yandex mirror to Russian country only => this mirror would only be selected for Russian users * All East-Europe traffic will get to Hostico due to this one => if is not sustainable for them, then we'll also restrict them to Romania only * It means, in the worst case scenario, we'll back to the situation one month ago with traffic going to Germany and/or Belgium (except for Russian and Romania) so still better than before
Any objection of thing to consider if we set up this change next week?
We are starting the operation
Operation finished: both Yandex and Hostico mirrors are now set up as "Country only". It means than these mirrors will only be selected to serve traffic originating from the country on which they are marked from (Russia and Romania).
You can see this result in https://get.jenkins.io/war/2.464/jenkins.war?mirrorlist as example:
Ping @anonimus-asuseeepc (ref. https://github.com/jenkins-infra/helpdesk/issues/4160) @QMiqTx6DHn1bA9yaNaAbsD3CLG8gTmd4 @Stikus @Unknown-Guy could you confirm if it fixes your problems?
It seems all is good now, our Jenkins updated all plugins without issues.
It seems all is good now, our Jenkins updated all plugins without issues.
Many thanks!
Service(s)
get.jenkins.io
Summary
As per https://github.com/jenkins-infra/helpdesk/issues/4128#issuecomment-2160116396, the Jenkins Russian users recently went trough troubles when it comes to downloading Jenkins plugins/installers through our mirror system
get.jenkins.io
:As such, this issue is to find a solution (and track its resolution up to completion) from the Jenkins project infrastructure. The goals are:
Reproduction steps
No response