search

jenkins-infra / helpdesk

Open your Infrastructure related issues here for the Jenkins project
https://github.com/jenkins-infra/helpdesk/issues/new/choose
16 stars 10 forks source link

[INFRA-2393] updates.jenkins.io/download/plugins is unreliably slow and failing #2010

Closed jenkins-infra-bot closed 3 years ago

jenkins-infra-bot commented 4 years ago

updates.jenkins.io/download/plugins is unreliably slow and failing

 

--2019-12-20 09:13:00--  https://updates.jenkins.io/download/plugins/openshift-client/1.0.32/openshift-client.hpi
Resolving updates.jenkins.io (updates.jenkins.io)... 52.202.51.185
Connecting to updates.jenkins.io (updates.jenkins.io)|52.202.51.185|:443... failed: Operation timed out.
Retrying.--2019-12-20 09:14:16--  (try: 2)  https://updates.jenkins.io/download/plugins/openshift-client/1.0.32/openshift-client.hpi
Connecting to updates.jenkins.io (updates.jenkins.io)|52.202.51.185|:443...
connected.HTTP request sent, awaiting response... 302 Found
Location: http://mirrors.jenkins-ci.org/plugins/openshift-client/1.0.32/openshift-client.hpi [following]
--2019-12-20 09:14:56--  http://mirrors.jenkins-ci.org/plugins/openshift-client/1.0.32/openshift-client.hpi
Resolving mirrors.jenkins-ci.org (mirrors.jenkins-ci.org)... 52.202.51.185
Connecting to mirrors.jenkins-ci.org (mirrors.jenkins-ci.org)|52.202.51.185|:80...

 
Originally reported by akram, imported from: updates.jenkins.io/download/plugins is unreliably slow and failing
  • assignee: olblak
  • status: Closed
  • priority: Blocker
  • resolution: Fixed
  • resolved: 2020-12-07T03:31:36+01:00
  • imported: 2022/01/10
jenkins-infra-bot commented 4 years ago

ianw:

Something's not right .. Similar question just posted on SO, within an hour before: Jenkins Plugin Manager fails to install/update a plugin because of a read timeout

Some plugin files were downloaded and installed successfully, others failed with the following error:

    Failed to install git-client
    java.net.SocketTimeoutException: Read timed out

Downloading via curl was successful, although I noticed that after successful connection, the download process was stalled at 0% for about a minute. Then it suddenly started, and finished in seconds.

jenkins-infra-bot commented 4 years ago

acsipak:

Ian Williams my SO question does not cover the same issue. The problem I covered in the question kept happening to us for months. As far as I see, the problem covered in this Jira issue surfaced on 2019-12-20.

jenkins-infra-bot commented 4 years ago

speranskiy:

Same here

Downloading plugin: variant from https://updates.jenkins.io/2.204/latest/variant.hpicurl: (28) Connection timed out after 20000 milliseconds
07:42:07 Failure (28) Retrying in 1 seconds...
curl: (28) Connection timed out after 20001 milliseconds
07:42:09 Failure (28) Retrying in 1 seconds...curl: (28) Connection timed out after 20000 milliseconds
07:42:12 Failure (28) Retrying in 1 seconds...
curl: (28) Operation timed out after 20000 milliseconds with 0 out of 0 bytes received
07:42:14 Failure (28) Retrying in 1 seconds...
jenkins-infra-bot commented 4 years ago

oleg_nenashev:

It keeps failing here and there. Apart from the impact on Jenkins users, jenkins.io local development is also almost blocked by the issue.

jenkins-infra-bot commented 4 years ago

vfreex:

Same here. Not sure if anyone will bring the update center back during the holidays.

jenkins-infra-bot commented 4 years ago

oleg_nenashev:

I investigated the issues a bit, and it looks like the only thing I can do with my permissions is the service restart by updating https://github.com/jenkins-infra/jenkins-infra . I am not sure it is a good idea, waiting for Olivier Vernin to comment on it. IIUC all people with SSH access to the host are on holidays. CC Arnaud Héritier Andrew Bayer Daniel Beck R. Tyler Croy

jenkins-infra-bot commented 4 years ago

olblak:

I confirm that something unusual is happening on that machine every day

jenkins-infra-bot commented 4 years ago

olblak:

I saw a lot of errors regarding not enough database connection available, and it appeared that three different version of postgresql were running on that machine Postgres 9.3, 9.5 and 10. I stopped version 9.5 and 10, the restarted postresreql 9.3 and apache. Mirrorbrain seems to be happy again

jenkins-infra-bot commented 4 years ago

olblak:

I'll look further in one week

jenkins-infra-bot commented 4 years ago

vfreex:

Olivier Vernin It seems to me that the update center backs to normal. Thank you.

jenkins-infra-bot commented 4 years ago

alexhraber:

Hello, this is happening again.

I'm building a docker image and pulling plugins per the officially supported `plugin.txt` installation method in docker.

I run my image build locally and it fails randomly pulling various plugins (never the same plugins). Then I've also tried w/ Google Cloud Build and it fails for other various plugins. I can go to the updates.jenkins.io/latest site and see the plugins and download the ones that are failing, so it seems to be higher than normal load or something?

I first notice this ~ 20 minutes ago, and have had anywhere from 1 to 10 plugins fail, each a unique list of plugins failing in ~20 attempts.

 

Thanks, sorry to bring up a closed ticket.

Edit1: this has been working in the past, going on 2 years in docker using this plugin installation/update step in my Dockerfile

 

Edit2:

Random list of plugin failures example 1: 

Some plugins failed to download! Not downloaded: ssh-agent
Not downloaded: ownership
Not downloaded: pipeline-github-lib
Not downloaded: ant
Not downloaded: extra-columns
Not downloaded: multiple-scms
Not downloaded: kpp-management-plugin
Not downloaded: workflow-cps-global-lib
Not downloaded: jquery-detached
Not downloaded: ghprb

Random list of plugin failures example 2: 

Some plugins failed to download! Not downloaded: icon-shim
Not downloaded: shelve-project-plugin
Not downloaded: github-branch-source
Not downloaded: mailer
Not downloaded: claim
Not downloaded: ssh-credentials
Not downloaded: ssh-agent
Not downloaded: email-ext
Not downloaded: simple-theme-plugin
Not downloaded: lockable-resources
Not downloaded: jquery

 Random list of plugin failures example 3:

Some plugins failed to download! Not downloaded: node-iterator-api
Not downloaded: chromedriverStep
Not downloaded: pagerdutyStep
Not downloaded: pipeline-rest-apiStep
Not downloaded: credentialsStep
Not downloaded: build-monitor-pluginStep
Not downloaded: build-failure-analyzerStep

The live build failure looks like so (one entry for each plugin that randomly fails to install):

Failed to download plugin: jquery or jquery-plugin
jenkins-infra-bot commented 4 years ago

olblak:

I just restarted apache service on that machine, it should now go back to normal

jenkins-infra-bot commented 4 years ago

alexhraber:

you rock Olivier Vernin, trying again 

jenkins-infra-bot commented 4 years ago

alexhraber:

success. Thanks much Olivier Vernin 

jenkins-infra-bot commented 4 years ago

sapcik:

Hey Olivier Vernin,

I think it needs to be restarted again, as we can't install properly even clean Jenkins instance. 

It needs a lot of time to load http://updates.jenkins-ci.org/update-center.json 

Thanks,
Wojciech

jenkins-infra-bot commented 4 years ago

yogeek:

Hello  

Same problem here : all my docker builds are failing with "Connection timed out" today...

jenkins-infra-bot commented 4 years ago

olblak:

While it was smaller than yesterday, indeed this morning we had a peak on this machine as you can see on this graph.

Yesterday we configured a CDN for pkg.jenkins.io, this reduced the load on that machine and it also beneficiate to updates.jenkins.io, we still have some improvement that should come

jenkins-infra-bot commented 4 years ago

d4nyll:

Hi Olivier Vernin, I am not able to download any plugins whilst building my Docker image.

curl: (28) Resolving timed out after 20528 milliseconds
{{ 20:11:22 Failure (28) Retrying in 1 seconds...}}
{{ curl: (28) Resolving timed out after 20528 milliseconds}}
{{ 20:11:22 Failure (28) Retrying in 1 seconds...}}
{{ curl: (28) Resolving timed out after 20530 milliseconds}}
{{ 20:11:22 Failure (28) Retrying in 1 seconds...}}
{{ curl: (28) Resolving timecdu rol:u t( 28a)f tReers o2l0v5i2n9g mtiilmleids eocuotn dasf}}
{{ ter 20527 milliseconds}}
{{ 20:11:22 Failure (28) Retrying in 1 seconds...}}
{{ 20:11:22 Failure (28) Retrying in 1 seconds...}}
{{ curl: (28) Resolving timed out after 20525 milliseconds}}
{{ curl: (28) Resolving timed out after 20527 milliseconds}}
{{ curl: (28) Resolving timed out after 20530 milliseconds}}
{{ curl: (28) Resolving timed out after 20528 milliseconds}}
{{ 20:11:22 Failure (28) Retrying in 1 seconds...}}
{{ 20:11:22 Failure (28) Retrying in 1 seconds...}}
{{ 20:11:22 Failure (28) Retrying in 1 seconds...}}
{{ 20:11:22 Failure (28) Retrying in 1 seconds...}}

Finishing with:

 

WAR bundled plugins:
Installed plugins:
*:
Some plugins failed to download! Not downloaded: authorize-project
Not downloaded: git
Not downloaded: credentials-binding
Not downloaded: antisamy-markup-formatter
Not downloaded: ant
Not downloaded: timestamper
Not downloaded: pam-auth
Not downloaded: mailer
Not downloaded: pipeline-github-lib
Not downloaded: cloudbees-folder
Not downloaded: subversion
Not downloaded: ws-cleanup
Not downloaded: build-timeout
Not downloaded: matrix-auth
Not downloaded: ssh-slaves
Not downloaded: ldap
Not downloaded: email-ext
Not downloaded: configuration-as-code
Not downloaded: pipeline-stage-view
Not downloaded: github-branch-source
Not downloaded: gradle
Not downloaded: workflow-aggregator

 

Is there a more permanent solution to restarting the server manually each time?

jenkins-infra-bot commented 4 years ago

yogeek:

The same problem occured today 4 times in a row. 

Have the CDN already been set up ?

jenkins-infra-bot commented 4 years ago

olblak:

I made multiple configuration tweak on the machine hosting the update center

Since then the machine has been quite stable in term of response time as you can see on the diagram

jenkins-infra-bot commented 3 years ago

mpawlowski:

Hi Olivier Vernin

today I faced similar issue during downloading plugins:

Can you take a look?

jenkins-infra-bot commented 3 years ago

jaapcrezee:

Caused: java.io.IOException: Failed to load https://updates.jenkins.io/download/plugins/workflow-step-api/2.23/workflow-step-api.hpi to /var/lib/jenkins/plugins/workflow-step-api.jpi.tmp
{{ at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1287)}}
Caused: java.io.IOException: Failed to download from https://updates.jenkins.io/download/plugins/workflow-step-api/2.23/workflow-step-api.hpi (redirected to: https://get.jenkins.io/plugins/workflow-step-api/2.23/workflow-step-api.hpi)
{{ at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1321)}}
{{ at hudson.model.UpdateCenter$DownloadJob._run(UpdateCenter.java:1869)}}
{{ at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:2147)}}
{{ at hudson.model.UpdateCenter$DownloadJob.run(UpdateCenter.java:1843)}}
{{ at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)}}
{{ at java.util.concurrent.FutureTask.run(FutureTask.java:266)}}
{{ at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:118)}}
{{ at java.lang.Thread.run(Thread.java:748)}}

jenkins-infra-bot commented 3 years ago

kkumarma:

it seems all are facing the same issue .....Do any one know what is the issue ? I hope, Things will be fixed soon

jenkins-infra-bot commented 3 years ago

tony_k:

2020-11-13T12:58:48.447446915Z applying Jenkins configuration 2020-11-13T12:58:48.447483236Z disable Setup Wizard 2020-11-13T12:58:48.471116665Z remove all plugins from shared volume 2020-11-13T12:58:48.472091072Z download plugins 2020-11-13T12:58:48.498435340Z WARN: install-plugins.sh is deprecated, please switch to jenkins-plugin-cli 2020-11-13T12:58:48.503519327Z Creating initial locks... 2020-11-13T12:58:48.565091458Z Analyzing war /usr/share/jenkins/jenkins.war... 2020-11-13T12:58:48.683618486Z Registering preinstalled plugins... 2020-11-13T12:59:54.373155697Z Using version-specific update center: https://updates.jenkins.io/dynamic-2.248/... 2020-11-13T12:59:54.373186817Z Downloading plugins... 2020-11-13T12:59:54.376165938Z Downloading plugin: ansicolor from https://updates.jenkins.io/download/plugins/ansicolor/0.7.3/ansicolor.hpi

a few plugin downloads seem to succeed here, but then...

2020-11-13T12:59:54.491017865Z Downloading plugin: workflow-aggregator from https://updates.jenkins.io/download/plugins/workflow-aggregator/2.6/workflow-aggregator.hpi 2020-11-13T12:59:55.385793396Z curl: (22) The requested URL returned error: 404 2020-11-13T12:59:55.389307941Z 12:59:55 Failure (22) Retrying in 1 seconds... 2020-11-13T13:01:04.898937987Z curl: (22) The requested URL returned error: 504 2020-11-13T13:01:04.899447311Z curl: (22) The requested URL returned error: 504 2020-11-13T13:01:04.903211958Z 13:01:04 Failure (22) Retrying in 1 seconds... 2020-11-13T13:01:04.903448030Z 13:01:04 Failure (22) Retrying in 1 seconds... 2020-11-13T13:01:04.917791062Z curl: (22) The requested URL returned error: 504 2020-11-13T13:01:04.920893034Z 13:01:04 Failure (22) Retrying in 1 seconds... 2020-11-13T13:01:04.923511774Z curl: (22) The requested URL returned error: 504 2020-11-13T13:01:04.926831128Z 13:01:04 Failure (22) Retrying in 1 seconds... 2020-11-13T13:01:04.942433999Z curl: (22) The requested URL returned error: 504 2020-11-13T13:01:04.945480121Z 13:01:04 Failure (22) Retrying in 1 seconds... 2020-11-13T13:01:42.084727197Z curl: (22) The requested URL returned error: 502 2020-11-13T13:01:42.088248487Z 13:01:42 Failure (22) Retrying in 1 seconds...

combination of 404, 504 and 502 until retry attempts are exhausted and then jenkins goes on to install with no plugins

jenkins-infra-bot commented 3 years ago

JIRAUSER130169:

+1 - Any update on when this can be addressed. All plugin downloads/updates are failing. Please advise.

Caused: java.io.IOException: Failed to download from https://updates.jenkins.io/download/plugins/config-file-provider/3.7.0/config-file-provider.hpi (redirected to: https://get.jenkins.io/plugins/config-file-provider/3.7.0/config-file-provider.hpi)
at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1321)
at hudson.model.UpdateCenter$DownloadJob._run(UpdateCenter.java:1869)
at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:2147)
at hudson.model.UpdateCenter$DownloadJob.run(UpdateCenter.java:1843)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:118)
at java.lang.Thread.run(Thread.java:748)
2020-11-13 18:31:34.034+0000 [id=204] INFO h.model.UpdateCenter$DownloadJob#run: Starting the installation of branch-api on behalf of matrixpsuser
2020-11-13 18:31:35.827+0000 [id=204] INFO h.m.UpdateCenter$UpdateCenterConfiguration#download: Downloading branch-api
2020-11-13 18:31:35.828+0000 [id=204] SEVERE h.model.UpdateCenter$DownloadJob#run: Failed to install branch-api
java.io.IOException: Server returned HTTP response code: 503 for URL: https://get.jenkins.io/plugins/branch-api/2.6.3/branch-api.hpi
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1900)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
at sun.net.www.protocol.http.HttpURLConnection.getHeaderField(HttpURLConnection.java:3061)
at java.net.URLConnection.getHeaderFieldLong(URLConnection.java:629)
at java.net.URLConnection.getContentLengthLong(URLConnection.java:501)
at java.net.URLConnection.getContentLength(URLConnection.java:485)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getContentLength(HttpsURLConnectionImpl.java:412)
at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1264)

jenkins-infra-bot commented 3 years ago

markewaite:

Resolved Nov 14, 2020 by switching get.jenkins.io to a different server that uses different storage

jenkins-infra-bot commented 3 years ago

tony_k:

thanks Mark Waite !

i had a successful install about an hour ago, all good !

jenkins-infra-bot commented 3 years ago

JIRAUSER130252:

I'm still receiving timeouts when trying to pull plugins. 

 

> bootstrap4-api depends on font-awesome-api:5.15.1-1,jquery3-api:3.5.1-2,popper-api:1.16.0-7
Downloading plugin: font-awesome-api from https://updates.jenkins.io/dynamic-2.248//latest/font-awesome-api.hpi
Downloading plugin: jquery3-api from https://updates.jenkins.io/dynamic-2.248//latest/jquery3-api.hpi
Downloading plugin: popper-api from https://updates.jenkins.io/dynamic-2.248//latest/popper-api.hpi
> checks-api depends on plugin-util-api:1.4.0,workflow-cps:2.83,workflow-job:2.40,workflow-step-api:2.23
curl: (22) The requested URL returned error: 404 Not Found
16:07:37 Failure (22) Retrying in 1 seconds...
curl: (22) The requested URL returned error: 404 Not Found
curl: (22) The requested URL returned error: 404 Not Found
16:07:37 Failure (22) Retrying in 1 seconds...
16:07:37 Failure (22) Retrying in 1 seconds...
ccuurrll:: ((2222)) The rTehqeu ersted Uequested URL returned error: 404 Not Found
RL returned error: 404 Not Found
16:07:37 Failure (22) Retrying in 1 seconds...
16:07:37 Failure (22) Retrying in 1 seconds...
curl: (28) Connection timed out after 20000 milliseconds
16:07:40 Failure (28) Retrying in 1 seconds...
curl: (28) Connection timed out after 20000 milliseconds
16:07:41 Failure (28) Retrying in 1 seconds...
curl: (28) Connection timed out after 20001 milliseconds
16:07:41 Failure (28) Retrying in 1 seconds...

 

 

Some plugins failed to download! Not downloaded: workflow-api
Not downloaded: pipeline-stage-view
Not downloaded: structs
Not downloaded: cloudbees-folder
Not downloaded: script-security
Not downloaded: authentication-tokens
Not downloaded: apache-httpcomponents-client-4-api
Not downloaded: kubernetes-credentials
Not downloaded: github-branch-source
Not downloaded: workflow-cps
Not downloaded: workflow-support
Not downloaded: workflow-durable-task-step
Not downloaded: pipeline-input-step
Not downloaded: workflow-cps-global-lib
Not downloaded: workflow-basic-steps
Not downloaded: credentials
Not downloaded: pipeline-build-step
Not downloaded: workflow-job
Not downloaded: scm-api
Not downloaded: workflow-scm-step
Not downloaded: pipeline-milestone-step
Not downloaded: pipeline-model-definition
Not downloaded: pipeline-stage-step

jenkins-infra-bot commented 3 years ago

olblak:

Stephen Gordon Can you confirm that you can reach one of the mirrors from your location
-> https://get.jenkins.io/plugins/plugin-util-api/1.4.0/plugin-util-api.hpi?mirrorlist

curl -I -L https://get.jenkins.io/plugins/plugin-util-api/1.4.0/plugin-util-api.hpi should return 200 OK

I just took one of the plugins you are trying to download to see if it's available

jenkins-infra-bot commented 3 years ago

JIRAUSER130252:

Olivier Vernin yes this worked for me. 

jenkins-infra-bot commented 3 years ago

markewaite:

As far as I can tell, this is resolved