Closed jenkins-infra-bot closed 3 years ago
ianw:
Something's not right .. Similar question just posted on SO, within an hour before: Jenkins Plugin Manager fails to install/update a plugin because of a read timeout
Some plugin files were downloaded and installed successfully, others failed with the following error:
Failed to install git-client
java.net.SocketTimeoutException: Read timed outDownloading via curl was successful, although I noticed that after successful connection, the download process was stalled at 0% for about a minute. Then it suddenly started, and finished in seconds.
Ian Williams my SO question does not cover the same issue. The problem I covered in the question kept happening to us for months. As far as I see, the problem covered in this Jira issue surfaced on 2019-12-20.
Same here
Downloading plugin: variant from https://updates.jenkins.io/2.204/latest/variant.hpicurl: (28) Connection timed out after 20000 milliseconds
07:42:07 Failure (28) Retrying in 1 seconds...
curl: (28) Connection timed out after 20001 milliseconds
07:42:09 Failure (28) Retrying in 1 seconds...curl: (28) Connection timed out after 20000 milliseconds
07:42:12 Failure (28) Retrying in 1 seconds...
curl: (28) Operation timed out after 20000 milliseconds with 0 out of 0 bytes received
07:42:14 Failure (28) Retrying in 1 seconds...
It keeps failing here and there. Apart from the impact on Jenkins users, jenkins.io local development is also almost blocked by the issue.
Same here. Not sure if anyone will bring the update center back during the holidays.
I investigated the issues a bit, and it looks like the only thing I can do with my permissions is the service restart by updating https://github.com/jenkins-infra/jenkins-infra . I am not sure it is a good idea, waiting for Olivier Vernin to comment on it. IIUC all people with SSH access to the host are on holidays. CC Arnaud Héritier Andrew Bayer Daniel Beck R. Tyler Croy
I confirm that something unusual is happening on that machine every day
I saw a lot of errors regarding not enough database connection available, and it appeared that three different version of postgresql were running on that machine Postgres 9.3, 9.5 and 10. I stopped version 9.5 and 10, the restarted postresreql 9.3 and apache. Mirrorbrain seems to be happy again
I'll look further in one week
Olivier Vernin It seems to me that the update center backs to normal. Thank you.
Hello, this is happening again.
I'm building a docker image and pulling plugins per the officially supported `plugin.txt` installation method in docker.
I run my image build locally and it fails randomly pulling various plugins (never the same plugins). Then I've also tried w/ Google Cloud Build and it fails for other various plugins. I can go to the updates.jenkins.io/latest site and see the plugins and download the ones that are failing, so it seems to be higher than normal load or something?
I first notice this ~ 20 minutes ago, and have had anywhere from 1 to 10 plugins fail, each a unique list of plugins failing in ~20 attempts.
Thanks, sorry to bring up a closed ticket.
Edit1: this has been working in the past, going on 2 years in docker using this plugin installation/update step in my Dockerfile
Edit2:
Random list of plugin failures example 1:
Some plugins failed to download! Not downloaded: ssh-agent Not downloaded: ownership Not downloaded: pipeline-github-lib Not downloaded: ant Not downloaded: extra-columns Not downloaded: multiple-scms Not downloaded: kpp-management-plugin Not downloaded: workflow-cps-global-lib Not downloaded: jquery-detached Not downloaded: ghprb
Random list of plugin failures example 2:
Some plugins failed to download! Not downloaded: icon-shim Not downloaded: shelve-project-plugin Not downloaded: github-branch-source Not downloaded: mailer Not downloaded: claim Not downloaded: ssh-credentials Not downloaded: ssh-agent Not downloaded: email-ext Not downloaded: simple-theme-plugin Not downloaded: lockable-resources Not downloaded: jquery
Random list of plugin failures example 3:
Some plugins failed to download! Not downloaded: node-iterator-api
Not downloaded: chromedriverStep
Not downloaded: pagerdutyStep
Not downloaded: pipeline-rest-apiStep
Not downloaded: credentialsStep
Not downloaded: build-monitor-pluginStep
Not downloaded: build-failure-analyzerStep
The live build failure looks like so (one entry for each plugin that randomly fails to install):
Failed to download plugin: jquery or jquery-plugin
I just restarted apache service on that machine, it should now go back to normal
you rock Olivier Vernin, trying again
success. Thanks much Olivier Vernin
Hey Olivier Vernin,
I think it needs to be restarted again, as we can't install properly even clean Jenkins instance.
It needs a lot of time to load http://updates.jenkins-ci.org/update-center.json
Thanks,
Wojciech
Hello
Same problem here : all my docker builds are failing with "Connection timed out" today...
Hi Olivier Vernin, I am not able to download any plugins whilst building my Docker image.
curl: (28) Resolving timed out after 20528 milliseconds
{{ 20:11:22 Failure (28) Retrying in 1 seconds...}}
{{ curl: (28) Resolving timed out after 20528 milliseconds}}
{{ 20:11:22 Failure (28) Retrying in 1 seconds...}}
{{ curl: (28) Resolving timed out after 20530 milliseconds}}
{{ 20:11:22 Failure (28) Retrying in 1 seconds...}}
{{ curl: (28) Resolving timecdu rol:u t( 28a)f tReers o2l0v5i2n9g mtiilmleids eocuotn dasf}}
{{ ter 20527 milliseconds}}
{{ 20:11:22 Failure (28) Retrying in 1 seconds...}}
{{ 20:11:22 Failure (28) Retrying in 1 seconds...}}
{{ curl: (28) Resolving timed out after 20525 milliseconds}}
{{ curl: (28) Resolving timed out after 20527 milliseconds}}
{{ curl: (28) Resolving timed out after 20530 milliseconds}}
{{ curl: (28) Resolving timed out after 20528 milliseconds}}
{{ 20:11:22 Failure (28) Retrying in 1 seconds...}}
{{ 20:11:22 Failure (28) Retrying in 1 seconds...}}
{{ 20:11:22 Failure (28) Retrying in 1 seconds...}}
{{ 20:11:22 Failure (28) Retrying in 1 seconds...}}
Finishing with:
WAR bundled plugins:
Installed plugins:
*:
Some plugins failed to download! Not downloaded: authorize-project
Not downloaded: git
Not downloaded: credentials-binding
Not downloaded: antisamy-markup-formatter
Not downloaded: ant
Not downloaded: timestamper
Not downloaded: pam-auth
Not downloaded: mailer
Not downloaded: pipeline-github-lib
Not downloaded: cloudbees-folder
Not downloaded: subversion
Not downloaded: ws-cleanup
Not downloaded: build-timeout
Not downloaded: matrix-auth
Not downloaded: ssh-slaves
Not downloaded: ldap
Not downloaded: email-ext
Not downloaded: configuration-as-code
Not downloaded: pipeline-stage-view
Not downloaded: github-branch-source
Not downloaded: gradle
Not downloaded: workflow-aggregator
Is there a more permanent solution to restarting the server manually each time?
I made multiple configuration tweak on the machine hosting the update center
Since then the machine has been quite stable in term of response time as you can see on the diagram
today I faced similar issue during downloading plugins:
Can you take a look?
Caused: java.io.IOException: Failed to load https://updates.jenkins.io/download/plugins/workflow-step-api/2.23/workflow-step-api.hpi to /var/lib/jenkins/plugins/workflow-step-api.jpi.tmp
{{ at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1287)}}
Caused: java.io.IOException: Failed to download from https://updates.jenkins.io/download/plugins/workflow-step-api/2.23/workflow-step-api.hpi (redirected to: https://get.jenkins.io/plugins/workflow-step-api/2.23/workflow-step-api.hpi)
{{ at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1321)}}
{{ at hudson.model.UpdateCenter$DownloadJob._run(UpdateCenter.java:1869)}}
{{ at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:2147)}}
{{ at hudson.model.UpdateCenter$DownloadJob.run(UpdateCenter.java:1843)}}
{{ at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)}}
{{ at java.util.concurrent.FutureTask.run(FutureTask.java:266)}}
{{ at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:118)}}
{{ at java.lang.Thread.run(Thread.java:748)}}
it seems all are facing the same issue .....Do any one know what is the issue ? I hope, Things will be fixed soon
2020-11-13T12:58:48.447446915Z applying Jenkins configuration 2020-11-13T12:58:48.447483236Z disable Setup Wizard 2020-11-13T12:58:48.471116665Z remove all plugins from shared volume 2020-11-13T12:58:48.472091072Z download plugins 2020-11-13T12:58:48.498435340Z WARN: install-plugins.sh is deprecated, please switch to jenkins-plugin-cli 2020-11-13T12:58:48.503519327Z Creating initial locks... 2020-11-13T12:58:48.565091458Z Analyzing war /usr/share/jenkins/jenkins.war... 2020-11-13T12:58:48.683618486Z Registering preinstalled plugins... 2020-11-13T12:59:54.373155697Z Using version-specific update center: https://updates.jenkins.io/dynamic-2.248/... 2020-11-13T12:59:54.373186817Z Downloading plugins... 2020-11-13T12:59:54.376165938Z Downloading plugin: ansicolor from https://updates.jenkins.io/download/plugins/ansicolor/0.7.3/ansicolor.hpi
a few plugin downloads seem to succeed here, but then...
2020-11-13T12:59:54.491017865Z Downloading plugin: workflow-aggregator from https://updates.jenkins.io/download/plugins/workflow-aggregator/2.6/workflow-aggregator.hpi 2020-11-13T12:59:55.385793396Z curl: (22) The requested URL returned error: 404 2020-11-13T12:59:55.389307941Z 12:59:55 Failure (22) Retrying in 1 seconds... 2020-11-13T13:01:04.898937987Z curl: (22) The requested URL returned error: 504 2020-11-13T13:01:04.899447311Z curl: (22) The requested URL returned error: 504 2020-11-13T13:01:04.903211958Z 13:01:04 Failure (22) Retrying in 1 seconds... 2020-11-13T13:01:04.903448030Z 13:01:04 Failure (22) Retrying in 1 seconds... 2020-11-13T13:01:04.917791062Z curl: (22) The requested URL returned error: 504 2020-11-13T13:01:04.920893034Z 13:01:04 Failure (22) Retrying in 1 seconds... 2020-11-13T13:01:04.923511774Z curl: (22) The requested URL returned error: 504 2020-11-13T13:01:04.926831128Z 13:01:04 Failure (22) Retrying in 1 seconds... 2020-11-13T13:01:04.942433999Z curl: (22) The requested URL returned error: 504 2020-11-13T13:01:04.945480121Z 13:01:04 Failure (22) Retrying in 1 seconds... 2020-11-13T13:01:42.084727197Z curl: (22) The requested URL returned error: 502 2020-11-13T13:01:42.088248487Z 13:01:42 Failure (22) Retrying in 1 seconds...
combination of 404, 504 and 502 until retry attempts are exhausted and then jenkins goes on to install with no plugins
+1 - Any update on when this can be addressed. All plugin downloads/updates are failing. Please advise.
Caused: java.io.IOException: Failed to download from https://updates.jenkins.io/download/plugins/config-file-provider/3.7.0/config-file-provider.hpi (redirected to: https://get.jenkins.io/plugins/config-file-provider/3.7.0/config-file-provider.hpi)
at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1321)
at hudson.model.UpdateCenter$DownloadJob._run(UpdateCenter.java:1869)
at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:2147)
at hudson.model.UpdateCenter$DownloadJob.run(UpdateCenter.java:1843)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:118)
at java.lang.Thread.run(Thread.java:748)
2020-11-13 18:31:34.034+0000 [id=204] INFO h.model.UpdateCenter$DownloadJob#run: Starting the installation of branch-api on behalf of matrixpsuser
2020-11-13 18:31:35.827+0000 [id=204] INFO h.m.UpdateCenter$UpdateCenterConfiguration#download: Downloading branch-api
2020-11-13 18:31:35.828+0000 [id=204] SEVERE h.model.UpdateCenter$DownloadJob#run: Failed to install branch-api
java.io.IOException: Server returned HTTP response code: 503 for URL: https://get.jenkins.io/plugins/branch-api/2.6.3/branch-api.hpi
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1900)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
at sun.net.www.protocol.http.HttpURLConnection.getHeaderField(HttpURLConnection.java:3061)
at java.net.URLConnection.getHeaderFieldLong(URLConnection.java:629)
at java.net.URLConnection.getContentLengthLong(URLConnection.java:501)
at java.net.URLConnection.getContentLength(URLConnection.java:485)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getContentLength(HttpsURLConnectionImpl.java:412)
at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1264)
Resolved Nov 14, 2020 by switching get.jenkins.io to a different server that uses different storage
I'm still receiving timeouts when trying to pull plugins.
> bootstrap4-api depends on font-awesome-api:5.15.1-1,jquery3-api:3.5.1-2,popper-api:1.16.0-7
Downloading plugin: font-awesome-api from https://updates.jenkins.io/dynamic-2.248//latest/font-awesome-api.hpi
Downloading plugin: jquery3-api from https://updates.jenkins.io/dynamic-2.248//latest/jquery3-api.hpi
Downloading plugin: popper-api from https://updates.jenkins.io/dynamic-2.248//latest/popper-api.hpi
> checks-api depends on plugin-util-api:1.4.0,workflow-cps:2.83,workflow-job:2.40,workflow-step-api:2.23
curl: (22) The requested URL returned error: 404 Not Found
16:07:37 Failure (22) Retrying in 1 seconds...
curl: (22) The requested URL returned error: 404 Not Found
curl: (22) The requested URL returned error: 404 Not Found
16:07:37 Failure (22) Retrying in 1 seconds...
16:07:37 Failure (22) Retrying in 1 seconds...
ccuurrll:: ((2222)) The rTehqeu ersted Uequested URL returned error: 404 Not Found
RL returned error: 404 Not Found
16:07:37 Failure (22) Retrying in 1 seconds...
16:07:37 Failure (22) Retrying in 1 seconds...
curl: (28) Connection timed out after 20000 milliseconds
16:07:40 Failure (28) Retrying in 1 seconds...
curl: (28) Connection timed out after 20000 milliseconds
16:07:41 Failure (28) Retrying in 1 seconds...
curl: (28) Connection timed out after 20001 milliseconds
16:07:41 Failure (28) Retrying in 1 seconds...
Some plugins failed to download! Not downloaded: workflow-api
Not downloaded: pipeline-stage-view
Not downloaded: structs
Not downloaded: cloudbees-folder
Not downloaded: script-security
Not downloaded: authentication-tokens
Not downloaded: apache-httpcomponents-client-4-api
Not downloaded: kubernetes-credentials
Not downloaded: github-branch-source
Not downloaded: workflow-cps
Not downloaded: workflow-support
Not downloaded: workflow-durable-task-step
Not downloaded: pipeline-input-step
Not downloaded: workflow-cps-global-lib
Not downloaded: workflow-basic-steps
Not downloaded: credentials
Not downloaded: pipeline-build-step
Not downloaded: workflow-job
Not downloaded: scm-api
Not downloaded: workflow-scm-step
Not downloaded: pipeline-milestone-step
Not downloaded: pipeline-model-definition
Not downloaded: pipeline-stage-step
Stephen Gordon Can you confirm that you can reach one of the mirrors from your location
-> https://get.jenkins.io/plugins/plugin-util-api/1.4.0/plugin-util-api.hpi?mirrorlist
curl -I -L https://get.jenkins.io/plugins/plugin-util-api/1.4.0/plugin-util-api.hpi should return 200 OK
I just took one of the plugins you are trying to download to see if it's available
Olivier Vernin yes this worked for me.
As far as I can tell, this is resolved
updates.jenkins.io/download/plugins is unreliably slow and failing
Originally reported by akram, imported from: updates.jenkins.io/download/plugins is unreliably slow and failing