search

geerlingguy / ansible-role-jenkins

Ansible Role - Jenkins CI
https://galaxy.ansible.com/geerlingguy/jenkins/
MIT License
831 stars 748 forks source link

Intermittent package install failure on Google Compute #92

Closed gregorskii closed 4 years ago

gregorskii commented 8 years ago

Hi there,

Seeing an intermittent issue on this role where installing packages fails due to this issue:

https://issues.jenkins-ci.org/browse/JENKINS-35197

It appears the issue is with Jenkins 2.0, and per the readme the package install may have issues with Jenkins 2, however I do have success installing at least half the time.

I do however have a large amount of plugins I am installing and it appears the more times it has to install a plugin the more likely it is to happen.

I have updated my role config as follows:

jenkins_admin_username: admin
jenkins_admin_password: ""
jenkins_admin_password_file: /var/lib/jenkins/secrets/initialAdminPassword

jenkins_init_changes:
  - option: "JAVA_ARGS"
    value: "-Dhudson.diyChunking=false"

jenkins_plugins:
  # Jenkins Recommended
  - subversion
  - cloudbees-folder
  - antisamy-markup-formatter
  - build-timeout
  - credentials-binding
  - timestamper
  - ws-cleanup
  - ant
  - gradle
  - workflow-scm-step
  - workflow-step-api
  - workflow-cps
  - workflow-support
  - workflow-basic-steps
  - pipeline-input-step
  - pipeline-milestone-step
  - pipeline-build-step
  - pipeline-stage-view
  - workflow-multibranch
  - workflow-durable-task-step
  - workflow-api
  - workflow-cps-global-lib
  - workflow-step-api
  - workflow-job
  - workflow-aggregator
  - github-organization-folder
  - ssh-slaves
  - matrix-auth
  - pam-auth
  - matrix-project
  - email-ext
  - mailer
  - credentials
  - git-client
  - external-monitor-job
  - mapdb-api
  - plain-credentials
  - script-security
  - structs
  - token-macro
  # Blue Ocean
  - metadata
  - favorite
  - blueocean-web
  - blueocean-rest-impl
  - blueocean-commons
  - blueocean-dashboard
  - blueocean-personalization
  - blueocean-jwt
  - blueocean-config
  - blueocean-rest
  - pipeline-model-definition
  - blueocean-pipeline-api-impl
  # Oraganization
  - junit
  - github-oauth
  - simple-theme-plugin
  - git-parameter
  - scm-sync-configuration
  - ssh-credentials
  - ansicolor
  - slack
  - github
  - ghprb
  - build-pipeline-plugin
  - conditional-buildstep
  - parameterized-trigger
  - ssh
  - extensible-choice-parameter
  - multiple-scms
  - view-job-filters
  - embeddable-build-status
  - preSCMbuildstep
  - postbuild-task
  - configurationslicing
  - docker-build-step
  - docker-custom-build-environment
  - jira
  - greenballs
  - nested-view
  - saferestart
  - job-dsl

The error output from Ansible is this:

==> googlecompute: starting sftp subsystem
    googlecompute: failed: [default] (item=subversion) => {"changed": true, "cmd": ["java", "-jar", "/opt/jenkins-cli.jar", "-s", "http://localhost:8080/", "install-plugin", "subversion", "--username", "admin", "--password-file", "/var/lib/jenkins/secrets/initialAdminPassword"], "delta": "0:00:09.593071", "end": "2016-10-09 20:55:19.003563", "failed": true, "item": "subversion", "rc": 255, "start": "2016-10-09 20:55:09.410492", "stderr": "Oct 09, 2016 8:55:18 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run\nSEVERE: I/O error in channel Chunked connection to http://localhost:8080/cli\njava.io.IOException: Unexpected termination of the channel\n\tat hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:73)\nCaused by: java.io.EOFException\n\tat java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2353)\n\tat java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2822)\n\tat java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:804)\n\tat java.io.ObjectInputStream.<init>(ObjectInputStream.java:301)\n\tat hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)\n\tat hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)\n\tat hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)\n\nhudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel\n\tat hudson.remoting.Request.abort(Request.java:303)\n\tat hudson.remoting.Channel.terminate(Channel.java:863)\n\tat hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:92)\n\tat ......remote call to Chunked connection to http://localhost:8080/cli(Native Method)\n\tat hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1433)\n\tat hudson.remoting.Request.call(Request.java:172)\n\tat hudson.remoting.Channel.call(Channel.java:796)\n\tat hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:252)\n\tat com.sun.proxy.$Proxy3.main(Unknown Source)\n\tat hudson.cli.CLI.execute(CLI.java:332)\n\tat hudson.cli.CLI._main(CLI.java:503)\n\tat hudson.cli.CLI.main(CLI.java:384)\nCaused by: java.io.IOException: Unexpected termination of the channel\n\tat hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:73)\nCaused by: java.io.EOFException\n\tat java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2353)\n\tat java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2822)\n\tat java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:804)\n\tat java.io.ObjectInputStream.<init>(ObjectInputStream.java:301)\n\tat hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)\n\tat hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)\n\tat hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)", "stdout": "Installing subversion from update center", "stdout_lines": ["Installing subversion from update center"], "warnings": []}

As a solution to this problem is it possible to run a check that the error output is not in the results of running the command on each iteration of the loop?

I tried doing so but the task just skipped when it was in place. I tried adding this:

- name: Install Jenkins plugins using password-file.
  command: >
    java -jar {{ jenkins_jar_location }} -s http://{{ jenkins_hostname }}:{{ jenkins_http_port }}{{ jenkins_url_prefix | default('') }}/
    install-plugin {{ item }}
    --username {{ jenkins_admin_username }}
    --password-file {{ jenkins_admin_password_file }}
    creates={{ jenkins_home }}/plugins/{{ item }}.jpi
  with_items: "{{ jenkins_plugins }}"
  when: adminpasswordfile.stat.exists == True
  register: plugin_install_result
  until: "'Unexpected termination of the channel' not in plugin_install_result"
  retries: 5
  delay: 10
  notify: restart jenkins

I am also finding a couple other smaller issues:

Intermittently the jenkins_admin_password_file appears to be missing on some runs, and then the "Install Jenkins plugins using password-file." task skips.

Sometimes the password task runs even though the jenkins_admin_password field is an empty string.

stale[bot] commented 4 years ago

This issue has been marked 'stale' due to lack of recent activity. If there is no further activity, the issue will be closed in another 30 days. Thank you for your contribution!

Please read this blog post to see the reasons why I mark issues as stale.

stale[bot] commented 4 years ago

This issue has been closed due to inactivity. If you feel this is in error, please reopen the issue or file a new issue with the relevant details.