Closed timblaktu closed 3 years ago
timblaktu
commented
3 years ago Going back to version 2.222.3-1 of this role results in similar issues. There are a lot of "failed to load plugin" errors at startup. Really confused about what changed. Let me know if anyone has any ideas about how to troubleshoot jenkins startup and plugin dependency issues using this role. Thanks.
timblaktu
commented
3 years ago Related, I recently noticed the Jenkins team completed a new plugin installation/management tool. I may use this tomorrow to try to figure out what's going on.
@tobias-richter I wonder if this tool has a useful purpose integrated into this or your jenkins_plugins role?
timblaktu
commented
3 years ago I discovered that when I upgraded my ansible-jenkins-pipeline-library role, ansible-galaxy did not update the ansible-jenkins-plugins role to the new version. This must be at least part of the cause of the problem.
After removing and reinstalling the role, and confirming all roles are at the correct versions, I'm now getting some different behavior. Some of the plugins have installed successfully:
changed: [jenkins-testing] => (item={u'key': u'git', u'value': {u'state': u'present'}})
changed: [jenkins-testing] => (item={u'key': u'monitoring', u'value': {u'state': u'present'}})
changed: [jenkins-testing] => (item={u'key': u'versioncolumn', u'value': {u'state': u'present'}})
changed: [jenkins-testing] => (item={u'key': u'build-monitor-plugin', u'value': {u'state': u'present'}})
changed: [jenkins-testing] => (item={u'key': u'workflow-job', u'value': {u'state': u'present'}})
changed: [jenkins-testing] => (item={u'key': u'workflow-multibranch', u'value': {u'state': u'present'}})
changed: [jenkins-testing] => (item={u'key': u'copyartifact', u'value': {u'state': u'present'}})but most of them are encountering this failure mode:
failed: [jenkins-testing] (item={u'key': u'build-timeout', u'value': {u'state': u'present'}}) => changed=true
ansible_loop_var: item
details: 'Connection failure: timed out'
item:
key: build-timeout
value:
state: present
msg: Cannot install plugin...after re-running the playbook a few times to workaround my need for a local mirror, all plugins seem to have been installed now, but very few of them actually load, and this task is now my first failure:
TASK [wcm_io_devops.jenkins_plugins : assert that there are no failed plugins.]
with msg:
"33 plugin(s) failed to load!"
The full list of failed plugins with theirs reasons, as printed from gather_jenkins_facts - debug, is shown below. I believe this indicates the same errors as I reported in the UPDATE to the original report above, but now these errors are caught by the now-updated jenkins_facts and jenkins_plugins roles.
I will now start tracing through your code to see how plugin versions are specified and "locked", to see why it's not working in my environment.
- jenkins_facts_plugins_failed
- atlassian-bitbucket-server-integration: cause: |- Failed to load: Bitbucket Server Integration (2.1.1) - Failed to load: Pipeline (2.6) blueocean: cause: |- Failed to load: Blue Ocean (1.24.3) - Failed to load: Bitbucket Pipeline for Blue Ocean (1.24.3) blueocean-autofavorite: cause: |- Failed to load: Autofavorite for Blue Ocean (1.2.4) - Failed to load: Favorite (2.3.2) blueocean-bitbucket-pipeline: cause: |- Failed to load: Bitbucket Pipeline for Blue Ocean (1.24.3) - Failed to load: Pipeline implementation for Blue Ocean (1.24.3) blueocean-config: cause: |- Failed to load: Config API for Blue Ocean (1.24.3) - Plugin is missing: blueocean-jwt (1.24.3) blueocean-events: cause: |-
Failed to load: Events API for Blue Ocean (1.24.3)
- Failed to load: Pipeline implementation for Blue Ocean (1.24.3) blueocean-git-pipeline:
cause: |-
Failed to load: Git Pipeline for Blue Ocean (1.24.3)
- Failed to load: Pipeline implementation for Blue Ocean (1.24.3)
blueocean-jira:
cause: |- Failed to load: JIRA Integration for Blue Ocean (1.24.3) - Failed to load: Jenkins Jira plugin (3.1.3) blueocean-personalization: cause: |- Failed to load: Personalization for Blue Ocean (1.24.3) - Plugin is missing: blueocean-dashboard (1.24.3) blueocean-pipeline-api-impl: cause: |- Failed to load: Pipeline implementation for Blue Ocean (1.24.3) - Failed to load: REST Implementation for Blue Ocean (1.24.3) blueocean-pipeline-editor: cause: |- Failed to load: Blue Ocean Pipeline Editor (1.24.3) - Plugin is missing: blueocean-dashboard (1.24.3) - Failed to load: Pipeline implementation for Blue Ocean (1.24.3)
blueocean-rest-impl:
cause: |-
Failed to load: REST Implementation for Blue Ocean (1.24.3)
- Plugin is missing: blueocean-jwt (1.24.3)
- Failed to load: JUnit Plugin (1.47)
bootstrap4-api:
cause: |-
Failed to load: Bootstrap 4 API Plugin (4.5.3-1)
- Plugin is missing: font-awesome-api (5.15.1-1)
cloudbees-bitbucket-branch-source: [84/91514] cause: |-
Failed to load: Bitbucket Branch Source Plugin (2.9.6)
- Plugin is missing: mercurial (2.8)
conditional-buildstep: cause: |- Failed to load: Conditional BuildStep (1.4.1) - Failed to load: Maven Integration plugin (3.8) copyartifact: cause: |- Failed to load: Copy Artifact Plugin (1.45.3) - Failed to load: Matrix Project Plugin (1.18) docker-workflow: cause: |- Failed to load: Docker Pipeline (1.25) - Failed to load: Pipeline: Basic Steps (2.23) favorite: cause: |- Failed to load: Favorite (2.3.2) - Failed to load: Matrix Project Plugin (1.18) git-parameter: cause: |- Failed to load: Git Parameter Plug-In (0.9.13) - Failed to load: JUnit Plugin (1.47) github: cause: |-
Failed to load: GitHub plugin (1.32.0)
- Plugin is missing: github-api (1.114.2)
github-branch-source:
cause: |-
Failed to load: GitHub Branch Source Plugin (2.9.2)
- Failed to load: GitHub plugin (1.32.0)
htmlpublisher:
cause: |-
Failed to load: HTML Publisher plugin (1.25)
- Failed to load: Matrix Project Plugin (1.18)
jira:
cause: |-
Failed to load: Jenkins Jira plugin (3.1.3)
- Failed to load: Matrix Project Plugin (1.18)
junit:
cause: |-
Failed to load: JUnit Plugin (1.47)
- Failed to load: Bootstrap 4 API Plugin (4.5.3-1)
lockable-resources:
cause: |-
Failed to load: Lockable Resources plugin (2.10)
- Failed to load: Matrix Project Plugin (1.18)
matrix-project:
cause: |-
Failed to load: Matrix Project Plugin (1.18)
- Failed to load: JUnit Plugin (1.47)
maven-plugin:
cause: |-
Failed to load: Maven Integration plugin (3.8)
- Failed to load: JUnit Plugin (1.47)
pipeline-model-definition:
cause: |-
Failed to load: Pipeline: Declarative (1.7.2)
- Failed to load: Pipeline: Basic Steps (2.23)
prometheus:
cause: |-
Failed to load: Prometheus metrics plugin (2.0.8)
- Failed to load: JUnit Plugin (1.47)
workflow-aggregator:
cause: |-
Failed to load: Pipeline (2.6)
- Failed to load: Lockable Resources plugin (2.10)
workflow-basic-steps:
cause: |-
Failed to load: Pipeline: Basic Steps (2.23)
- Jenkins (2.263) or higher required
ws-cleanup:
cause: |-
Failed to load: Jenkins Workspace Cleanup Plugin (0.38)
- Failed to load: Matrix Project Plugin (1.18)
xunit:
cause: |-
Failed to load: xUnit plugin (2.4.0)
- Failed to load: JUnit Plugin (1.47)OK, looking at the docs on the role variables for the jenkins_plugins role, I immediately see that each items in the jenkins_plugins_present list supports a version: key.
I suppose I misinterpreted the list of plugin versions updated on your Releases page. I ASSumed that this meant the meta role was handling the specification of all versions for me.
...and that assumption was correct:
Looking at where the meta/main.yml for this role specifies the jenkinsplugins* role variables, and then over at defaults.yml, where these variables are defined, I see that you ARE indeed specifying all these versions.
What I got wrong was that I didn't realize that _I was overriding your plugin version specifications when I define my own jenkins_pipeline_library_plugins_present list where I call your meta role_. And because my plugins list did not specify versions, I was probably causing those plugins to default to "latest".
So now, I'll go and fix this, to make sure my invocation of your meta role specifies something like:
my_jenkins_plugins_present:
- name: active-directory
- name: audit-trail
.
.
.
jenkins_pipeline_library_plugins_present: "{{ jenkins_pipeline_library_plugins_present + my_jenkins_plugins_present }}"What I'm unsure of is how to ensure nothing in my_jenkins_plugins_present overlaps with jenkins_pipeline_library_plugins_present. Will report back when I get this fixed.
UPDATE: can't seem to get around the recursive loop detected in template string error whenever I try to define jenkins_pipeline_library_plugins_present as the combination of itself and my_jenkins_plugins_present list. I've tried + operator, and jinja2 combine filter so far, and setting it as a fact in previous task, no dice so far..
timblaktu
commented
3 years ago It appears Ansible does not support self referencing variable definitions like this, so I had no recourse but to explicitly define all the overlapping jenkins plugins in my invocation of your meta role, by copying them from your defaults. Not ideal (bc I'll have to do this manual work every time I pull in a new release) but hey I've already spent at least an hour googling how to do this to no end..
So, in the end I ended up defining the superset of jenkins plugins in my list, including your default versions, and that got me further.
Now I'm down to 3 load errors:
- atlassian-bitbucket-server-integration:
- cause: |-
Failed to load: Bitbucket Server Integration (2.1.1)
- Failed to load: Pipeline (2.6)
- pipeline-stage-view:
- cause: |-
Failed to load: Pipeline: Stage View Plugin (2.19)
- Update required: Pipeline: REST API Plugin (2.18) to be updated to 2.19 or higher
- workflow-aggregator:
- cause: |-
Failed to load: Pipeline (2.6)
- Failed to load: Pipeline: Stage View Plugin (2.19)It appears that root cause for this failure chain is that pipeline-stage-view got installed at 2.19, whereas your defaults/main.yml -specifies it at 2.18. Perhaps this is some remnant from past run of this playbook in which I was inadvertently telling it to "install latest". I will now manually uninstall/reinstall pipeline-stage-view at 2.18, and try again.
timblaktu
commented
3 years ago OK, I think I'm through the last of these problems after reinstalling pipeline-stage-view at 2.18. The moral of that story is "take care with ansible-galaxy". I should probably write a task for all my playbooks to inspect the contents of the roles dirs to make sure they're aligned with requirements.yml..
tobias-richter
commented
3 years ago @timblaktu sorry for the late response. You are right. Ansible Galaxy does not update dependencies automatically when running install. I suggest to always use the -f / --force and try to reproduce issues on a vanilla system, for example using a technology like vagrant.
After working through the "A later version is already installed" problem and the Java version Check bug in Jenkins Debian Package I'm now running jenkins_pipeline_library version 2.249.2-1, but Jenkins is crashing at first startup, with what looks like a lot of plugin load errors in /var/log/jenkins.log (see below for excerpt).
The only clue that I can see in this so far is that it appears that several of these failed plugins report a version newer than what your release page indicates should be installed for that version. For example, the log indicates
Failed to load: Branch API Plugin (2.6.3), however your release page indicates 2.6.0 should be installed by this release. In this example, the Branch API Plugin 2.6.3 fails to load indicating that "Jenkins (2.264) or higher required".So it appears that something in the plugin version locking is not working properly in this role.
Let me know what I can do to help troubleshoot this.
Here is the play that is invoking this role:
and here is my
requirements.yml, which I install roles and collections from ansible-galaxy for this project:This configuration was working fine using version 2.222.3 of your role. I upgraded over the course of troubleshooting the "A later version is already installed" problem and the Java version Check bug in Jenkins Debian Package.
Perhaps I will revert the cluster of machines I run this on to their post-OS-install snapshots and repeat provisioning with the old version as a sanity check. Let me know if you have any ideas.. Thanks!