search

jenkinsci / hashicorp-vault-plugin

Jenkins plugin to populate environment variables from secrets stored in HashiCorp's Vault.
https://plugins.jenkins.io/hashicorp-vault-plugin/
MIT License
216 stars 143 forks source link

Upgrading to the lastest plugin (364.vf5d54b_3dc313) breaks any Org or pipeline that uses with Vault. #326

Open ScottWatsonWork opened 5 months ago

ScottWatsonWork commented 5 months ago

Jenkins and plugins versions report

Environment ```text Jenkins: 2.401.1 OS: Linux - 5.15.0-1051-azure Java: 11.0.19 - Eclipse Adoptium (OpenJDK 64-Bit Server VM) --- ace-editor:1.1 ant:487.vd79d090d4ea_e antisamy-markup-formatter:159.v25b_c67cd35fb_ apache-httpcomponents-client-4-api:4.5.14-150.v7a_b_9d17134a_5 authentication-tokens:1.53.v1c90fd9191a_b_ authorize-project:1.7.0 azure-commons:1.1.3 azure-credentials:254.v64da_8176c83a azure-keyvault:200.v115e9b_1644d5 azure-sdk:132.v62b_48eb_6f32f blueocean:1.27.4 blueocean-autofavorite:1.2.5 blueocean-bitbucket-pipeline:1.27.4 blueocean-commons:1.27.4 blueocean-config:1.27.4 blueocean-core-js:1.27.4 blueocean-dashboard:1.27.4 blueocean-display-url:2.4.2 blueocean-events:1.27.4 blueocean-executor-info:1.27.4 blueocean-git-pipeline:1.27.4 blueocean-github-pipeline:1.27.4 blueocean-i18n:1.27.4 blueocean-jira:1.27.4 blueocean-jwt:1.27.4 blueocean-personalization:1.27.4 blueocean-pipeline-api-impl:1.27.4 blueocean-pipeline-editor:1.27.4 blueocean-pipeline-scm-api:1.27.4 blueocean-rest:1.27.4 blueocean-rest-impl:1.27.4 blueocean-web:1.27.4 bootstrap4-api:4.6.0-6 bootstrap5-api:5.3.0-1 bouncycastle-api:2.28 branch-api:2.1109.vdf225489a_16d build-history-manager:1.7.0 build-history-metrics-plugin:112.v476124de7dfc build-timeout:1.31 caffeine-api:3.1.6-115.vb_8b_b_328e59d8 checks-api:2.0.0 cloudbees-bitbucket-branch-source:809.vc1d904b_30426 cloudbees-folder:6.815.v0dd5a_cb_40e0e command-launcher:100.v2f6722292ee8 commons-lang3-api:3.12.0-36.vd97de6465d5b_ commons-text-api:1.10.0-36.vc008c8fcda_7b_ credentials:1254.vb_96f366e7b_a_d credentials-binding:604.vb_64480b_c56ca_ data-tables-api:1.13.4-3 display-url-api:2.3.7 docker-commons:419.v8e3cd84ef49c docker-workflow:563.vd5d2e5c4007f durable-task:507.v050055d0cb_dd echarts-api:5.4.0-5 email-ext:2.99 favorite:2.4.2 font-awesome-api:6.4.0-1 git:5.1.0 git-client:4.4.0 git-server:99.va_0826a_b_cdfa_d github:1.37.1 github-api:1.314-431.v78d72a_3fe4c3 github-branch-source:1728.v859147241f49 github-oauth:0.39 google-container-registry-auth:0.3 google-oauth-plugin:1.0.9 gradle:2.8 handlebars:3.0.8 handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953 hashicorp-vault-pipeline:1.4 hashicorp-vault-plugin:364.vf5d54b_3dc313 htmlpublisher:1.31 instance-identity:173.va_37c494ec4e5 ionicons-api:56.v1b_1c8c49374e jackson2-api:2.15.2-350.v0c2f3f8fc595 jakarta-activation-api:2.0.1-3 jakarta-mail-api:2.0.1-3 javax-activation-api:1.2.0-6 javax-mail-api:1.6.2-9 jaxb:2.3.8-1 jdk-tool:66.vd8fa_64ee91b_d jenkins-design-language:1.27.4 jersey2-api:2.39.1-2 jira:3.10 jjwt-api:0.11.5-77.v646c772fddb_0 jquery-detached:1.2.1 jquery3-api:3.7.0-1 jsch:0.2.8-65.v052c39de79b_2 junit:1214.va_2f9db_3e6de0 kubernetes:3952.v88e3b_0cf300b_ kubernetes-cd:2.3.1 kubernetes-cli:1.12.0 kubernetes-client-api:6.4.1-215.v2ed17097a_8e9 kubernetes-credentials:0.10.0 kubernetes-credentials-provider:1.225.v14f9e6b_28f53 kubernetes-pipeline-devops-steps:1.6 ldap:682.v7b_544c9d1512 lockable-resources:1171.v7a_4699ec2e7e mailer:457.v3f72cb_e015e5 matrix-auth:3.1.8 matrix-project:789.v57a_725b_63c79 mercurial:1260.vdfb_723cdcc81 metrics:4.2.18-439.v86a_20b_a_8318b_ mina-sshd-api-common:2.10.0-69.v28e3e36d18eb_ mina-sshd-api-core:2.10.0-69.v28e3e36d18eb_ momentjs:1.1.1 oauth-credentials:0.645.ve666a_c332668 okhttp-api:4.11.0-145.vcb_8de402ef81 pam-auth:1.10 pipeline-build-step:496.v2449a_9a_221f2 pipeline-github-lib:42.v0739460cda_c4 pipeline-graph-analysis:202.va_d268e64deb_3 pipeline-groovy-lib:656.va_a_ceeb_6ffb_f7 pipeline-input-step:468.va_5db_051498a_4 pipeline-milestone-step:111.v449306f708b_7 pipeline-model-api:2.2141.v5402e818a_779 pipeline-model-definition:2.2141.v5402e818a_779 pipeline-model-extensions:2.2141.v5402e818a_779 pipeline-rest-api:2.33 pipeline-stage-step:305.ve96d0205c1c6 pipeline-stage-tags-metadata:2.2141.v5402e818a_779 pipeline-stage-view:2.33 plain-credentials:143.v1b_df8b_d3b_e48 plugin-util-api:3.3.0 popper-api:1.16.1-3 popper2-api:2.11.6-2 pubsub-light:1.17 resource-disposer:0.22 scm-api:676.v886669a_199a_a_ script-security:1251.vfe552ed55f8d simple-theme-plugin:160.vb_76454b_67900 slack:664.vc9a_90f8b_c24a_ snakeyaml-api:1.33-95.va_b_a_e3e47b_fa_4 sonar:2.15 sse-gateway:1.26 ssh-credentials:305.v8f4381501156 ssh-slaves:2.877.v365f5eb_a_b_eec sshd:3.303.vefc7119b_ec23 structs:324.va_f5d6774f3a_d timestamper:1.25 token-macro:359.vb_cde11682e0c trilead-api:2.84.v72119de229b_7 variant:59.vf075fe829ccb workflow-aggregator:596.v8c21c963d92d workflow-api:1215.v2b_ee3e1b_dd39 workflow-basic-steps:1017.vb_45b_302f0cea_ workflow-cps:3691.v28b_14c465a_b_b_ workflow-cps-global-lib:609.vd95673f149b_b workflow-durable-task-step:1247.v7f9dfea_b_4fd0 workflow-job:1308.v58d48a_763b_31 workflow-multibranch:756.v891d88f2cd46 workflow-scm-step:415.v434365564324 workflow-step-api:639.v6eca_cd8c04a_a_ workflow-support:839.v35e2736cfd5c ws-cleanup:0.45 ```

What Operating System are you using (both controller, and any agents involved in the problem)?

I am using a kubernetes cluster to host my jenkins 2.401.1-LTS. No need to run an agent as the controller is the one that is having a problem.

Reproduction steps

  1. Update Hashicorp vault plug in from 360.v0a_1c04cf807d to 364.vf5d54b_3dc313

image

Expected Results

Was expecting my jobs to work and be accessible after only updating the vault plugin.

Actual Results

Now if I click on the org or a repo with a pipeline that uses vault I get a stacktrace and cannot access that repo.

I uncovered this when I was trying to upgrade to 2.426.3. From trial and error I have narrowed down the problem to the upgrade to this plugin. So I am not sure if we have something strange in our Jenkinsfile or what but this plugin version is not happy with something.

Anything else?

I am not sure if this is expected or not but the section in my config.xml for my repo is pointing to version 3.60. I am wondering if that is why it is greyed out in the image I uploaded.

` </com.datapipe.jenkins.vault.credentials.VaultAppRoleCredential>

vault_bot_sre https://vault.tools.copr/ui/vault/secrets?namespace=ssc%2Fcxai%2Fsre Etc/UTC not Important ssc/cxai/sre not important roleID goes here approle ` ### Are you interested in contributing a fix? _No response_
clintonsteiner commented 3 months ago

This issue also effected me

clintonsteiner commented 3 months ago

Coming back to say downgrading fixed the issue

clintonsteiner commented 1 month ago

I'm no java dev, but think https://github.com/jenkinsci/hashicorp-vault-plugin/commit/f5d54b3dc313b540011887dc420e6cd0488fe24d is the cause

@bluesliverx Any thoughts?

JaminenB commented 1 month ago

I also ran into this issue where the folder is visible, but the jobs it contains are not visible. Jobs still execute on their cadence or hooks, which is great but leaves users unable to edit jobs.

Upgraded from 360.v0a_1c04cf807d to 363.va_f8c1627db_b_a, a 'compatible' plugin upgrade. Downgrading back to the previous version fixed the issue.

bordenit commented 2 weeks ago

It's been months and no fix. Does anyone have a workaround yet? We have several Jenkins instances, and it only happens on 1 instance. Would prefer not to maintain separate images or reconfigure all those jobs.

bluesliverx commented 1 week ago

Sorry for the long delay here. Not sure exactly what the issue is. Any chance we can get some logs from Jenkins itself or what the actual error is?

ScottWatsonWork commented 1 week ago

@bluesliverx This was the error I was seeing.

`THIS IS THE ERROR WHEN I CLICK ON THE OrganizationFolder

2024-02-26 20:05:50.473+0000 [id=92] WARNING h.i.i.InstallUncaughtExceptionHandler#handleException: Caught unhandled exception with ID 1794f616-c655-4f0c-9e9c-41eacde97d1d java.lang.NullPointerException at jenkins.branch.OrganizationFolderViewHolder.ensureViews(OrganizationFolderViewHolder.java:123) at jenkins.branch.OrganizationFolderViewHolder.getPrimaryView(OrganizationFolderViewHolder.java:111) at com.cloudbees.hudson.plugins.folder.AbstractFolder$1.primaryView(AbstractFolder.java:270) at hudson.model.ViewGroupMixIn.getPrimaryView(ViewGroupMixIn.java:172) at com.cloudbees.hudson.plugins.folder.AbstractFolder.getPrimaryView(AbstractFolder.java:741) at jenkins.branch.OrganizationFolder.getPrimaryView(OrganizationFolder.java:652) at com.cloudbees.hudson.plugins.folder.AbstractFolder.getStaplerFallback(AbstractFolder.java:787) at com.cloudbees.hudson.plugins.folder.AbstractFolder.getStaplerFallback(AbstractFolder.java:145) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:827) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900) at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:289) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:59) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:698) at org.kohsuke.stapler.Stapler.service(Stapler.java:248) at javax.servlet.http.HttpServlet.service(HttpServlet.java:590) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764) at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:157) at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:248) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:129) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:81) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:60) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:160) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:160) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:94) at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:54) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99) at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:110) at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:101) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:221) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:97) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:117) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:63) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:111) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:172) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:53) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:86) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:38) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:549) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1383) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1305) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) at org.eclipse.jetty.server.Server.handle(Server.java:563) at org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:416) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:385) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:272) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.lambda$new$0(AdaptiveExecutionStrategy.java:140) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:934) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1078) at java.base/java.lang.Thread.run(Thread.java:829)`

And from Jenkins console output I would see things like this

2024-02-26 20:01:01.090+0000 [id=34] WARNING h.util.RobustReflectionConverter#doUnmarshal: Cannot convert type com.cloudbees.hudson.plugins.folder.properties.FolderCredentialsProvider$FolderCredentialsProperty to type jenkins.branch.MultiBranchProject 2024-02-26 20:01:01.091+0000 [id=34] WARNING h.util.RobustReflectionConverter#doUnmarshal: Cannot convert type com.cloudbees.hudson.plugins.folder.properties.FolderCredentialsProvider$FolderCredentialsProperty to type com.cloudbees.hudson.plugins.folder.AbstractFolder 2024-02-26 20:01:01.092+0000 [id=34] WARNING h.util.RobustReflectionConverter#doUnmarshal: Cannot convert type com.cloudbees.hudson.plugins.folder.properties.FolderCredentialsProvider$FolderCredentialsProperty to type hudson.model.Saveable 2024-02-26 20:01:01.092+0000 [id=34] WARNING h.util.RobustReflectionConverter#doUnmarshal: Cannot convert type com.cloudbees.hudson.plugins.folder.properties.FolderCredentialsProvider$FolderCredentialsProperty to type jenkins.branch.MultiBranchProject 2024-02-26 20:01:07.613+0000 [id=34] WARNING h.util.RobustReflectionConverter#doUnmarshal: Cannot convert type hudson.util.DescribableList to type jenkins.branch.MultiBranchProject 2024-02-26 20:01:07.613+0000 [id=34] WARNING h.util.RobustReflectionConverter#doUnmarshal: Cannot convert type hudson.util.DescribableList to type com.cloudbees.hudson.plugins.folder.AbstractFolder 2024-02-26 20:01:07.615+0000 [id=34] WARNING h.util.RobustReflectionConverter#doUnmarshal: Cannot convert type hudson.util.DescribableList to type hudson.model.Saveable 2024-02-26 20:01:07.615+0000 [id=34] WARNING h.util.RobustReflectionConverter#doUnmarshal: Cannot convert type hudson.util.DescribableList to type jenkins.branch.MultiBranchProject 2024-02-26 20:01:08.365+0000 [id=34] WARNING h.util.RobustReflectionConverter#doUnmarshal: Cannot convert type hudson.util.DescribableList to type jenkins.branch.MultiBranchProject 2024-02-26 20:01:08.365+0000 [id=34] WARNING h.util.RobustReflectionConverter#doUnmarshal: Cannot convert type hudson.util.DescribableList to type com.cloudbees.hudson.plugins.folder.AbstractFolder 2024-02-26 20:01:08.366+0000 [id=34] WARNING h.util.RobustReflectionConverter#doUnmarshal: Cannot convert type hudson.util.DescribableList to type hudson.model.Saveable 2024-02-26 20:01:08.367+0000 [id=34] WARNING h.util.RobustReflectionConverter#doUnmarshal: Cannot convert type hudson.util.DescribableList to type jenkins.branch.MultiBranchProject

bluesliverx commented 1 week ago

I created a new org on our Jenkins and didn't see any issues, I wonder if this is a problem with an existing org after the upgrade.

Looking at the stack trace, it might have something to do with the owner being null, which sounds like the problem identified in #324 that was already fixed in release https://github.com/jenkinsci/hashicorp-vault-plugin/releases/tag/368.v48134f694db_f. Have you tried upgrading again to see if this is fixed?

ScottWatsonWork commented 1 week ago

Thanks @bluesliverx I didn't have the nerve to try the upgrade again as I had way too many people screaming at me to get it back up from the last time. However, other people have come here recently looking for a solution so maybe they can say which version of the plugin they tried without success. @JaminenB @bordenit

bordenit commented 1 week ago

@bluesliverx @ScottWatsonWork The last one we installed in production environment that was having this issue is 367.v8a_1ee1cccf3a. I will try the 368... you referenced. That 368... version is already in our engineering environment. We have 6 Jenkins instances and only one production instance has this issue. I will try the 368... version during our next outage window. Thanks.

dwnusbaum commented 16 hours ago

The serialization compatibility mistake in #223 (changing the type of a serialized field from Calendar to Map<String, Calendar>) that was reported in #323, #324, and again in this issue, seems to have been particularly problematic.

The fix for the issue was changing the relevant field to be transient in https://github.com/jenkinsci/hashicorp-vault-plugin/pull/325, which is in release 366.v3b_57135510d6. If you are having problems but are already running that version or a newer one, upgrading further will not help you.

Unfortunately, from some testing locally, it seems that in this particular case where the type was changed to a Map, marking the field transient does not fix the issue for users who were upgrading from older versions and already had a Calendar tokenExpiry serialized into their config.xml file: they still see the error despite the field now being transient (this seems unexpected, I am looking into this a bit more. EDIT: Turns out this is the expected behavior of XStream).

At this point I see no simple solution that does not require users to manually edit their folder's config.xml files to delete the offending tokenExpiry fields (that is the best fix I am aware of right now). Perhaps it could be fixed by implementing a custom XStream Converter for VaultAppRoleCredential and any other affected types that is able to handle tokenExpiry specially (by always ignoring it), but I do not know. Downgrading to a version prior to 364.vf5d54b_3dc313, which is the first one that picked up https://github.com/jenkinsci/hashicorp-vault-plugin/pull/223, should suppress the issue, but any upgrade would re-introduce it, so you'd be stuck forever, and in general downgrading is not compatible, so you would have to carefully check all changes before attempting such a downgrade.

Note that in general, Jenkins is supposed to handle deserialization errors and mistakes like this a bit more gracefully. The desired behavior is just that the broken value gets skipped, but the overall object otherwise loads fine. In this case, the folder property with the credential should have been removed, but the folder itself should have loaded fine, which would have meant that everyone would have just needed to reconfigure their vault credentials. I am looking a bit more to try to understand what exactly went wrong in this case to see if anything can be improved in Jenkins core.

dwnusbaum commented 16 hours ago

@jglick reminded me that probably renaming the transient Map<String, Calendar> tokenExpiry field to tokenExpiry2 or similar so that XStream stops looking at it should fix the issue.

dwnusbaum commented 16 hours ago

https://github.com/jenkinsci/hashicorp-vault-plugin/pull/336 should fix the issue.

dwnusbaum commented 14 hours ago

https://github.com/jenkinsci/jenkins/pull/9653 experiments with some core changes that would have limited the impact of the mistake to simply removing the Vault-related property from the folder rather than corrupting the folder itself.