credentials.xml is kept from boot to boot even if the encryption keys seems to change, thus secrets inside becomes useless

guilcy commented 1 week ago

Jenkins and plugins versions report

Jenkins with casc on kubernetes, credentials strings are empty when requested by jenkins scripts

I am using jenkins inside kubernetes, I use casc to load the controler configuration. At first boot, everything is fine. At second boot, all credentials are lost. I noticed that the config.xml is regenerated at each boot, and that the hash of the secrets inside are different each time. Concerning the credentials.xml file, it is only generate one time at first boot. On second boot any credentials are empty when requested from the script console inside jenkins. It is strange but it seems jenkins is regenerating the encryption keys at each boot. Thus, any configuration that is not newly generated contains hash that cannot be decoded by jenkins again. A I right ? In addition I had a look to appy_config.sh (from init container), it is erasing all xml config file but not the credentials.xml... is it expected, shouldn't it be erased so that casc get a chance to create it newly (with its new encryption key) ? Can you confirm the analyse is right: jenkins encryption key are generated again at each kubernetes deployment (each time I delete the pod) - I am not kubernetes specialist, sorry ? Why is credentials.xml not removed by apply_config.sh of the init container ? Or maybe I miss something in the configuration. But all our JCASC settings were working fine inside a VM. Not sure why casc is not overring credentials.xml at each boot. But on VM it is performing a kind of merge if I remember well, I guess that because the keys are already defined it is not generating again with is weird, but that's another problem probably. ```Jenkins: 2.440.1 OS: Linux - 5.15.0-1073-azure --- ace-editor:1.1 allure-jenkins-plugin:2.31.1 analysis-collector:2.0.0 analysis-core:1.96 analysis-model-api:12.1.0 ant:497.v94e7d9fffa_b_9 antisamy-markup-formatter:162.v0e6ec0fcfcf6 apache-httpcomponents-client-4-api:4.5.14-208.v438351942757 apache-httpcomponents-client-5-api:5.3.1-1.0 artifactdeployer:1.3 asm-api:9.7-33.v4d23ef79fcc8 atlassian-bitbucket-server-integration:4.0.0 authentication-tokens:1.53.v1c90fd9191a_b_ azure-ad:507.vea_a_a_167b_d05c azure-cli:0.9 azure-commons:1.1.3 azure-container-agents:253.vd2f5cd5c5040 azure-credentials:312.v0f3973cd1e59 azure-credentials-ext:1.0 azure-keyvault:251.vcfe31c013dc7 azure-sdk:174.va_89c1df897d2 badge:1.9.1 basic-branch-build-strategies:81.v05e333931c7d batch-task:1.398.v550a_8e8ca_e5f behave-testresults-publisher:0.0.11 bitbucket:241.v6d24a_57f9359 blueocean:1.27.11 blueocean-autofavorite:1.2.5 blueocean-bitbucket-pipeline:1.27.11 blueocean-commons:1.27.11 blueocean-config:1.27.11 blueocean-core-js:1.27.11 blueocean-dashboard:1.27.11 blueocean-display-url:2.4.2 blueocean-events:1.27.11 blueocean-git-pipeline:1.27.11 blueocean-github-pipeline:1.27.11 blueocean-i18n:1.27.11 blueocean-jira:1.27.11 blueocean-jwt:1.27.11 blueocean-personalization:1.27.11 blueocean-pipeline-api-impl:1.27.11 blueocean-pipeline-editor:1.27.11 blueocean-pipeline-scm-api:1.27.11 blueocean-rest:1.27.11 blueocean-rest-impl:1.27.11 blueocean-web:1.27.11 bootstrap5-api:5.3.3-1 bouncycastle-api:2.30.1.77-225.v26ea_c9455fd9 branch-api:2.1152.v6f101e97dd77 build-failure-analyzer:2.5.0 build-metrics:1.3 build-monitor-plugin:1.14-860.vd06ef2568b_3f build-timeout:1.32 built-on-column:1.4 caffeine-api:3.1.8-133.v17b_1ff2e0599 categorized-view:1.13 cccc:0.6 checks-api:2.0.2 cloud-stats:336.v788e4055508b_ cloudbees-bitbucket-branch-source:883.v041fa_695e9c2 cloudbees-folder:6.901.vb_4c7a_da_75da_3 cmakebuilder:4.1.1 cobertura:1.17 code-coverage-api:4.99.0 codesonar:3.5.0 command-launcher:107.v773860566e2e commons-compress-api:1.26.1-2 commons-httpclient3-api:3.1-3 commons-lang3-api:3.13.0-62.v7d18e55f51e2 commons-text-api:1.11.0-95.v22a_d30ee5d36 conditional-buildstep:1.4.3 config-file-provider:968.ve1ca_eb_913f8c configuration-as-code:1775.v810dc950b_514 configurationslicing:548.ve92d48e66b_f8 console-tail:15.vc54cb_4ca_6981 copy-project-link:106.veb_028794a_844 copyartifact:722.v0662a_9b_e22a_c coverage:1.13.0 cppcheck:1.26 credentials:1337.v60b_d7b_c7b_c9f credentials-binding:657.v2b_19db_7d6e6d csp:1.2 cucumber-perf:2.0.9 cucumber-reports:5.8.1 cucumber-testresult-plugin:0.10.1 custom-tools-plugin:0.8 cvs:2.19.1 dashboard-view:2.508.va_74654f026d1 data-tables-api:2.0.3-1 database:230.v701f20b_8b_f95 database-mysql:63.va_0596d2b_1438 disk-usage:1.2 display-url-api:2.200.vb_9327d658781 docker-commons:439.va_3cb_0a_6a_fb_29 docker-java-api:3.3.4-86.v39b_a_5ede342c docker-plugin:1.6 docker-workflow:572.v950f58993843 doxygen:178.v6ea_ef5f7dfdb drmemory-plugin:1.12 dtkit-api:3.0.2 durable-task:555.v6802fe0f0b_82 echarts-api:5.5.0-1 eddsa-api:0.3.0-4.v84c6f0f4969e email-ext:2.105 envinject:2.908.v66a_774b_31d93 envinject-api:1.199.v3ce31253ed13 extended-choice-parameter:381.v360a_25ea_017c extended-read-permission:53.v6499940139e5 extensible-choice-parameter:1.8.1 external-monitor-job:215.v2e88e894db_f8 favorite:2.208.v91d65b_7792a_c file-operations:214.v2e7dc7f25757 font-awesome-api:6.5.1-3 forensics-api:2.4.0 git:5.2.1 git-client:4.7.0 git-server:114.v068a_c7cc2574 github:1.38.0 github-api:1.318-461.v7a_c09c9fa_d63 github-branch-source:1785.v99802b_69816c gitlab-api:5.3.0-91.v1f9a_fda_d654f gitlab-branch-source:703.vc5fd7effcc6b_ global-build-stats:293.vd7b_d6e361475 groovy-label-assignment:1.2.0 groovy-postbuild:228.vcdb_cf7265066 gson-api:2.10.1-15.v0d99f670e0a_7 h2-api:11.1.4.199-12.v9f4244395f7a_ handlebars:3.0.8 handy-uri-templates-2-api:2.1.8-30.v7e777411b_148 htmlpublisher:1.33 http_request:1.18 icon-shim:3.0.0 influxdb:3.6.1 instance-identity:185.v303dc7c645f9 ionicons-api:70.v2959a_b_74e3cf ivy:2.5 jackson2-api:2.17.0-379.v02de8ec9f64c jakarta-activation-api:2.1.3-1 jakarta-mail-api:2.1.3-1 javadoc:243.vb_b_503b_b_45537 javax-activation-api:1.2.0-6 javax-mail-api:1.6.2-9 jaxb:2.3.9-1 jdk-tool:73.vddf737284550 jenkins-design-language:1.27.11 jenkins-multijob-plugin:627.v7c23cef20a_6a jersey2-api:2.42-147.va_28a_44603b_d5 jira:3.13 jjwt-api:0.11.5-112.ve82dfb_224b_a_d jnr-posix-api:3.1.19-2 job-dsl:1.87 job-import-plugin:3.6 job-restrictions:0.8 jobConfigHistory:1229.v3039470161a_d joda-time-api:2.12.7-29.v5a_b_e3a_82269a_ jquery:1.12.4-1 jquery-detached:1.2.1 jquery3-api:3.7.1-2 jsch:0.2.16-86.v42e010d9484b_ json-api:20240303-41.v94e11e6de726 json-path-api:2.9.0-58.v62e3e85b_a_655 junit:1265.v65b_14fa_f12f0 kubernetes:4203.v1dd44f5b_1cf9 kubernetes-cli:1.12.1 kubernetes-client-api:6.10.0-240.v57880ce8b_0b_2 kubernetes-credentials:0.11 ldap:719.vcb_d039b_77d0d leastload:3.0.0 lockable-resources:1246.v28b_e4cc6fa_16 logging:1.0.0 mail-watcher-plugin:1.19 mailer:472.vf7c289a_4b_420 mapdb-api:1.0.9-40.v58107308b_7a_7 matrix-auth:3.2.2 matrix-project:822.824.v14451b_c0fd42 maven-plugin:3.23 memory-map:2.2.1 (53614) mercurial:1260.vdfb_723cdcc81 metrics:4.2.21-449.v6960d7c54c69 mina-sshd-api-common:2.12.1-101.v85b_e08b_780dd mina-sshd-api-core:2.12.1-101.v85b_e08b_780dd momentjs:1.1.1 monitoring:1.98.0 multiple-scms:0.8 mysql-api:8.3.0-21.v2837a_a_360d57 nested-view:1.33 nexus-artifact-uploader:2.14 nexus-jenkins-plugin:3.15.438.vf87a_0dc45166 nodelabelparameter:1.12.0 oic-auth:4.371.vc7c0c06e8a_f5 okhttp-api:4.11.0-172.vda_da_1feeb_c6e openshift-client:1.1.0.424.v829cb_ccf8798 pam-auth:1.10 parameterized-scheduler:262.v00f3d90585cc parameterized-trigger:787.v665fcf2a_830b_ patch-parameter:1.2 pipeline-build-step:540.vb_e8849e1a_b_d8 pipeline-global-lib-nexus:0.3.2 pipeline-graph-analysis:216.vfd8b_ece330ca_ pipeline-groovy-lib:704.vc58b_8890a_384 pipeline-input-step:491.vb_07d21da_1a_fb_ pipeline-maven:1396.veb_f07b_2fc1d8 pipeline-maven-api:1396.veb_f07b_2fc1d8 pipeline-milestone-step:119.vdfdc43fc3b_9a_ pipeline-model-api:2.2184.v0b_358b_953e69 pipeline-model-declarative-agent:1.1.1 pipeline-model-definition:2.2184.v0b_358b_953e69 pipeline-model-extensions:2.2184.v0b_358b_953e69 pipeline-rest-api:2.34 pipeline-stage-step:312.v8cd10304c27a_ pipeline-stage-tags-metadata:2.2184.v0b_358b_953e69 pipeline-stage-view:2.34 pipeline-utility-steps:2.16.2 plain-credentials:179.vc5cb_98f6db_38 plot:2.1.12 plugin-usage-plugin:4.4 plugin-util-api:4.1.0 pmd:4.0.0 popper-api:1.16.1-3 popper2-api:2.11.6-4 postbuild-task:1.9 prism-api:1.29.0-13 pubsub-light:1.18 pyenv-pipeline:2.1.2 rebuild:332.va_1ee476d8f6d resource-disposer:0.23 rich-text-publisher-plugin:1.5 role-strategy:713.vb_3837801b_8cc run-condition:1.7 saferestart:0.7 saml:4.464.vea_cb_75d7f5e0 scm-api:689.v237b_6d3a_ef7f scoring-load-balancer:81.vb_82d531ff049 script-security:1326.vdb_c154de8669 shared-workspace:1.0.2 shiningpanda:0.24 short-workspace-path:0.3 show-build-parameters:1.0 sidebar-link:2.4.1 snakeyaml-api:2.2-111.vc6598e30cc65 sonar:2.17.2 sse-gateway:1.26 ssh-agent:346.vda_a_c4f2c8e50 ssh-credentials:337.v395d2403ccd4 ssh-slaves:2.948.vb_8050d697fec sshd:3.322.v159e91f6a_550 structs:337.v1b_04ea_4df7c8 subversion:1256.vee91953217b_6 support-core:1427.v083f1d9372a_f svnmerge:2.6 tasks:4.53 test-stability:2.3 text-file-operations:1.3.2 thinBackup:1.19 throttle-concurrents:2.14 timestamper:1.26 token-macro:400.v35420b_922dcb_ translation:1.16 trilead-api:2.142.v748523a_76693 variant:60.v7290fc0eb_b_cd virtualbox:0.7 warnings-ng:11.2.2 windows-azure-storage:419.v4046cd70d2e3 workflow-aggregator:596.v8c21c963d92d workflow-api:1291.v51fd2a_625da_7 workflow-basic-steps:1049.v257a_e6b_30fb_d workflow-cps:3894.vd0f0248b_a_fc4 workflow-cps-global-lib:588.v576c103a_ff86 workflow-cps-global-lib-http:2.48.0 workflow-durable-task-step:1336.v768003e07199 workflow-job:1400.v7fd111b_ec82f workflow-multibranch:773.vc4fe1378f1d5 workflow-scm-step:427.v4ca_6512e7df1 workflow-step-api:657.v03b_e8115821b_ workflow-support:896.v175a_a_9c5b_78f ws-cleanup:0.45 xunit:3.1.4 ```

What Operating System are you using (both controller, and any agents involved in the problem)?

jenkins/jenkins:2.440.1-jdk11

Reproduction steps

Start jenkins on kubernetes with a pvc for /var/jenkins_home, with jcasc defining some credentials (via some http server).
Dump the credentials from the console script, I can view all secrets
Delete the pod
Start again jenkins
Dump the credentials from the console script, secrets are empty strings

Expected Results

Step 5 should be like step 2

Actual Results

On step 5, credentials are empty string when requested from jenkins, even if the credentials.xml contains a hash for it. The hash is the same at each boot, but hash' in config.xml is changing at each boot.

Anything else?

Our secrets are stored in a Azure Key vault, we rely on jenkins casc mecanism to gather the secrets at boot time from the key vault. We are not using kubernetes secrets for that as, it is shared with existing jenkins on VM. The only solution I am thinking about is to add one line in the apply_config.sh, or another init container.

I know jdk11 is old, migration to a more recent jdk is planned as soon as this behavior is fixed (we are stuck today with this problem).

timja commented 1 week ago

Can you supply some more detailed steps please.

What is in your JCasC?

JCasC should manage secrets like this just fine.

Also Azure Key Vault can manage some basic secret types so you don't even need to load them into Jenkins at all: https://github.com/jenkinsci/azure-keyvault-plugin#azure-key-vault-credentials-provider

guilcy commented 6 days ago

Hello, The jcasc is working just fine, it is gathered over https, so does Azure Key Vault. As I said the first boot is just getting everything fine. And we are using a very similar jcasc on our VM (with connection to Azure Key vault). But I see that all config files are trashed by helm before second boot, except the credentials.xml. Not sure why casc is not overinding that file on the second boot, and keep the original values (ie encoded during the first boot). The two point I can see is that

hash value in confi.xml change between boots, so I deduct the encryption key is changing (even if it is not the case on the filesystem).
hash value in the credentials.xml is not changing between the 2 boots, thus data is properly decoded by jenkins after first boot, and only empty string after second boot (tested from script console in jenkins).

Jcasc and Azure key vault are not changed between the boots.

The only way I found to get a right credentials.xml (ie that jenkins get good credentials) and access to the credential inside jenkins script, is to delete credentials.xml before rebooting (thus casc generate it just fine, on subsequent boot).

It is difficult for me to push here the casc file, not sure have the right for that (it is a production data) - I need to setup a specific test.

Should casc have been overriding the credentials.xml at every boot ? Maybe there is a setting that is preventing that behavior to operate properly ?

Thanks for your help !

guilcy commented 6 days ago

Maybe I push the issue in the wrong place, it might be more an issue with the helm-chart ? Is there a way to move that ticket over there ?

timja commented 6 days ago

I've moved it for you.

Can you provide a minimal case please?

Yes on every startup the credentials should be reset.

We need to see the config variables you've used, you can remove anything unrelated and anonymise it but just make sure it still reproduces the issue

guilcy commented 6 days ago

@timja please find attached our helm config, I removed any string linked to our infra (hoppe I did not let something critical). Logic should not be changed, but I removed cloud, permanent node and view definition from jcasc.

anonymized-helm-value-jenkins(1).txt

jcasc-anonymized-nocloud-nopermnode.txt

timja commented 6 days ago

you have a client id and client secret in the helm values, I hope those aren't valid!

timja commented 6 days ago

The Azure Key Vault credentials provider could be used for a number of these credentials so you don't actually need to load them into Jenkins: https://github.com/jenkinsci/azure-keyvault-plugin#azure-key-vault-credentials-provider

You add tags to the credentials for things like the username and the type of Jenkins credential you want it to be.

in terms of simplifying this its normally recommended you create your own values file that only has your customisations in it which then makes it easy to see whats changed.

For an improvement, its better that you don't supply a service principal in the java opts and instead use a managed identity since its credential less and just needs to be assigned to the pod via workload identity.

I can't see specifically anything causing your issue though.

guilcy commented 6 days ago

Is it normal that the helm chart is not removing the credentials.xml file at startup ? Or is not normal that the Jcasc is not updating it when rebooting (encoded value does not change while they change for the config.xml).

guilcy commented 6 days ago

you have a client id and client secret in the helm values, I hope those aren't valid!

Fuck I missed those.

timja commented 6 days ago

Is it normal that the helm chart is not removing the credentials.xml file at startup ? Or is not normal that the Jcasc is not updating it when rebooting (encoded value does not change while they change for the config.xml).

no it should update it on boot, are there logs on startup of it running? If you click reload on the configuration as code page what happens?

guilcy commented 8 hours ago

@timja thanks to your simple suggestion to try to load the jcasc from the GUI. I figure out the source of the problem partially. My jcasc was including a prxoy setting which was not appropriate. Thus at first boot up (no proxy settings), it works, then at second boot there was a proxy settings (loaded from jcasc some how on first boot), and access to the Key vault was doomed.

Now that is fixed, I noticed a few other error message and upgrade a few plugin. Things start to operate properly, but not on first boot.

On first boot, Azure Key vault data are loaded, but not the jenkins credentials build from it. I mean if I have credential declaration like that:

      - "basicSSHUserPrivateKey":
          "description": "My key"
          "id": "ValueEnvFromAzureKeyVault"
          "privateKeySource":
            "directEntry":
              "privateKey": "${KeyEnvFromAzureKeyVault}" 
          "scope": "GLOBAL"
          "username": "Jenkins"
          "usernameSecret": true

"privateKey" is loaded as null

KeyEnvFromAzureKeyVault is at expected value (loaded from keyvault).

If I load the jcasc again from the GUI, then privateKey is not empty. If I boot up another time , then privateKey is not empty.

I need to understand what happens now on the initial boot (which after the plugin upgrade, is different than with the plugin version above). Updated plugins:

    - bouncycastle-api:2.30.1.77-225.v26ea_c9455fd9
    - commons-lang3-api
    - commons-text-api:1.12.0-119.v73ef73f2345d
    - configuration-as-code:1836.vccda_4a_122a_a_e
    - display-url-api:2.200.vb_9327d658781
    - font-awesome-api:6.5.1-3
    - gson-api:2.10.1-15.v0d99f670e0a_7
    - mailer:472.vf7c289a_4b_420
    - prism-api:1.29.0-15
    - scm-api:689.v237b_6d3a_ef7f
    - snakeyaml-api:2.2-111.vc6598e30cc65
    - workflow-api:1291.v51fd2a_625da_7
    - workflow-step-api:657.v03b_e8115821b_

jenkinsci / helm-charts