search

jenkinsci / hashicorp-vault-plugin

Jenkins plugin to populate environment variables from secrets stored in HashiCorp's Vault.
https://plugins.jenkins.io/hashicorp-vault-plugin/
MIT License
217 stars 143 forks source link

Trying to download git repo within withVault block breaks #311

Closed rscottwatson closed 1 year ago

rscottwatson commented 1 year ago

Jenkins and plugins versions report

Environment ```text Jenkins: 2.401.3 OS: Linux - 5.4.0-1094-azure Java: 11.0.19 - Eclipse Adoptium (OpenJDK 64-Bit Server VM) --- ace-editor:1.1 ansicolor:1.0.3 ant:497.v94e7d9fffa_b_9 antisamy-markup-formatter:162.v0e6ec0fcfcf6 apache-httpcomponents-client-4-api:4.5.14-150.v7a_b_9d17134a_5 authentication-tokens:1.53.v1c90fd9191a_b_ authorize-project:1.7.1 azure-commons:1.1.3 blueocean:1.27.5 blueocean-autofavorite:1.2.5 blueocean-bitbucket-pipeline:1.27.5 blueocean-commons:1.27.5 blueocean-config:1.27.5 blueocean-core-js:1.27.5 blueocean-dashboard:1.27.5 blueocean-display-url:2.4.2 blueocean-events:1.27.5 blueocean-executor-info:1.27.5 blueocean-git-pipeline:1.27.5 blueocean-github-pipeline:1.27.5 blueocean-i18n:1.27.5 blueocean-jira:1.27.5 blueocean-jwt:1.27.5 blueocean-personalization:1.27.5 blueocean-pipeline-api-impl:1.27.5 blueocean-pipeline-editor:1.27.5 blueocean-pipeline-scm-api:1.27.5 blueocean-rest:1.27.5 blueocean-rest-impl:1.27.5 blueocean-web:1.27.5 bootstrap4-api:4.6.0-6 bootstrap5-api:5.3.0-1 bouncycastle-api:2.29 branch-api:2.1122.v09cb_8ea_8a_724 build-timeout:1.31 caffeine-api:3.1.6-115.vb_8b_b_328e59d8 checks-api:2.0.0 cloudbees-bitbucket-branch-source:825.va_6a_dc46a_f97d cloudbees-folder:6.815.v0dd5a_cb_40e0e command-launcher:106.vb_a_b_8f751309c commons-lang3-api:3.13.0-62.v7d18e55f51e2 commons-text-api:1.10.0-68.v0d0b_c439292b_ credentials:1271.v54b_1c2c6388a_ credentials-binding:631.v861c06d062b_4 data-tables-api:1.13.5-1 display-url-api:2.3.8 docker-commons:439.va_3cb_0a_6a_fb_29 docker-workflow:563.vd5d2e5c4007f durable-task:513.vc48a_a_075a_d93 echarts-api:5.4.0-5 email-ext:2.100 favorite:2.4.3 font-awesome-api:6.4.0-2 git:5.2.0 git-client:4.4.0 git-server:99.va_0826a_b_cdfa_d github:1.37.2 github-api:1.314-431.v78d72a_3fe4c3 github-branch-source:1732.v3f1889a_c475b_ github-oauth:588.vf696a_350572a_ gradle:2.8.2 handlebars:3.0.8 handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953 hashicorp-vault-pipeline:1.4 hashicorp-vault-plugin:360.v0a_1c04cf807d htmlpublisher:1.32 instance-identity:173.va_37c494ec4e5 ionicons-api:56.v1b_1c8c49374e jackson2-api:2.15.2-350.v0c2f3f8fc595 jakarta-activation-api:2.0.1-3 jakarta-mail-api:2.0.1-3 javax-activation-api:1.2.0-6 javax-mail-api:1.6.2-9 jaxb:2.3.8-1 jdk-tool:73.vddf737284550 jenkins-design-language:1.27.5 jersey2-api:2.40-1 jira:3.10 jjwt-api:0.11.5-77.v646c772fddb_0 jquery-detached:1.2.1 jquery3-api:3.7.0-1 jsch:0.2.8-65.v052c39de79b_2 junit:1217.v4297208a_a_b_ce kubernetes:3952.v88e3b_0cf300b_ kubernetes-cd:2.3.1 kubernetes-cli:1.12.0 kubernetes-client-api:6.4.1-215.v2ed17097a_8e9 kubernetes-credentials:0.10.0 kubernetes-credentials-provider:1.225.v14f9e6b_28f53 kubernetes-pipeline-devops-steps:1.6 ldap:694.vc02a_69c9787f lockable-resources:1185.v0c528656ce04 mailer:463.vedf8358e006b_ matrix-auth:3.1.10 matrix-project:802.v8013b_40c7edc mercurial:1260.vdfb_723cdcc81 metrics:4.2.18-442.v02e107157925 mina-sshd-api-common:2.10.0-69.v28e3e36d18eb_ mina-sshd-api-core:2.10.0-69.v28e3e36d18eb_ momentjs:1.1.1 okhttp-api:4.11.0-157.v6852a_a_fa_ec11 pam-auth:1.10 pipeline-build-step:505.v5f0844d8d126 pipeline-github-lib:42.v0739460cda_c4 pipeline-graph-analysis:202.va_d268e64deb_3 pipeline-groovy-lib:671.v07c339c842e8 pipeline-input-step:477.v339683a_8d55e pipeline-milestone-step:111.v449306f708b_7 pipeline-model-api:2.2144.v077a_d1928a_40 pipeline-model-definition:2.2144.v077a_d1928a_40 pipeline-model-extensions:2.2144.v077a_d1928a_40 pipeline-rest-api:2.33 pipeline-stage-step:305.ve96d0205c1c6 pipeline-stage-tags-metadata:2.2144.v077a_d1928a_40 pipeline-stage-view:2.33 plain-credentials:143.v1b_df8b_d3b_e48 plugin-util-api:3.3.0 popper-api:1.16.1-3 popper2-api:2.11.6-2 pubsub-light:1.17 resource-disposer:0.23 scm-api:676.v886669a_199a_a_ script-security:1264.vecf66020eb_7d simple-theme-plugin:160.vb_76454b_67900 slack:664.vc9a_90f8b_c24a_ snakeyaml-api:1.33-95.va_b_a_e3e47b_fa_4 sse-gateway:1.26 ssh-credentials:308.ve4497b_ccd8f4 ssh-slaves:2.916.vd17b_43357ce4 sshd:3.312.v1c601b_c83b_0e structs:324.va_f5d6774f3a_d timestamper:1.26 token-macro:383.v36161104b_002 trilead-api:2.84.v72119de229b_7 variant:59.vf075fe829ccb workflow-aggregator:596.v8c21c963d92d workflow-api:1251.vd4889a_b_0a_065 workflow-basic-steps:1042.ve7b_140c4a_e0c workflow-cps:3731.ve4b_5b_857b_a_d3 workflow-cps-global-lib:609.vd95673f149b_b workflow-durable-task-step:1278.v94b_dc2b_50c6f workflow-job:1316.vd2290d3341a_f workflow-multibranch:756.v891d88f2cd46 workflow-scm-step:415.v434365564324 workflow-step-api:639.v6eca_cd8c04a_a_ workflow-support:848.v5a_383b_d14921 ws-cleanup:0.45 ```

What Operating System are you using (both controller, and any agents involved in the problem)?

Running on kubernetes 1.24.6 on AKS

Reproduction steps

This is producing the error for me.

def now = new Date()
def label = 'acr-sync-' + now.format("yyMMdd.HHmm", TimeZone.getTimeZone('America/Los_Angeles'))

def download_repo() {
    dir(targetDir){
        withVault(configuration: [engineVersion: 2, timeout: 60, vaultCredentialId: VAULT_BOT, vaultNamespace: VAULT_NAMESPACE, vaultUrl: 'https://vault.url'], vaultSecrets: [[path: VAULT_GBAAS_PATH, secretValues: [[vaultKey: 'git_bot_username'], [vaultKey: 'git_bot_token']]]]){
                sh """ echo 'My working directory is '; pwd; env; echo ${git_bot_username} """
                 git 'https://github.com/rtyley/small-test-repo'
        }
    }
}

podTemplate(label: label, yaml: """apiVersion: v1
kind: Pod
metadata:
  namespace: terraform
  labels:
    ${label}: yes
spec:
  imagePullSecrets:
    - name: acr-sp-secret
  containers:
    - name: terraform-base
      imagePullPolicy: Always
      image: "${docker_registry}/base-images/terraform:v4.8"
      command:
        - cat
      tty: true
""")

{
    node(label) {
        stage ('download repo') {
             download_repo()
        }
    }
}

Expected Results

The repo should be downloaded

Actual Results

The recommended git tool is: NONE
No credentials specified
Cloning the remote Git repository
[Pipeline] }
[Pipeline] // withVault
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 10.244.3.221/10.244.3.221:58770
        at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784)
        at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
        at hudson.remoting.Channel.call(Channel.java:1000)
        at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:153)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:138)
        at com.sun.proxy.$Proxy186.execute(Unknown Source)
        at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1226)
        at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1309)
        at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:129)
        at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:97)
        at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:84)
        at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Also:   org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: e60ebf5b-c68e-46f3-8392-d8fb07742821
java.lang.IllegalStateException: Not running on the Jenkins controller JVM
    at jenkins.util.JenkinsJVM.checkJenkinsJVM(JenkinsJVM.java:46)
    at org.jenkinsci.plugins.credentialsbinding.masking.SecretPatterns.getAggregateSecretPattern(SecretPatterns.java:57)
    at com.datapipe.jenkins.vault.log.MaskingConsoleLogFilter.lambda$decorateLogger$0(MaskingConsoleLogFilter.java:43)
    at org.jenkinsci.plugins.credentialsbinding.masking.SecretPatterns$MaskingOutputStream.eol(SecretPatterns.java:93)
    at hudson.console.LineTransformationOutputStream.eol(LineTransformationOutputStream.java:61)
    at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:57)
    at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:75)
    at java.base/java.io.PrintStream.write(Unknown Source)
    at java.base/sun.nio.cs.StreamEncoder.writeBytes(Unknown Source)
    at java.base/sun.nio.cs.StreamEncoder.implFlushBuffer(Unknown Source)
    at java.base/sun.nio.cs.StreamEncoder.flushBuffer(Unknown Source)
    at java.base/java.io.OutputStreamWriter.flushBuffer(Unknown Source)
    at java.base/java.io.PrintStream.newLine(Unknown Source)
    at java.base/java.io.PrintStream.println(Unknown Source)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:807)
    at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:170)
    at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)
    at hudson.remoting.UserRequest.perform(UserRequest.java:211)
    at hudson.remoting.UserRequest.perform(UserRequest.java:54)
    at hudson.remoting.Request$2.run(Request.java:377)
    at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
    at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:125)
    at java.base/java.lang.Thread.run(Unknown Source)

Could not update commit status, please check if your scan credentials belong to a member of the organization or a collaborator of the repository and repo:status scope is selected

GitHub has been notified of this commit’s build result

Anything else?

No response

ansig commented 1 year ago

We also have this problem. It happens whenever we call anything that requires a node within a withVault block.

For example, we found it when using "tool":

pipeline {
    agent any
    stages {
        stage('Foo') {
            steps {
                withVault(
                    configuration: [timeout: 60, vaultCredentialId: 'jenkins-vault-app-role', vaultUrl: "https://vault.example.com:8200"],
                    vaultSecrets: [[path: "secret/apps/jenkins", secretValues: [
                        [envVar: 'FOO', vaultKey: 'BAR']]]]) {
                    echo "${tool 'govc-v0.24.0'}"
                }
            }
        }
    }
    post {
        cleanup {
            deleteDir()
        }
    }
}

Output:

java.lang.IllegalStateException: Not running on the Jenkins controller JVM
    at jenkins.util.JenkinsJVM.checkJenkinsJVM(JenkinsJVM.java:46)
    at org.jenkinsci.plugins.credentialsbinding.masking.SecretPatterns.getAggregateSecretPattern(SecretPatterns.java:57)
    at com.datapipe.jenkins.vault.log.MaskingConsoleLogFilter.lambda$decorateLogger$0(MaskingConsoleLogFilter.java:43)
    at org.jenkinsci.plugins.credentialsbinding.masking.SecretPatterns$MaskingOutputStream.eol(SecretPatterns.java:93)
    at hudson.console.LineTransformationOutputStream.eol(LineTransformationOutputStream.java:61)
    at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:57)
    at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:75)
    at java.base/java.io.PrintStream.write(PrintStream.java:568)
    at java.base/sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:234)
    at java.base/sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:313)
    at java.base/sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:111)
    at java.base/java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:178)
    at java.base/java.io.PrintStream.writeln(PrintStream.java:723)
    at java.base/java.io.PrintStream.println(PrintStream.java:1051)
    at hudson.Launcher.printCommandLine(Launcher.java:817)
    at hudson.Launcher.maskedPrintCommandLine(Launcher.java:832)
    at hudson.Launcher$LocalLauncher.launch(Launcher.java:977)
    at hudson.Launcher$ProcStarter.start(Launcher.java:509)
    at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1398)
    at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1340)
    at hudson.remoting.UserRequest.perform(UserRequest.java:211)
    at hudson.remoting.UserRequest.perform(UserRequest.java:54)
    at hudson.remoting.Request$2.run(Request.java:377)
    at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:833)

But if we just do, say, "echo" it works fine:

pipeline {
    agent any
    stages {
        stage('Foo') {
            steps {
                withVault(
                    configuration: [timeout: 60, vaultCredentialId: 'jenkins-vault-app-role', vaultUrl: "https://vault.example.com:8200"],
                    vaultSecrets: [[path: "secret/apps/jenkins", secretValues: [
                        [envVar: 'FOO', vaultKey: 'BAR']]]]) {
                    echo "${FOO}"
                }
            }
        }
    }
    post {
        cleanup {
            deleteDir()
        }
    }
}

Output:

[Pipeline] stage
[Pipeline] { (Foo)
[Pipeline] echo
Warning: A secret was passed to "echo" using Groovy String interpolation, which is insecure.
         Affected argument(s) used the following variable(s): [FOO]
         See https://jenkins.io/redirect/groovy-string-interpolation for details.
****
[Pipeline] }

It does not matter if we use withVault in the Pipeline options, even if that is run with "agent none" and the agent is assigned in the stage instead:

pipeline {
    agent none
    options {
        withVault(
            configuration: [timeout: 60, vaultCredentialId: 'jenkins-vault-app-role', vaultUrl: "https://vault.example.com:8200"],
            vaultSecrets: [[path: "secret/apps/jenkins", secretValues: [
                [envVar: 'FOO', vaultKey: 'BAR']]]])
    }
    stages {
        stage('Foo') {
            agent any
            steps {
                echo "${tool 'govc-v0.24.0'}"
            }
            post {
                cleanup {
                    deleteDir()
                }
            }
        }
    }
}

So it seems related specifically to printing masked values. I believe it is related to a new version of the credentials-binding plugin (631.v861c06d062b_4) that includes the following PR: https://github.com/jenkinsci/credentials-binding-plugin/pull/260

The PR is related to: https://www.jenkins.io/security/advisory/2023-04-12/#SECURITY-3075

It might be related to: https://www.jenkins.io/security/advisory/2023-05-16/#SECURITY-3077

alviss7 commented 1 year ago

I confirm that it's linked to this PR, I've downgraded the version of this plugin to make my pipelines work again.

jglick commented 1 year ago

Tracked as https://issues.jenkins.io/browse/JENKINS-71788 but unfortunately this plugin seems to have kept both GitHub Issues and Jira open, so it is unclear which is authoritative.