search

jenkinsci / hashicorp-vault-plugin

Jenkins plugin to populate environment variables from secrets stored in HashiCorp's Vault.
https://plugins.jenkins.io/hashicorp-vault-plugin/
MIT License
217 stars 143 forks source link

Plugin auth using AWS IAM doesn't work unless I re-save the Jenkins Credential form in the Jenkins UI #330

Closed jonkipu closed 3 months ago

jonkipu commented 3 months ago

Jenkins and plugins versions report

Environment ```text Jenkins: 2.458 OS: Linux - 5.10.215-203.850.amzn2.aarch64 Java: 21.0.3 - Eclipse Adoptium (OpenJDK 64-Bit Server VM) --- amazon-ecs:1.49 antisamy-markup-formatter:162.v0e6ec0fcfcf6 apache-httpcomponents-client-4-api:4.5.14-208.v438351942757 asm-api:9.7-33.v4d23ef79fcc8 aws-credentials:231.v08a_59f17d742 aws-java-sdk-ec2:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-ecs:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-efs:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-minimal:1.12.696-451.v0651a_da_9ca_ec bootstrap5-api:5.3.3-1 bouncycastle-api:2.30.1.77-225.v26ea_c9455fd9 branch-api:2.1169.va_f810c56e895 caffeine-api:3.1.8-133.v17b_1ff2e0599 cloudbees-folder:6.940.v7fa_03b_f14759 commons-lang3-api:3.13.0-62.v7d18e55f51e2 commons-text-api:1.11.0-109.vfe16c66636eb_ configuration-as-code:1775.v810dc950b_514 credentials:1337.v60b_d7b_c7b_c9f credentials-binding:677.vdc9d38cb_254d display-url-api:2.204.vf6fddd8a_8b_e9 durable-task:555.v6802fe0f0b_82 font-awesome-api:6.5.2-1 git:5.2.2 git-client:4.7.0 gson-api:2.10.1-15.v0d99f670e0a_7 hashicorp-vault-plugin:368.v48134f694db_f instance-identity:185.v303dc7c645f9 ionicons-api:74.v93d5eb_813d5f jackson2-api:2.17.0-379.v02de8ec9f64c jakarta-activation-api:2.1.3-1 jakarta-mail-api:2.1.3-1 javax-activation-api:1.2.0-6 javax-mail-api:1.6.2-9 jaxb:2.3.9-1 job-dsl:1.87 joda-time-api:2.12.7-29.v5a_b_e3a_82269a_ json-api:20240303-41.v94e11e6de726 mailer:472.vf7c289a_4b_420 mina-sshd-api-common:2.12.1-101.v85b_e08b_780dd mina-sshd-api-core:2.12.1-101.v85b_e08b_780dd pipeline-build-step:540.vb_e8849e1a_b_d8 pipeline-groovy-lib:704.vc58b_8890a_384 pipeline-input-step:495.ve9c153f6067b_ pipeline-milestone-step:119.vdfdc43fc3b_9a_ pipeline-model-api:2.2198.v41dd8ef6dd56 pipeline-model-definition:2.2198.v41dd8ef6dd56 pipeline-model-extensions:2.2198.v41dd8ef6dd56 pipeline-stage-step:312.v8cd10304c27a_ pipeline-stage-tags-metadata:2.2198.v41dd8ef6dd56 plain-credentials:182.v468b_97b_9dcb_8 plugin-util-api:4.1.0 prism-api:1.29.0-15 scm-api:690.vfc8b_54395023 script-security:1336.vf33a_a_9863911 snakeyaml-api:2.2-111.vc6598e30cc65 ssh-credentials:337.v395d2403ccd4 structs:337.v1b_04ea_4df7c8 trilead-api:2.142.v748523a_76693 variant:60.v7290fc0eb_b_cd workflow-aggregator:596.v8c21c963d92d workflow-api:1291.v51fd2a_625da_7 workflow-basic-steps:1058.vcb_fc1e3a_21a_9 workflow-cps:3894.3896.vca_2c931e7935 workflow-durable-task-step:1336.v768003e07199 workflow-job:1415.v4f9c9131248b_ workflow-multibranch:791.v28fb_f74dfca_e workflow-scm-step:427.v4ca_6512e7df1 workflow-step-api:657.v03b_e8115821b_ workflow-support:907.v6713a_ed8a_573 ```

What Operating System are you using (both controller, and any agents involved in the problem)?

Only controller is involved, running Alpine Linux

Reproduction steps

I'm using AWS IAM auth to auth to my Vault instance (non-cluster)

Using CasC to configure the plugin and credential for it:

unclassified:
  hashicorpVault:
    configuration:
      vaultCredentialId: "vaultAWSCredential"
      vaultUrl: "https://my-vault-instance"
credentials:
  system:
    domainCredentials:
      - credentials:
          - vaultAwsIamCredential:
              id: "vaultAWSCredential"
              scope: GLOBAL

I have enabled all required auth, secret engines and policies on the Vault side.

I have configured a test VaultSSHUsernamePassword credential to try and fetch from vault (a secret I have previously created on vault)

I am using the 'Test Vault Secrets Retrieval' button to test it and am getting this error:

FAILED to retrieve username key:
com.datapipe.jenkins.vault.exception.VaultPluginException: Key username could not be found in path kv/test

I have tried to fetch the credential via a pipeline job but am getting 'Access Denied' error.

The only way I got this to work was to access the vaultAWSCredential credential via the Jenkins UI re-save the form (without changing anything), from that point all secret fetching started to work.

I then re-cycled the vault container and again the errors appeared, I then proceeded to re-save the form again and everything started working, I repeated this several times and have confirmed that this is the only change I made to make this work.

I also started the Vault container with debug logs and it seems that until I re-save the form, and trying to fetch secrets no logs are written on the Vault side in regards to secret fetching, when I re-save the form and try to fetch secrets I do see logs relating to auth:

auth.aws.auth_aws_d6a5be79: submitting caller identity request: endpoint=https://sts.amazonaws.com
identity: creating a new entity...

Another change I noticed is that when trying to fetch secrets before re-saving the form, the Jenkins log shows the following:

com.datapipe.jenkins.vault.credentials.common.VaultHelper getVaultSecret
Retrieving vault secret path=kv/test engineVersion=2
com.datapipe.jenkins.vault.credentials.common.VaultHelper retrieveVaultCredentials
Retrieving vault credential ID : vaultAWSCredential
com.datapipe.jenkins.vault.credentials.AbstractVaultTokenCredentialWithExpiration
Expiration for  is java.util.GregorianCalendar .... 
com.datapipe.jenkins.vault.credentials.AbstractVaultTokenCredentialWithExpiration
Auth token is still valid for policies ''

But when re-saving the form and fetching secrets the logs read as:

com.datapipe.jenkins.vault.credentials.common.VaultHelper getVaultSecret
Retrieving vault secret path=kv/test engineVersion=2
com.datapipe.jenkins.vault.credentials.common.VaultHelper retrieveVaultCredentials
Retrieving vault credential ID : vaultAWSCredential
com.datapipe.jenkins.vault.AwsHelper
Creating GetCallerIdentity request
com.datapipe.jenkins.vault.AwsHelper
Acquiring AWS credentials
com.datapipe.jenkins.vault.AwsHelper
AWS Access Key ID: ...
com.datapipe.jenkins.vault.AwsHelper
Signing GetCallerIdentity request
com.datapipe.jenkins.vault.credentials.common.VaultHelper getVaultSecret
Retrieving vault secret path=kv/test engineVersion=2
com.datapipe.jenkins.vault.credentials.common.VaultHelper retrieveVaultCredentials
Retrieving vault credential ID : vaultAWSCredential

Expected Results

Secret fetching should work out of the box

Actual Results

Errors about secret not being found at path or access denied

Anything else?

No response

Are you interested in contributing a fix?

No response

jonkipu commented 3 months ago

Added some debug logs to the plugin and rebuilt it, seems that this stems from the plugin caching its auth token with Vault, when restarting vault to test I guess the token was still cached (although I saw no logs in Vault to match this), when updating the credentials I guess it must have reset the token's expiry and generated a new token