Vault credentials not found for KV version 2 based secrets using Vault token as credential

[pasuder]

Based on following example, I tried to setup working retrieval of secrets from Vault KV version 2 engine and was unable to have it working:

    def secrets = [
        [path: 'secret/testing', engineVersion: 1, secretValues: [
            [envVar: 'testing', vaultKey: 'value_one'],
            [envVar: 'testing_again', vaultKey: 'value_two']]],
        [path: 'secret/another_test', engineVersion: 2, secretValues: [
            [vaultKey: 'another_test']]]
    ]

Working example of scripted pipeline for KV version 1 secret engine:

node {
    def secrets = [
        [path: 'secret/testing/testing', engineVersion: 1, secretValues: [
            [envVar: 'testing', vaultKey: 'key']]]
    ]

    def configuration = [vaultUrl: 'https://vault/',
                         vaultCredentialId: 'vault_token',
                         engineVersion: 1]

    withVault([configuration: configuration, vaultSecrets: secrets]) {
        sh 'echo $testing'
    }
}

Not working example of scripted pipeline for KV version 2 secret engine:

node {
    def secrets = [
        [path: 'secret/testing_v2/testing', engineVersion: 2, secretValues: [
            [envVar: 'testing', vaultKey: 'key_v2']]]
    ]

    def configuration = [vaultUrl: 'https://vault/',
                         vaultCredentialId: 'vault_token',
                         engineVersion: 2]

    withVault([configuration: configuration, vaultSecrets: secrets]) {
        sh 'echo $testing'
    }
}

Build error:

com.datapipe.jenkins.vault.exception.VaultPluginException: Vault credentials not found for 'secret/testing_v2/testing'
    at com.datapipe.jenkins.vault.VaultAccessor.responseHasErrors(VaultAccessor.java:235)
    at com.datapipe.jenkins.vault.VaultAccessor.retrieveVaultSecrets(VaultAccessor.java:170)
    at com.datapipe.jenkins.vault.VaultBindingStep$Execution.doStart(VaultBindingStep.java:115)
    at org.jenkinsci.plugins.workflow.steps.GeneralNonBlockingStepExecution.lambda$run$0(GeneralNonBlockingStepExecution.java:77)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Finished: FAILURE

vault_token is a token used to access Vault configured with JCasC jenkins.yml

credentials:
  system:
    domainCredentials:
    - credentials:
      - vaultTokenCredential:
          id: "vault_token"
          scope: GLOBAL
          token: "${JCASC_VAULT_TOKEN}"

Vault secrets retrievals using Vault CLI:

/ # vault read secret/testing/testing
Key                 Value
---                 -----
refresh_interval    <some value>
key                 value
/ # vault read secret/testing_v2/testing
WARNING! The following warnings were returned from Vault:

  * Invalid path for a versioned K/V secrets engine. See the API docs for the
  appropriate API endpoints to use. If using the Vault CLI, use 'vault kv get'
  for this operation.

/ # vault kv get secret/testing_v2/testing
======= Metadata =======
Key                Value
---                -----
created_time       <some value>
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

===== Data =====
Key       Value
---       -----
key_v2    value_v2
/ # 

Versions:

• Jenkins: 2.331
• Vault: 1.9.1
• HashiCorp Vault Plugin: 336.v182c0fbaaeb7

May I ask if KV v2 secrets retrieval does work? If yes, how to setup it? Thanks!

[pasuder]

Same behavior on Jenkins 2.333 and Vault 1.9.3

[pasuder]

I went deep..

tl;dr: %2F instead of / same issue https://github.com/jenkinsci/hashicorp-vault-plugin/issues/75#issuecomment-582105896

Longer version:

A bit more context of deployment where Jenkins and Vault are setup. Both are deployed with images pulled from docker.io, runs on the same server and are behind Traefik. Communication between Jenkins and Vault happens over Traefik. Did not check what Traefik does with URLs, but what I found is following:

That part of code:

https://github.com/jenkinsci/hashicorp-vault-plugin/blob/182c0fbaaeb77e222d49822656e29b8c40422f7f/src/main/java/com/datapipe/jenkins/vault/VaultAccessor.java#L114-L122

Should return call this:

    public LogicalResponse read(final String path) throws VaultException {
        if (this.engineVersionForSecretPath(path).equals(2)) {
            return read(path, true, logicalOperations.readV2);
        } else return read(path, true, logicalOperations.readV1);
    }

For Vault URL creation, that helper is called:

    public static String adjustPathForReadOrWrite(final String path, final int prefixPathLength,
            final Logical.logicalOperations operation) {
        final List<String> pathSegments = getPathSegments(path);
        if (operation.equals(Logical.logicalOperations.readV2) || operation
                .equals(Logical.logicalOperations.writeV2)) {
            // Version 2
            final StringBuilder adjustedPath = new StringBuilder(
                    addQualifierToPath(pathSegments, prefixPathLength, "data"));
            if (path.endsWith("/")) {
                adjustedPath.append("/");
            }
            return adjustedPath.toString();
        } else {
            // Version 1
            return path;
        }
    }

It does the job with setting data as per README (permalink).

And the question is: at which level that %2F is converted to /? In this plugin, in that external library used to access Vault, somewhere in Jenkins, on Traefik (it does SSL termination)?

[pasuder]

Excluded Traefik from communication between Jenkins and Vault - used direct URL aka http://vault:8200 and same error with / in secret engine name:

com.datapipe.jenkins.vault.exception.VaultPluginException: Vault credentials not found for 'secret/testing_v2/testing'

With secret%2Ftesting_v2/testing as path it does work fine.

[hchakroun]

Take a look to this : https://github.com/jenkinsci/hashicorp-vault-plugin/issues/209