VaultException : 403 Forbidden for KV and DB Secret engines

[AnujaB219]

Issue:

The app is not able to read from Vault and fails with

org.springframework.vault.VaultException: Status 403 Forbidden [db/oracle/dev]: 1 error occurred:

Sep 06 09:55:31 <server> java[2764496]:         * permission denied

Sep 06 09:55:31 <server> java[2764496]:         at org.springframework.vault.client.VaultResponses.buildException(VaultResponses.java:84)

Sep 06 09:55:31 <server> java[2764496]:         at org.springframework.vault.core.VaultTemplate.lambda$doRead$5(VaultTemplate.java:472)

Sep 06 09:55:31 <server> java[2764496]:         at org.springframework.vault.core.VaultTemplate.doWithSession(VaultTemplate.java:451)

Sep 06 09:55:31 <server> java[2764496]:         at org.springframework.vault.core.VaultTemplate.doRead(VaultTemplate.java:461)

Sep 06 09:55:31 <server> java[2764496]:         at org.springframework.vault.core.VaultTemplate.read(VaultTemplate.java:356)

 Caused by: org.springframework.web.client.HttpClientErrorException$Forbidden: 403 Forbidden: "{"errors":["1 error occurred:\n\t* permission denied\n\n"]}<EOL>"

Sep 09 15:04:20 <server> java[673780]:         at org.springframework.web.client.HttpClientErrorException.create(HttpClientErrorException.java:109)

Sep 09 15:04:20 <server> java[673780]:         at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:183)

Sep 09 15:04:20 <server> java[673780]:         at org.sp

OR when I remove DB details :

 MessageStatus 403 Forbidden [sec_kv/data/jks/mq/sit]: permission denied

Sep 06 16:19:32 <server> java[3994638]: 2024-09-06T16:19:32.666+10:00  INFO 3994638 --- [cls-message-router] [           main] au.com.nab.service.VaultServiceAzure     : org.springframework.web.client.HttpClientErrorException$Forbidden: 403 Forbidden: "{"errors":["permission denied"]}<EOL>"

Sep 06 16:19:32 <server> java[3994638]: 2024-09-06T16:19:32.666+10:00 ERROR 3994638 --- [cls-message-router] [           main] au.com.nab.config.JmsConfig              : MQ Queue Connection Factory failed

Sep 06 16:19:32 <server> java[3994638]: java.lang.IllegalStateException: Exception occurred while fetching the password from Hashi Vault {} [Ljava.lang.StackTraceEle

CONFIG:

I have a Hashicorp vault with the below secret engines:

$vault version

Vault v1.16.2+ent (9ae30c2ae273xxxxx), built 2024-04-22T16:26:15Z

$vault secrets list

Path Type Accessor Description

---

db/oracle/dev/ database database_xxxx. A database secret engine

sec_kv/ kv kv_xxxxxx A Key/Value secret store

…..

Values I need to read from Spring App:

$vault kv get -mount="sec_kv" "jks/mq/sit"

======= Secret Path =======

sec_kv/data/jks/mq/sit

======= Metadata =======

Key Value

---

created_time 2024-08-21T06:38:48.524009746Z

custom_metadata

deletion_time n/a

destroyed false

version 960

====== Data ======

Key Value

---

password XXX

$vault read db/oracle/dev/static-creds/mydbuser

Key Value

---

last_vault_rotation 2024-08-30T12:32:35.605946627+10:00

password xxxxxxxxxxxxx

rotation_period 2160h

ttl 2040h50m14s

username mydbuser

Application.yaml:

spring:
  application:
    name: message-router
  cloud:
    vault:
      azure-msi:
        metadata-service: http://169.254.169.254/metadata/instance?api-version=2017-12-01
        identity-token-service: http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fmanagement.azure.com%2F
      authentication: azure_msi
      fail-fast: true
      enabled: false

Application-azure-sit.yaml:

spring:
  config:
    activate:
      on-profile: azure-sit
    import: vault://sec_kv, vault://db/oracle/dev
  cloud:
    vault:
      azure-msi:
        azure-path: az/mountpathforapp
        role: appuser
      namespace: ns_nonprod
      kv:
        enabled: true
        backend: sec_kv
        default-context: jks/mq/sit
        profile-separator: /
      uri: https://<<hostname>>:8200
      enabled: true
      database:
        enabled: true
        role: appuser
        backend: db/oracle/dev
        username-property: spring.datasource.username
        password-property: spring.datasource.password
        static-role: true

ACL Policy for appuser:

# Config Azure

path "auth/azure/config" {

  capabilities = ["read"]

}

# Config Azure for new mount point naming convention

path "auth/az/+/config" {

  capabilities = ["read"]

}

# Read Azure Auth Config

path "auth/az/+/+/config" {

  capabilities = ["read"]

}

# List Azure roles

path "auth/azure/role" {

  capabilities = ["read", "list"]

}

# List Azure roles

path "auth/az/+/role" {

  capabilities = ["read", "list"]

}

# List Azure roles for new mount point naming convention

path "auth/az/+/+/role" {

  capabilities = ["read", "list"]

}

# Manage Azure roles for new mount point naming convention

path "auth/azure/role/*" {

  capabilities = ["create", "read", "update", "delete", "list"]

  denied_parameters = {

    "token_policies" = ["*admin*", "*operator*"]

  }

}

# Manage Azure roles for new mount point naming convention

path "auth/az/+/role/*" {

  capabilities = ["create", "read", "update", "delete", "list"]

  denied_parameters = {

    "token_policies" = ["*admin*", "*operator*"]

  }

}

# Manage Azure roles

path "auth/az/+/+/role/*" {

  capabilities = ["create", "read", "update", "delete", "list"]

  denied_parameters = {

    "token_policies" = ["*admin*", "*operator*"]

  }

}

# Manage Secrets

path "sec_*" {

  capabilities = ["create", "read", "update", "delete", "list", "patch"]

}

# Manage secrets under child namespace

path "+/sec_*" {

  capabilities = ["create", "read", "update", "delete", "list"]

}

# Revoke

path "sys/revoke" {

   capabilities = [ "update" ]

}

# Deny all permission to Crypto path

path "sec_crypto/*" {

  capabilities = ["deny"]

}

# List namespaces

path "sys/namespaces*" {

  capabilities = ["read", "list"]

}

# Manage Static Secrets Engine

path "sys/mounts/sec_*" {

  capabilities = ["create", "read", "update", "delete", "list"]

  allowed_parameters = {

    "*" = []

    "type" = ["kv-v2", "kv"]

  }

}

# List Secrets

path "sys/mounts" {

  capabilities = ["read", "list"]

}

# List auth backends

path "sys/auth" {

  capabilities = ["read", "list"]

}

# Read config

path "ad/+/config" {

  capabilities = ["read", "list"]

}

# List and Read Static Roles

path "ad/+/static-role/*" {

  capabilities = ["read", "list"]

}

# Read Static Creds

path "ad/+/static-cred/*" {

  capabilities = ["read", "list"]

}

# Rotate Static Creds

path "ad/+/rotate-role/*" {

  capabilities = ["read", "list", "update"]

}

# Rotate Root Creds

path "ad/+/rotate-root" {

  capabilities = ["read", "list", "update"]

}

#DB config

path "db/oracle/dev/*" {

  capabilities = [ "read" ]

}

path "db/oracle/dev/static-creds/cls_appluser/*" {

  capabilities = [ "read" ]

}

path "db/oracle/dev/static-creds/cls_appluser" {

  capabilities = [ "read" ]

}

path "db/oracle/dev/config/*" {

  capabilities = ["read", "list"]

}

#/

path "db/oracle/dev/config/*" {

  capabilities = ["update"]

  allowed_parameters = {

    "connection_url" = []

    "password" = []

    "password_policy" = []

    "username" = []

  }

}

#/

path "db/oracle/dev/static-roles/*" {

  capabilities = ["create", "read", "list", "update", "delete"]

  allowed_parameters = {

    "*" = []

    "rotation_period" = ["2d", "8d", "15d", "31d", "45d", "90d"]

  }

}

#/

path "db/oracle/dev/roles/*" {

  capabilities = ["create", "read", "list", "update", "delete"]

  allowed_parameters = {

    "*" = []

    "rotation_period" = ["2d", "8d", "15d", "31d", "45d", "90d"]

  }

}

#read credentials from the database engine

path "db/oracle/dev/creds/*" {

  capabilities = ["read", "list", "update"]

}

#read credentials from the database engine

path "db/oracle/dev/static-creds/*" {

  capabilities = ["read", "list", "update"]

}

#Key-Value

path "sec_kv/cls-message-router" {

    capabilities = [ "read" ]

}

path "sec_kv/cls-message-router/*" {

    capabilities = [ "read" ]

}

#Application

path "sec_kv/application" {

    capabilities = [ "read" ]

}

path "sec_kv/application/*" {

    capabilities = [ "read" ]

}

I am using the below :

"org.springframework.vault:spring-vault-core:3.1.2", "org.springframework.cloud:spring-cloud-starter-vault-config:3.1.2", "org.springframework.cloud:spring-cloud-vault-config-databases:3.1.2",

And springboot version 3.3,0. I tried adding Revoke as mentioned in issue . But I still get the 403. Any idea please?

[mp911de]

Make sure to import import: vault:// to enable database mounts. Otherwise, import: vault://db/oracle/dev considers db/oracle/dev a key-value mount.

The other issue is that path sec_kv/data/jks/mq/sit doesn't match sec_kv/cls-message-router/* nor sec_kv/application roles. Please fix your roles respective application config.

[AnujaB219]

I added import: vault://

and set kv: enabled: true backend: sec_kv default-context: jks/mq/sit profile-separator: / application-name: data/jks/mq/sit

The ACL policy had the below : Access to Seckv : path "sec*" { capabilities = ["create", "read", "update", "delete", "list", "patch"] }

and I added : path "sec_kv/data/*" { capabilities = [ "read" ] } path "sec_kv/data/jks/mq/sit" { capabilities = [ "read" ] }

which gave me the exception - `2024-09-10T13:26:55.185+10:00 INFO 857641 --- [message-router] [main] o.s.v.c.e.LeaseAwareVaultPropertySource : Vault location [sec_kv/data/jks/mq/sit] not resolvable: Not found

2024-09-10T13:26:55.185+10:00 INFO 857641 --- [message-router] [main] o.s.v.c.e.LeaseAwareVaultPropertySource : Vault location [sec_kv/data/jks/mq] not resolvable: Not found

2024-09-10T13:26:55.185+10:00 INFO 857641 --- [message-router] [main] o.s.v.c.e.LeaseAwareVaultPropertySource : Vault location [sec_kv/jks/mq] not resolvable: Not found

2024-09-10T13:26:55.611+10:00 WARN 857641 --- [message-router] [main] s.c.a.AnnotationConfigApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanDefinitionStoreException: I/O failure while processing configuration class [org.springframework.cloud.vault.config.VaultHealthIndicatorConfiguration] . . Caused by: java.io.FileNotFoundException: class path resource [org/springframework/boot/actuate/autoconfigure/health/CompositeHealthContributorConfiguration.class] cannot be opened because it does not exist`

I updated :springbootVersion=3.3.3 and below to "org.springframework.cloud:spring-cloud-starter-vault-config:4.1.3", "org.springframework.cloud:spring-cloud-vault-config-databases:3.1.2",

I then get:

VaultTemplate created successfully from URIhttps://<host>:8200

2024-09-10T16:24:54.670+10:00  INFO 1439072 --- [message-router] [main] au.com.nab.service.VaultServiceAzure     : Trying to read from Vault

2024-09-10T16:24:54.693+10:00  INFO 1439072 --- [message-router] [main] au.com.nab.service.VaultServiceAzure     : **MessageI/O error on GET request for "https://<host>:8200/v1/sec_kv/data/jks/mq/sit":** java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext)

2024-09-10T16:24:54.694+10:00  INFO 1439072 --- [message-router] [main] au.com.nab.service.VaultServiceAzure     : java.net.SocketException: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext

[AnujaB219]

I also tried removing KV config from the properties, to just test with database: spring: config: activate: on-profile: sit import: vault:// cloud: vault: azure-msi: azure-path: az/ role: clsappuser namespace: ns_nonprod uri: https://:8200/ enabled: true database: enabled: true role: backend: db/oracle/dev username-property: spring.datasource.username password-property: spring.datasource.password static-role: true

Which leads to this:

Sep 10 17:03:48 java[1551615]: org.springframework.vault.VaultException: Status 403 Forbidden [secret/cls-message-router/sit]: 1 error occurred:

Sep 10 17:03:48 java[1551615]: * permission denied

Sep 10 17:03:48 java[1551615]: at org.springframework.vault.client.VaultResponses.buildException(VaultResponses.java:84)

Sep 10 17:03:48 java[1551615]: at org.springframework.vault.core.VaultTemplate.lambda$doRead$5(VaultTemplate.java:472)

Sep 10 17:03:48 java[1551615]: at org.springframework.vault.core.VaultTemplate.doWithSession(VaultTemplate.java:451)

Sep 10 17:03:48 java[1551615]: at org.springframework.vault.core.VaultTemplate.doRead(VaultTemplate.java:461)

Sep 10 17:03:48 java[1551615]: at org.springframework.vault.core.VaultTemplate.read(VaultTemplate.java:356)

Whereas the DB engine is at db/oracle/dev and I want to read details for static role