VaultException: Status 403 Forbidden

Describe the bug Spring Cloud Vault Databases in 2.3.5.RELEASE throws an exception every time when I stop the spring application or restart it using dev-tools.

Sample I created a database secrets: database/incidence-mysql with the role incidence-service.

The exception trace

st-incidence-service    | 2021-01-13 19:40:33.162  WARN 52 --- [      Thread-11] LeaseEventPublisher$LoggingErrorListener : [RequestedSecret [path='database/incidence-mysql/creds/incidence-service', mode=RENEW]] Lease [leaseId='database/incidence-mysql/creds/incidence-service/lease was hidden', leaseDuration=PT1H, renewable=true] Status 403 Forbidden: 1 error occurred:
st-incidence-service    |       * permission denied
st-incidence-service    | 
st-incidence-service    | ; nested exception is org.springframework.web.client.HttpClientErrorException$Forbidden: 403 Forbidden: [{"errors":["1 error occurred:\n\t* permission denied\n\n"]}
st-incidence-service    | ]
st-incidence-service    | 
st-incidence-service    | org.springframework.vault.VaultException: Status 403 Forbidden: 1 error occurred:
st-incidence-service    |       * permission denied
st-incidence-service    | 
st-incidence-service    | ; nested exception is org.springframework.web.client.HttpClientErrorException$Forbidden: 403 Forbidden: [{"errors":["1 error occurred:\n\t* permission denied\n\n"]}
st-incidence-service    | ]
st-incidence-service    |       at org.springframework.vault.client.VaultResponses.buildException(VaultResponses.java:63) ~[spring-vault-core-2.2.0.RELEASE.jar:2.2.0.RELEASE]
st-incidence-service    |       at org.springframework.vault.core.VaultTemplate.doWithSession(VaultTemplate.java:391) ~[spring-vault-core-2.2.0.RELEASE.jar:2.2.0.RELEASE]
st-incidence-service    |       at org.springframework.vault.core.lease.SecretLeaseContainer.doRevokeLease(SecretLeaseContainer.java:785) ~[spring-vault-core-2.2.0.RELEASE.jar:2.2.0.RELEASE]
st-incidence-service    |       at org.springframework.vault.core.lease.SecretLeaseContainer.destroy(SecretLeaseContainer.java:503) ~[spring-vault-core-2.2.0.RELEASE.jar:2.2.0.RELEASE]

I'm using AppRole authentication.

# Vault
spring.cloud.vault.uri=uri removed
spring.cloud.vault.authentication=APPROLE
spring.cloud.vault.app-role.role-id=role id removed
spring.cloud.vault.app-role.secret-id=secret id removed
spring.cloud.vault.app-role.app-role=incidence-service
spring.cloud.vault.app-role.app-role-path=approle

# Vault Databases (MySQL)
spring.cloud.vault.mysql.role=incidence-service
spring.cloud.vault.mysql.backend=database/incidence-mysql

With the following ACL policy

# Mount the AppRole auth method
path "sys/auth/approle" {
  capabilities = [ "create", "read", "update", "delete", "sudo" ]
}

# Configure the AppRole auth method
path "sys/auth/approle/*" {
  capabilities = [ "create", "read", "update", "delete" ]
}

# Create and manage roles
path "auth/approle/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# Key-Value
path "secret/incidence-service" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}
path "secret/incidence-service/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# Application
path "secret/application" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}
path "secret/application/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# Database
path "database/incidence-mysql/creds/incidence-service" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}
path "sys/renew/database/incidence-mysql/creds/incidence-service/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

However, when I manually generate the token with the command:

curl --request POST --data '{"role_id":"role-id removed", "secret_id":"secret-id removed"}' host-removed/v1/auth/approle/login

I am able to generate a token:

{"request_id":"value removed","lease_id":"","renewable":false,"lease_duration":0,"data":null,"wrap_info":null,"warnings":null,"auth":{"client_token":"token is here","accessor":"value removed","policies":["default","incidence-service"],"token_policies":["default","incidence-service"],"metadata":{"role_name":"incidence-service"},"lease_duration":600,"renewable":true,"entity_id":"value removed","token_type":"service","orphan":true}}

I consume the token to manually renew the lease, and to see if the error was due to the ACL policy:

curl -H "X-Vault-Token:token goes here" --request PUT host-removed/v1/sys/renew/database/incidence-mysql/creds/incidence-service/lease goes here

But the lease does get renewed, so I'm guessing this issue is related to Spring and not to my ACL policy:

{"request_id":"value removed","lease_id":"database/incidence-mysql/creds/incidence-service/value removed","renewable":true,"lease_duration":3600,"data":null,"wrap_info":null,"warnings":null,"auth":null}

What does the error 403 mean in this case? Am I doing something wrong?

spring-cloud / spring-cloud-vault

VaultException: Status 403 Forbidden #562