mysql-aurora-database-plugin usernames are not unique

[timkra]

Describe the bug I have a StatefulSet that request database credentials using consul-template as a sidecar from an Aurora Cluster. The username generated is not unique, which causes the second replica to not launch, as the username already exists. The username scheme is "v_AUTHBACKEND-NAMESPACE-SERVICEACCOUNT". E.g. "v_k8s-default-ic"

To Reproduce Steps to reproduce the behavior:

1. Create a StatefulSet
2. Use vault as an init-container to get a token.
3. Instruct consul-template to use this token and request database credentials from an Aurora Cluster
4. Look at the generated username

Expected behavior The name gets truncated and an UUID gets appended to the username.

Environment:

• Vault Server Version (retrieve with vault status): 1.6.0
• Vault init-container: vault:1.6.0
• Consul-template Container: consul-template:alpine

Vault init-container configuration file:

exit_after_auth = true
pid_file = "/home/vault/pidfile"

auto_auth {
    method "kubernetes" {
        mount_path = "auth/k8s"
        config = {
            role = "my-role"
        }
    }
    sink "file" {
        config = {
            path = "/etc/secrets/.vault-token"
        }
    }
}

Consul-template configuration file:

vault {
  renew_token = true
  vault_agent_token_file = "/etc/secrets/.vault-token"
  retry {
    backoff = "1s"
  }
}
template {
  destination = "/ct-files/database.conf"
  contents = <<EOH
user      = "{{- with secret "my-database-secret-engine/creds/my-role" }}{{.Data.username}}{{ end }}"
password  = "{{- with secret "my-database-secret-engine/creds/my-role" }}{{.Data.password}}{{ end }}"
}
EOH
}

Additional context When manually requesting credentials with the root token, the generated username is as expected.

[timkra]

I found that the issue is not present when running Vault Server 1.5.4 and consul-template 0.24.0-alpine. In this combination the username is as expceted.

[maksym-iv]

I can confirm that issue exists for Vault v1.6.1. Using userpass auth backend. Username to aurora is always not unique and consists of v_${TOKEN_DISPLAY_NAME}_.

UPD: https://github.com/hashicorp/vault/issues/10515#issuecomment-755260592

[stevenscg]

I believe that I am seeing this same behavior with Nomad v1.0.0 and Vault 1.6.0 and MySQL 5.7.

The error message in the system logs is Error 1396: Operation CREATE USER failed for 'v_token-58173a26'@'%'.

In my case, this is occurring within a single task while rendering a configuration file for the task. Within the template for the file, the "with secret" call is made several times with the same "creds" path. If I limit the calls to the same "creds" path by removing any duplicates, the task starts and becomes healthy as it has for all past versions over many years. Our use case here is several mysql databases that reside on the same host, but use a different configuration name in the application.

This makes me think that something might have changed in the multi-pass evaluation in consul-template and the issue might be more appropriate to report there and/or with nomad.

[stevenscg]

An additional finding is that I get the error when I call any 2 different "creds" paths in the same template. The only way to avoid the error in my setup is to call "with secret" at most 1 time.

With that, I'm back to thinking that the problem could be the truncated UUIDs in the generated max 16-characters username, i.e. v_${TOKEN_DISPLAY_NAME}-XXXXXXXX.

[stevenscg]

@timkra @xmackex The discussion and fixes referenced in https://github.com/hashicorp/vault/issues/10476 could be relevant for this issue. I'm going to rollback to 1.5.x and recheck when 1.7.0 comes out.

[maksym-iv]

UPD: I was mistaken, issue exists with v1.6.0 and mysql-aurora-database-plugin.

After upgrading to 1.6.1 and using mysql-plugin (luckily our Aurora RDS supports usernames up to 32 symbols) issue seems to be fixed. Though have not tested v1.6.1 with mysql-aurora-database-plugin.