Bug Report: Vault approle token is not refreshed after max-ttl is reached

Overview of the Issue

Vitess is unable to request a new Vault (approle) token after the initial token reaches max number of renewals.

On vttablet startup, Vitess is able to request a Vault token using the supplied approle role-id/secret-id combo. The token given has a lease TTL of 2hrs and max-lease TTL of 24h.

Currently, Vitess is unable to determine it has reached max renewal attempts, after which it should relogin against Vault and request a new token.

This currently causes it to not be able fetch the vttablet-credentials at the set polling interval.

Reproduction Steps

Launch any vttablet that uses vault for the credentials-server

      --db-credentials-server=vault
      --db-credentials-vault-addr=https://xxx:8200/
      --db-credentials-vault-path=secrets/vitess/vt-tablet/credentials.json
      --db-credentials-vault-tls-ca=privateca.crt
      --db-credentials-vault-ttl=60s

With VAULT_ROLEID and VAULT_SECRETID env variables passed.

Binary Version

vttablet --version
Version: 16.0.3 (Git revision a93cb55331d63eeb4a0dc6ed1b5f98429c350f04 branch 'heads/v16.0.3') built on Thu Sep 12 20:35:55 UTC 2024 by root@buildkitsandbox using go1.20.5 linux/amd64

Operating System and Environment details

$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

$ uname -sr
Linux 5.15.0-1068-aws

$ uname -m
x86_64

Log Fragments

vttablet log for token renewal fail: https://gist.github.com/flopex/33018e8ba15c7d11a8a8b7f12cfb2a32

vitessio / vitess