Token renewal fails using a cert auth method

[eswets]

Overview of the Issue

We're working on setting up this plugin, and running into the following issue. On every container pull, an attempt is made to renew the (cached) Vault token. This renewal fails as a bad request is made. So, I guess either we'd want to disable this renewal behaviour altogether, and fetch a new token once the current one is expired, or fix the renewal process (and probably also change when the renewal happens. Doing this on every docker pull seems too often?)

Reproduction Steps

Run with a cert auth method, see config below.

config.hcl

vault {
  address = "https://<redacted>/"
}

auto_auth {
  method "cert" {
    type = "cert"
    mount_path = "auth/puppet/cert"
    config = {
      type = "cert"
      client_cert = "/var/lib/puppet/ssl/certs/cert.pem"
      client_key = "/var/lib/puppet/ssl/private_keys/key.pem"
      secrets = {
              "ghcr.io" = "kv/stpst/staging/data/docker/ghcr.io"
            }
    }
  }

  sink "file" {
    config = {
      path = "/srv/docker-credentialhelper/tokens/staging"
    }
  }
}

Log Fragments

2021-12-21T11:13:57.976+0100 [ERROR] error renewing token:
  error=
  | Error making API request.
  | 
  | URL: PUT https://<redacted>/v1/auth/token/renew-self
  | Code: 400. Errors:
  | 
  | * client certificate must be supplied

[alexdulin]

Thanks for reporting the issue! It looks like you are connecting to Vault with TLS enabled but without specifying any of the tls cert options for Vaults ca cert or similar. Unless you have the CA already as part of your system's trusted certs, this could be the issue.

Does your Vault server require client certs by any chance as well?

The token renewal should not be dependent on the original auth method used, as all that is required is the token received from initial authentication.

[eswets]

Thanks for the swift response! The CA is indeed part of the systems trusted certs. We also don't require the client certs.

The thing that's strange is that fetching the initial token works fine. This means that the credential helper can successfully fetch a token from Vault (and fetches the docker credentials successfully as well, subsequently, meaning pulling images from the authenticated repo works fine). It's just that on every pull it seems to attempt this renewal, and error on it.

[alexdulin]

That is very interesting, and I don't know exactly why this would error with the message you are receiving. Our team is going to be starting vacations for the holiday soon and will be out of office, but I think we could take a closer look at this after the new year. Thank you again @eswets for your interest in and trying out the project. I'll follow up here after we find some time to look into this.

@dbellinghoven lets discuss this at some point soon and debug.

[eswets]

Cheers! If there is anything I can assist on to debug, let me know. For now, enjoy the holidays!