Vault: api error: status code 400: error configuring token validator: keyset configuration error

[ricardosilva86]

I am having issues with Gitlab CI integration in my Vault instance. On May 14th, Gitlab deprecated the CI_JOB_JWT and on May 15th I started to face the following error while my CI jobs try to authenticate with the JWT token signed by gitlab.com:

Using "vault" secret resolver...
ERROR: Job failed (system failure): resolving secrets: 
initializing Vault service: preparing authenticated client: 
authenticating Vault client: 
writing to Vault: api error: 
status code 400: 
error configuring token validator: 
keyset configuration error: 
Get "https://gitlab.com/.well-known/openid-configuration": dial tcp 172.65.251.78:443: i/o timeout

My vault gitlab role looks like this:

vault read auth/<jwt auth path>/config

Key                    Value             
bound_issuer           https://gitlab.com
default_role                             
jwks_ca_pem                              
jwks_url                                 
jwt_supported_algs     []                
jwt_validation_pubkeys []                
namespace_in_state     true              
oidc_client_id                           
oidc_discovery_ca_pem                    
oidc_discovery_url     https://gitlab.com
oidc_response_mode                       
oidc_response_types    []                
provider_config        {}      

Here is my CI configuration:

variables:
  VAULT_SERVER_URL: "https://<vault_url>"
  VAULT_AUTH_PATH: "<jwt auth path>"
[...]
tests:
  id_tokens:
    VAULT_ID_TOKEN:
      aud: https://gitlab.com
  secrets:
    SECRET:
      vault: <secret_path>@<mount_point>
      token: $VAULT_ID_TOKEN
      file: false

Gitlab is recommending id_tokens since a long ago as the desired way of authenticating. IMPORTANT: I am not using the CI_JOB_JWT anywhere.

Based on the error, is it possible to actually figure out what is wrong with the JWT auth plugin?

[yuriylesyuk]

Is your topology: gitlab.com and a private instance of vault?

Your immediate problem is flagged here:

Get "https://gitlab.com/.well-known/openid-configuration": dial tcp 172.65.251.78:443: i/o timeout

As a part of verifying signature of the JWT token that gitlab secret resolver sends to the auth jwt engine, it contacts gitlab.com and tries to fetch public keys. Because gitlab and vault implement OIDC, the public keys (jwks) base path is: .well-known/openid-configuration.

As your vault server resolved gitlab.com correctly into its public IP address (172.65.251.78), this part works OK, but some egress firewall rule prevents access to the gitlab.com:443.

To troubleshoot, you can log into your vault server and try to execute following curl command:

curl https://gitlab.com/.well-known/openid-configuration