search

googleapis / google-auth-library-nodejs

🔑 Google Auth Library for Node.js
Apache License 2.0
1.71k stars 374 forks source link

Workload Identity with AWS & IMDSv2 use expired token #1755

Closed whs closed 6 months ago

whs commented 7 months ago

Thanks for stopping by to let us know something could be better!

PLEASE READ: If you have a support contract with Google, please create an issue in the support console instead of filing on GitHub. This will ensure a timely response.

Environment details

Steps to reproduce

  1. Setup federation with AWS according to https://cloud.google.com/iam/docs/workload-identity-federation-with-other-clouds . The JSON file should have imdsv2_session_token_url.
  2. Observe that API calls to Google (we're testing with service account signJwt) are working normally
  3. Wait about an hour
  4. Observe that API calls are now failing

The error object we receive is:

GaxiosError
    at Gaxios._request (/app/node_modules/gaxios/build/src/gaxios.js:142:23)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async AwsClient.getAwsRoleName (/app/node_modules/google-auth-library/build/src/auth/awsclient.js:217:26)
    at async AwsRequestSigner.getCredentials (/app/node_modules/google-auth-library/build/src/auth/awsclient.js:110:34)
    at async AwsRequestSigner.getRequestOptions (/app/node_modules/google-auth-library/build/src/auth/awsrequestsigner.js:66:40)
    at async AwsClient.retrieveSubjectToken (/app/node_modules/google-auth-library/build/src/auth/awsclient.js:125:25)
    at async AwsClient.refreshAccessTokenAsync (/app/node_modules/google-auth-library/build/src/auth/baseexternalclient.js:288:30)
    at async AwsClient.getAccessToken (/app/node_modules/google-auth-library/build/src/auth/baseexternalclient.js:165:13)
    [scrubbed] {
  config: {
    url: 'http://169.254.169.254/latest/meta-data/iam/security-credentials',
    method: 'GET',
    responseType: 'text',
    headers: {
      'x-aws-ec2-metadata-token': '<private>',
      'User-Agent': 'google-api-nodejs-client/9.6.1',
      'x-goog-api-client': 'gl-node/16.20.2'
    },
    paramsSerializer: [Function: paramsSerializer],
    validateStatus: [Function: validateStatus],
    errorRedactor: [Function: defaultErrorRedactor]
  },
  response: {
    config: {
      url: 'http://169.254.169.254/latest/meta-data/iam/security-credentials',
      method: 'GET',
      responseType: 'text',
      headers: [Object],
      paramsSerializer: [Function: paramsSerializer],
      validateStatus: [Function: validateStatus],
      errorRedactor: [Function: defaultErrorRedactor]
    },
    data: '',
    headers: {
      connection: 'close',
      'content-length': '0',
      'content-type': 'text/plain',
      date: 'Wed, 07 Feb 2024 05:38:23 GMT',
      server: 'EC2ws'
    },
    status: 401,
    statusText: 'Unauthorized',
    request: {
      responseURL: 'http://169.254.169.254/latest/meta-data/iam/security-credentials'
    }
  },
  error: undefined,
  status: 401,
  [Symbol(gaxios-gaxios-error)]: '6.2.0'
}

I think this is because in https://github.com/googleapis/google-auth-library-nodejs/blob/3b19e9cfa0e7ca4ffd97fa0ebd96f065286573dc/src/auth/awsclient.ts#L151-L161 the IMDSv2 Session Token API is only called once when this.awsRequestSigner is null, then is reused for all subsequent requests.

However in getImdsV2SessionToken the x-aws-ec2-metadata-token-ttl-seconds is only configured to be 300s:

https://github.com/googleapis/google-auth-library-nodejs/blob/3b19e9cfa0e7ca4ffd97fa0ebd96f065286573dc/src/auth/awsclient.ts#L242

so after the application run for some times the GCP token expires and refreshing of the token fails because it use expired IMDSv2 token.

Semi-related but I checked how the Go SDK has implemented and it seems to be 1:1 match to this behavior and may also have similar problem https://cs.opensource.google/go/x/oauth2/+/refs/tags/v0.17.0:google/internal/externalaccount/aws.go;l=304