[JS] ACCESS_TOKEN_EXPIRED When Calling Embedder

[dongyangli1226]

Describe the bug I have a nodeJS application deployed on Google Cloud Functions. My application calls the embed function to calculate embeddings for given text (https://github.com/firebase/genkit/blob/5ed51299d96d32da2ccd86cb8d4f801e44edb64a/js/ai/src/embedder.ts#L105).

After running the application for sometime, the embed function will fail due to an error: DEFAULT 2024-07-21T19:58:40.939292Z ' "code": 401,\n' + DEFAULT 2024-07-21T19:58:40.939295Z ' "message": "Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.",\n' + DEFAULT 2024-07-21T19:58:40.939298Z ' "status": "UNAUTHENTICATED",\n' + DEFAULT 2024-07-21T19:58:40.939301Z ' "details": [\n' + DEFAULT 2024-07-21T19:58:40.939304Z ' {\n' + DEFAULT 2024-07-21T19:58:40.939309Z ' "@type": "type.googleapis.com/google.rpc.ErrorInfo",\n' + DEFAULT 2024-07-21T19:58:40.939312Z ' "reason": "ACCESS_TOKEN_EXPIRED",\n' + DEFAULT 2024-07-21T19:58:40.939315Z ' "domain": "googleapis.com",\n' + DEFAULT 2024-07-21T19:58:40.939318Z ' "metadata": {\n' + DEFAULT 2024-07-21T19:58:40.939322Z ' "method": "google.cloud.aiplatform.v1.PredictionService.Predict",\n' + DEFAULT 2024-07-21T19:58:40.939332Z ' "service": "aiplatform.googleapis.com"\n' +

Important note: this happened after 50-60 mins ish running the application from my last testing. Not sure if this is related? https://github.com/firebase/genkit/blob/main/js/plugins/vertexai/src/predict.ts#L34

To Reproduce Run the embed function in a google cloud function environment.

Expected behavior I was expecting that the token would be automatically refreshed when expired.

Screenshots If applicable, add screenshots to help explain your problem.

** Node version v20.5.0

Additional context Here's how I setup genkit:

export const config = configureGenkit({
  plugins: [
    firebase({
      projectId: envConfig.projectId,
    }),
    vertexAI({
      projectId: envConfig.projectId,
      location: "us-central1",
    }),
    dotprompt(),
  ],
  flowStateStore: "firebase",
  traceStore: "firebase",
  enableTracingAndMetrics: true,
  logLevel: "warn",
});

Here's how I use embed:

async function calculateEmbedding(content: string) {
    const embedding = await embed({
      embedder: textEmbeddingGecko,
      content,
    });

    return embedding;
}

[tekkeon]

We have a working theory we're testing right now on why this might be happening.

In the following code, GenKit's Vertex AI plugin caches the access token from GoogleAuth.getAccessToken for 50 minutes: https://github.com/firebase/genkit/blob/main/js/plugins/vertexai/src/predict.ts#L41-L53

Auth tokens live for 60 minutes by default, so it would make sense to cache it to reduce latency and load by not unnecessarily refetching the token. However, I believe this is leading to a cache coherency problem, because google-auth-library appears to already handle caching internally in getAccessToken. The description for the return value for getAccessToken is:

@return A promise that resolves with the current GCP access token response. If the current credential is expired, a new one is retrieved.

There is also corresponding logic for checking whether the token is about to expire, and using a cached version if it is not about to expire. This would indicate that if the token is not expired, then getAccessToken will not fetch a new one, and instead retrieve its own cached token.

If this theory is correct, then at the 50 minute+ mark when a new request comes in, getAccessToken will be called, and the token that is about to expire in 10 minutes will be cached for another 50 minutes. Thus, after 60 minutes from when the token was first retrieved, it will become expired but will still be cached and continue to be used for another 40 minutes. This would also explain our issue as it typically happens around the one hour mark.

The fix would be to simply not cache the token in the Vertex AI plugin at all and delegate the caching to google-auth-library. We're testing this out right now and will update if this works. Happy to submit a PR as well if it works out for us.

[dongyangli1226]

We just completed testing by removing the 50 mins cache TTL locally in the node_modules library (https://github.com/firebase/genkit/blob/main/js/plugins/vertexai/src/predict.ts#L41-L53). The auth issue didn't happen again after more than 1 hour from when the application first obtained an auth token. This further solidified our theory above. Let us know if the team has other perspectives.

[dongyangli1226]

Team has a change merged that removed the refresh token logic. Waiting for the next release to test and close this issue.