"Error: Could not load the default credentials" after upgrading to 8.0.0+

[deremer]

Yes, I think I am in the right place. I've isolated our issue to changing the version of this package in our package.json.

[REQUIRED] Step 2: Describe your environment

• Operating System version: Deployed Firebase Functions on Node 8 runtime.
• Firebase SDK version: this is about v8.0.0+ of this package
• Firebase Product: Firestore
• Node.js version: 8.15.0
• NPM version: Whatever is on GCP for Node 8

[REQUIRED] Step 3: Describe the problem

We recently updated to the latest version of firebase-admin. Previously we were on ~7.3.0. Since upgrade, our functions now always return

Error: Could not load the default credentials. Browse to https://cloud.google.com/docs/authentication/getting-started for more information. at GoogleAuth.getApplicationDefaultAsync (/srv/node_modules/google-gax/node_modules/google-auth-library/build/src/auth/googleauth.js:159:19) at <anonymous> at process._tickDomainCallback (internal/process/next_tick.js:229:7)

If we revert to v7.4.0 this no longer occurs.

However, if we go to v8.0.0 or later, our functions return this error.

Nothing else has changed.

At the top of our index file we do admin.initializeApp(); as we have always done.

I'd really like to avoid specifically setting the credential from a service account json as then we have to manage multiple json files for our multiple environments and have them present on local machine at deploy time.

Steps to reproduce:

This is happening on a variety of cloud functions. But the primary one we are researching is as follows...

Triggered by an .onCreate to a database that is not the primary default database. The function is invoked correctly and we can print to logs at the top of the function. We then hit a code block like...

const data = await admin
      .firestore()
      .collection(col_name)
      .doc(id)
      .get();

And that is where the error is thrown.

Relevant Code:

What's odd is that we attempted to create a new repo that reproduced this issue and we could not repro. However, it is very clearly a problem in our existing codebase, which has never had problems before.

It's very reproducible for us — simple update to 8.0.0 or later and our functions can no longer get the default credentials.

[google-oss-bot]

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

[deremer]

Updated comment b/c we narrowed it and realized it's a query of firestore causing the issue.

[hiranya911]

Hi @schmidt-sebastian. Any issues with ADC discovery/google-auth in the latest firestore libs?

This seems to be the relevant bit of code from google-auth:

    150         // Determine if we're running on GCE.
    151         let isGCE;
    152         try {
    153             isGCE = await this._checkIsGCE();
    154         }
    155         catch (e) {
    156             e.message = `Unexpected error determining execution environment: ${e.message}`;
    157             throw e;
    158         }
    159         if (!isGCE) {
    160             // We failed to find the default credentials. Bail out with an error.
    161             throw new Error('Could not load the default credentials. Browse to https://cloud.google.com/docs/authentication/getting-started for more information.');
    162         }

[hiranya911]

@deremer I just realized that you're trying to upgrade to v8 from v7. I originally thought you were just trying to upgrade to the latest version that went out this week.

v8 series has already been in heavy use for nearly 7 months. So I'm inclined to think this has something to do with your specific project/deployment. Possibly some conflict with your other dependencies? What does your package.json file look like?

[deremer]

Hmmm... I probably shouldn't post the full package.json here give how often npm audit shows red. Here are our GCP related packages...

"@google-cloud/bigquery": "~4.5.0",
"@google-cloud/logging": "~5.3.0",
"@google-cloud/pubsub": "~0.20.0",
"@google-cloud/storage": "~4.1.3",
"firebase-admin": "~7.4.0",
"firebase-functions": "~3.3.0",
"googleapis": "~46.0.0", (only used for Play Store IAP receipts)

"engines": {
    "node": "8"
  }

I'd be happy to share more details over email... if you go to the link in my profile and use the contact form on our website.

This project was started back in the summer of 2017 and there's quite a bit of history to it.

[hiranya911]

Doesn't reproduce to me. Tried with a simple function that has your dependencies plus a recent version of Admin SDK.

Package

{
  "name": "functions",
  "description": "Cloud Functions for Firebase",
  "scripts": {
    "lint": "eslint .",
    "serve": "firebase serve --only functions",
    "shell": "firebase functions:shell",
    "start": "npm run shell",
    "deploy": "firebase deploy --only functions",
    "logs": "firebase functions:log"
  },
  "engines": {
    "node": "8"
  },
  "dependencies": {
    "firebase-admin": "^8.6.0",
    "@google-cloud/bigquery": "~4.5.0",
    "@google-cloud/logging": "~5.3.0",
    "@google-cloud/pubsub": "~0.20.0",
    "@google-cloud/storage": "~4.1.3",
    "firebase-functions": "~3.3.0",
    "googleapis": "~46.0.0"
  },
  "devDependencies": {
    "eslint": "^5.12.0",
    "eslint-plugin-promise": "^4.0.1",
    "firebase-functions-test": "^0.1.6"
  },
  "private": true
}

index.js

const functions = require('firebase-functions');
const admin = require('firebase-admin');

admin.initializeApp();

exports.exampleDatabaseTrigger = functions.database.ref('/messages/{pushId}')
  .onCreate(async (snapshot, context) => {
    const snap = await admin.firestore().collection('messages').doc('test').get();
    console.log(snap.data());
  });

Log output from the deployed function:

3:27:11.825 PM | outlined_flag | exampleDatabaseTrigger | Function execution started |  
-- | -- | -- | -- | --
3:27:15.423 PM | info | exampleDatabaseTrigger | { name: 'Hiranya' }
-- | -- | -- | --
3:27:15.428 PM | outlined_flag | exampleDatabaseTrigger | Function execution took 3603 ms, finished with status: 'ok'
-- | -- | -- | --

[hiranya911]

Unless you can provide us a minimum repro there's not much we can do about this at the moment. I still highly doubt that this has anything to do with the SDK at all, given that the v8 series has been in heavy production use for a long time. Only thing I can suggest at this point is to contact support with more information -- ideally a scaled down version of your implementation which reproduces the issue.

[deremer]

I tried to recreate it too with a new project, but also couldn't get it to repro. Yet it's definitely happening with our deployed functions. So I'm at a loss how to resolve.

I think it must have something to do with Firestore because that function does RTDB queries with no issues, but then it hits the Firestore query and fails.

[deremer]

Unless you can provide us a minimum repro there's not much we can do about this at the moment. I still highly doubt that this has anything to do with the SDK at all, given that the v8 series has been in heavy production use for a long time. Only thing I can suggest at this point is to contact support with more information -- ideally a scaled down version of your implementation which reproduces the issue.

Fair enough. I suppose we can just explicitly reference a service account JSON too and see if that resolves it, even though I'd prefer to not have to do that

[hiranya911]

Error is coming from the google-auth-library, which is a dependency for Firestore. That explains why only the Firestore call is failing, but it doesn't explain how. Do you at least have a package-lock file you can share? I'd recommend including that in your correspondence with support.

[deremer]

Ok I'll take it to support... you've been more than helpful given the vague nature of the problem.

[deremer]

I can share a package-lock but not publicly b/c this is a project for a client

[hiranya911]

That's understandable. Support will provide your a means to share any artifacts and other sensitive information privately and securely.

[deremer]

Random thought but this project pre-dates Firestore. Is it possible that the default service account is somehow messed up from an IAM perspective?

[hiranya911]

I won't rule it out.

However, having looked at the code for google-auth-library and gcp-metadata packages, it seems that the code is failing to communicate with the local metadata server: https://github.com/googleapis/gcp-metadata/blob/master/src/index.ts#L173

This could be because either your instance is messed up, or because your deployment pulled in some old/buggy versions of those dependencies. In my test function I see that it has pulled in the latest versions of these packages. But I also see some old versions in the dependency tree:

$ npm ls gcp-metadata

├─┬ @google-cloud/logging@5.3.1
│ ├── gcp-metadata@3.3.0 
│ └─┬ google-auth-library@5.7.0
│   └── gcp-metadata@3.3.0  deduped
└─┬ @google-cloud/pubsub@0.20.1
  ├─┬ google-auth-library@2.0.2
  │ └── gcp-metadata@0.7.0 
  └─┬ google-gax@0.19.0
    └─┬ google-auth-library@1.6.1
      └── gcp-metadata@0.6.3

[deremer]

Interesting! We’ve delayed upgrading the pubsub function because of some breaking changes. That one is pretty old. I wonder...

[deremer]

Following up... I updated logging and pubsub to the latest and so far we have not had this issue pop-up!

So, with limited data to this point, I'm hopeful your hypothesis is correct that the old versions in the dependency tree were causing this. I'll report back with a thumbs up or down after more testing. Thank you so much for the help!

[schmidt-sebastian]

@deremer Thank you for the update. I am glad this got resolved!