firebase / firebase-admin-node

Firebase Admin Node.js SDK
https://firebase.google.com/docs/admin/setup
Apache License 2.0
1.63k stars 371 forks source link

"Error: Could not load the default credentials" after upgrading to 8.0.0+ #738

Closed deremer closed 4 years ago

deremer commented 4 years ago

Yes, I think I am in the right place. I've isolated our issue to changing the version of this package in our package.json.

[REQUIRED] Step 2: Describe your environment

[REQUIRED] Step 3: Describe the problem

We recently updated to the latest version of firebase-admin. Previously we were on ~7.3.0. Since upgrade, our functions now always return

Error: Could not load the default credentials. Browse to https://cloud.google.com/docs/authentication/getting-started for more information. at GoogleAuth.getApplicationDefaultAsync (/srv/node_modules/google-gax/node_modules/google-auth-library/build/src/auth/googleauth.js:159:19) at <anonymous> at process._tickDomainCallback (internal/process/next_tick.js:229:7)

If we revert to v7.4.0 this no longer occurs.

However, if we go to v8.0.0 or later, our functions return this error.

Nothing else has changed.

At the top of our index file we do admin.initializeApp(); as we have always done.

I'd really like to avoid specifically setting the credential from a service account json as then we have to manage multiple json files for our multiple environments and have them present on local machine at deploy time.

Steps to reproduce:

This is happening on a variety of cloud functions. But the primary one we are researching is as follows...

Triggered by an .onCreate to a database that is not the primary default database. The function is invoked correctly and we can print to logs at the top of the function. We then hit a code block like...

const data = await admin
      .firestore()
      .collection(col_name)
      .doc(id)
      .get();

And that is where the error is thrown.

Relevant Code:

What's odd is that we attempted to create a new repo that reproduced this issue and we could not repro. However, it is very clearly a problem in our existing codebase, which has never had problems before.

It's very reproducible for us — simple update to 8.0.0 or later and our functions can no longer get the default credentials.

google-oss-bot commented 4 years ago

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

deremer commented 4 years ago

Updated comment b/c we narrowed it and realized it's a query of firestore causing the issue.

hiranya911 commented 4 years ago

Hi @schmidt-sebastian. Any issues with ADC discovery/google-auth in the latest firestore libs?

This seems to be the relevant bit of code from google-auth:

    150         // Determine if we're running on GCE.
    151         let isGCE;
    152         try {
    153             isGCE = await this._checkIsGCE();
    154         }
    155         catch (e) {
    156             e.message = `Unexpected error determining execution environment: ${e.message}`;
    157             throw e;
    158         }
    159         if (!isGCE) {
    160             // We failed to find the default credentials. Bail out with an error.
    161             throw new Error('Could not load the default credentials. Browse to https://cloud.google.com/docs/authentication/getting-started for more information.');
    162         }
hiranya911 commented 4 years ago

@deremer I just realized that you're trying to upgrade to v8 from v7. I originally thought you were just trying to upgrade to the latest version that went out this week.

v8 series has already been in heavy use for nearly 7 months. So I'm inclined to think this has something to do with your specific project/deployment. Possibly some conflict with your other dependencies? What does your package.json file look like?

deremer commented 4 years ago

Hmmm... I probably shouldn't post the full package.json here give how often npm audit shows red. Here are our GCP related packages...

"@google-cloud/bigquery": "~4.5.0",
"@google-cloud/logging": "~5.3.0",
"@google-cloud/pubsub": "~0.20.0",
"@google-cloud/storage": "~4.1.3",
"firebase-admin": "~7.4.0",
"firebase-functions": "~3.3.0",
"googleapis": "~46.0.0", (only used for Play Store IAP receipts)
"engines": {
    "node": "8"
  }

I'd be happy to share more details over email... if you go to the link in my profile and use the contact form on our website.

This project was started back in the summer of 2017 and there's quite a bit of history to it.

hiranya911 commented 4 years ago

Doesn't reproduce to me. Tried with a simple function that has your dependencies plus a recent version of Admin SDK.

Package

{
  "name": "functions",
  "description": "Cloud Functions for Firebase",
  "scripts": {
    "lint": "eslint .",
    "serve": "firebase serve --only functions",
    "shell": "firebase functions:shell",
    "start": "npm run shell",
    "deploy": "firebase deploy --only functions",
    "logs": "firebase functions:log"
  },
  "engines": {
    "node": "8"
  },
  "dependencies": {
    "firebase-admin": "^8.6.0",
    "@google-cloud/bigquery": "~4.5.0",
    "@google-cloud/logging": "~5.3.0",
    "@google-cloud/pubsub": "~0.20.0",
    "@google-cloud/storage": "~4.1.3",
    "firebase-functions": "~3.3.0",
    "googleapis": "~46.0.0"
  },
  "devDependencies": {
    "eslint": "^5.12.0",
    "eslint-plugin-promise": "^4.0.1",
    "firebase-functions-test": "^0.1.6"
  },
  "private": true
}

index.js

const functions = require('firebase-functions');
const admin = require('firebase-admin');

admin.initializeApp();

exports.exampleDatabaseTrigger = functions.database.ref('/messages/{pushId}')
  .onCreate(async (snapshot, context) => {
    const snap = await admin.firestore().collection('messages').doc('test').get();
    console.log(snap.data());
  });

Log output from the deployed function:

3:27:11.825 PM | outlined_flag | exampleDatabaseTrigger | Function execution started |  
-- | -- | -- | -- | --
3:27:15.423 PM | info | exampleDatabaseTrigger | { name: 'Hiranya' }
-- | -- | -- | --
3:27:15.428 PM | outlined_flag | exampleDatabaseTrigger | Function execution took 3603 ms, finished with status: 'ok'
-- | -- | -- | --
hiranya911 commented 4 years ago

Unless you can provide us a minimum repro there's not much we can do about this at the moment. I still highly doubt that this has anything to do with the SDK at all, given that the v8 series has been in heavy production use for a long time. Only thing I can suggest at this point is to contact support with more information -- ideally a scaled down version of your implementation which reproduces the issue.

deremer commented 4 years ago

I tried to recreate it too with a new project, but also couldn't get it to repro. Yet it's definitely happening with our deployed functions. So I'm at a loss how to resolve.

I think it must have something to do with Firestore because that function does RTDB queries with no issues, but then it hits the Firestore query and fails.

deremer commented 4 years ago

Unless you can provide us a minimum repro there's not much we can do about this at the moment. I still highly doubt that this has anything to do with the SDK at all, given that the v8 series has been in heavy production use for a long time. Only thing I can suggest at this point is to contact support with more information -- ideally a scaled down version of your implementation which reproduces the issue.

Fair enough. I suppose we can just explicitly reference a service account JSON too and see if that resolves it, even though I'd prefer to not have to do that

hiranya911 commented 4 years ago

Error is coming from the google-auth-library, which is a dependency for Firestore. That explains why only the Firestore call is failing, but it doesn't explain how. Do you at least have a package-lock file you can share? I'd recommend including that in your correspondence with support.

deremer commented 4 years ago

Ok I'll take it to support... you've been more than helpful given the vague nature of the problem.

deremer commented 4 years ago

I can share a package-lock but not publicly b/c this is a project for a client

hiranya911 commented 4 years ago

That's understandable. Support will provide your a means to share any artifacts and other sensitive information privately and securely.

deremer commented 4 years ago

Random thought but this project pre-dates Firestore. Is it possible that the default service account is somehow messed up from an IAM perspective?

hiranya911 commented 4 years ago

I won't rule it out.

However, having looked at the code for google-auth-library and gcp-metadata packages, it seems that the code is failing to communicate with the local metadata server: https://github.com/googleapis/gcp-metadata/blob/master/src/index.ts#L173

This could be because either your instance is messed up, or because your deployment pulled in some old/buggy versions of those dependencies. In my test function I see that it has pulled in the latest versions of these packages. But I also see some old versions in the dependency tree:

$ npm ls gcp-metadata

├─┬ @google-cloud/logging@5.3.1
│ ├── gcp-metadata@3.3.0 
│ └─┬ google-auth-library@5.7.0
│   └── gcp-metadata@3.3.0  deduped
└─┬ @google-cloud/pubsub@0.20.1
  ├─┬ google-auth-library@2.0.2
  │ └── gcp-metadata@0.7.0 
  └─┬ google-gax@0.19.0
    └─┬ google-auth-library@1.6.1
      └── gcp-metadata@0.6.3
deremer commented 4 years ago

Interesting! We’ve delayed upgrading the pubsub function because of some breaking changes. That one is pretty old. I wonder...

deremer commented 4 years ago

Following up... I updated logging and pubsub to the latest and so far we have not had this issue pop-up!

So, with limited data to this point, I'm hopeful your hypothesis is correct that the old versions in the dependency tree were causing this. I'll report back with a thumbs up or down after more testing. Thank you so much for the help!

schmidt-sebastian commented 4 years ago

@deremer Thank you for the update. I am glad this got resolved!