firebase / firebase-tools

The Firebase Command Line Tools
MIT License
4.02k stars 938 forks source link

Emulator Functions Regression - Initial Function Requests Time Out #6765

Closed dereekb closed 8 months ago

dereekb commented 9 months ago

[REQUIRED] Environment info

firebase-tools:13.2.1

Platform:macOS

[REQUIRED] Test case

I recently updated a number of firebase dependencies on my library and noticed that hitting the emulator's Firebase functions took a long time and would often time out and be left with an internal error.

I realized that if I turned my wifi/internet connection off that they would run quickly again. I kept noticing the following line:

⚠  External network resource requested!
   - URL: "http://169.254.169.254/computeMetadata/v1/universe/universe_domain"

This is followed by the function timing out:

i  Your function timed out after ~60s. To configure this timeout, see
      https://firebase.google.com/docs/functions/manage-functions#set_timeout_and_memory_allocation.
i  Request to function failed: Error: socket hang up

I dug around and found out that this request was being made through gcp-metadata and timing out since it can't reach the specified IP which I'm guessing is an internal IP to Google Cloud.

I run the emulators in a Docker container and think that the environment is triggering some part of gcp-metadata or the firebase emulators to believe it is running on a Google Cloud instance and is timing out. I didn't notice this behavior while on firebase-tools version 11.17.0.

I checked the versions of gcp-metadata:

dereekb@dbMBP dbcomponents % npm ls gcp-metadata google-auth-library 
@dereekb/dbx-components@10.0.17 /Users/dereekb/development/git/dbcomponents
├─┬ @google-cloud/firestore@7.3.0
│ └─┬ google-gax@4.3.0
│   └─┬ google-auth-library@9.6.3
│     └── gcp-metadata@6.1.0
├─┬ @google-cloud/pubsub@4.2.0
│ └─┬ google-auth-library@9.6.3
│   └── gcp-metadata@6.1.0
├─┬ firebase-admin@12.0.0
│ └─┬ @google-cloud/storage@7.7.0
│   └─┬ google-auth-library@9.6.3
│     └── gcp-metadata@6.1.0
├─┬ firebase-tools@13.2.1                            <---- imports older dependencies
│ ├─┬ @google-cloud/pubsub@3.7.5
│ │ ├─┬ google-auth-library@8.9.0
│ │ │ └── gcp-metadata@5.3.0
│ │ └─┬ google-gax@3.6.1
│ │   └─┬ google-auth-library@8.9.0
│ │     └── gcp-metadata@5.3.0
│ └─┬ google-auth-library@7.14.1
│   └── gcp-metadata@4.3.1          <---- version 4.3.1
├── gcp-metadata@5.3.0
└─┬ google-auth-library@8.9.0
  └── gcp-metadata@5.3.0 deduped

For now as a work around I am using the environment variable GCE_METADATA_HOST to set the host to 0.0.0.0 for the docker container so that the request fails immediately.

⚠  External network resource requested!
   - URL: "http://0.0.0.0/computeMetadata/v1/universe/universe_domain"
 - Be careful, this may be a production service.

In the newer versions of gcp-metadata (^5.3.0) I'd be able to use METADATA_SERVER_DETECTION=none.

[REQUIRED] Steps to reproduce

You can run the project at https://github.com/dereekb/dbx-components on version v10.0.17 (version v10.0.18 will have the workaround added to the Dockerfile). Run both the server ./serve-server.sh and web client ./serve-web.sh and go to http://localhost:9010/demo/app/profile/view. Change the profile info to kick off a firebase functions request.

[REQUIRED] Expected behavior

Emulator functions should not attempt to connect to or time out while trying to reach "http://169.254.169.254/computeMetadata/v1/universe/universe_domain".

[REQUIRED] Actual behavior

The first emulator function invocation attempts to reach 169.254.169.254 and times out after 60 seconds, causing the initial invocation to return an internal error.

naoina commented 9 months ago

I encountered the same issue and probably it is caused by this changes of google-gax v4.3.0. I backed to use google-gax v4.2.1, then it works well even if firebase-tools v13.2.1.

joehan commented 9 months ago

Thanks for reporting this! This call is to the compute metadata server, which will only ever work on cloud compute instances. Ideally, google-gax would smoothly handle uses outside of cloud, so I've opened https://github.com/googleapis/gax-nodejs/issues/1566

inlined commented 8 months ago

A fix will hopefully come out in next week's CLI release. In the meantime you can work around the issue by putting METADATA_SERVER_DETECTION=none in $YOUR_FUNCTIONS_DIRECTORY/.env.local

inlined commented 8 months ago

Closing as the fix is in next week's release.

apeiniger commented 7 months ago

I'm seeing the same behavior in the newest version (13.6.1):

image

Tried to fix it via the env variable mentioned above but it doesn't work.

Any ideas what could cause this?

Akronae commented 7 months ago

Same issue with firebase-tools@13.0.3 on Windows WSL. However it's not happening on MacOS. image

Akronae commented 7 months ago

I forgot to npx firebase login on my machine. This solved the error.
Errors messages should really be improved.

EnduringBeta commented 6 months ago

A fix will hopefully come out in next week's CLI release. In the meantime you can work around the issue by putting METADATA_SERVER_DETECTION=none in $YOUR_FUNCTIONS_DIRECTORY/.env.local

The timeout this issue refers to is still happening to me in GitHub Actions, and this .env.local suggestion did not fix it.

Edit: This issue does not occur if I'm logged in via firebase login. The error does occur if I'm logged out (using FIREBASE_TOKEN), like in a CI environment.

Even though METADATA_SERVER_DETECTION="none" should be valid, and a search in my repo shows 6.1.0, it doesn't work for me. I've decided to go with OP's suggestion of GCE_METADATA_HOST="0.0.0.0".

Maybe my repo's 6.1.0 version doesn't matter since firebase-tools is global.

This is withfirebase-tools 13.8.3.

glorat commented 4 months ago

Just to note I have also run into this issue if I'm using a local development environment with no GCP (firebase nor application default) credentials and am running the emulator under a demo project. The GCE_METADATA_HOST="0.0.0.0" trick also works for me.

I'm on latest firebase-tools of 13.12.0

Since I have this reproduced on my local laptop, ping me if further diagnostics are needed to figure out what's up

james-dellow commented 3 months ago

I'm encountering this on firebase-tools 13.15.0 running on Windows 11. Neither METADATA_SERVER_DETECTION="none" or GCE_METADATA_HOST="0.0.0.0" worked. I'm only using functions, firestore, hosting in the emulator.

EnduringBeta commented 3 months ago

@inlined Should this issue be reopened? I and several others are reporting that we're still dealing with the same behavior.

afmeirelles commented 3 months ago

@inlined Should this issue be reopened? I and several others are reporting that we're still dealing with the same behavior.

I do think so. In my case, I'm running the emulator inside a docker container that has no credentials as well, so even with the ENV VARs set, I get the error 'GaxiosError: request to http://0.0.0.0/computeMetadata/v1/universe/universe-domain failed, reason: connect ECONNREFUSED 0.0.0.0:80\'

james-dellow commented 2 months ago

The METADATA_SERVER_DETECTION override isn't working for me. I've resolved this temporarily:

I'll open a separate issue for this.

jirawatee commented 1 month ago

I also found this issue with Firebase CLI v13.18.0 and I fixed it by adding "GCE_METADATA_HOST=localhost:8080" to .env.local

Hope this help :)

EnduringBeta commented 1 month ago

@joehan I suggest this issue be reopened. It may have a workaround, but people are still encountering it.

EduardJoy commented 1 month ago

Greetings!

This issue is still happening as of this moment.

image

The proposed fix, to set GCE_METADATA_HOST="0.0.0.0" causes an error. See attached.

image
rahul-sekhar commented 4 weeks ago

The METADATA_SERVER_DETECTION override didn't work for me either. I don't see this env variable being checked in the codepath in gcp-metadata 6.1.0.

I'm instead forcing the GCP residency check to false, and this seems to be working for me:

import { setGCPResidency } from "gcp-metadata";

if (process.env.FUNCTIONS_EMULATOR !== undefined) {
  console.log("Running in emulator mode - forcing GCP residency to false");
  setGCPResidency(false);
}
rahul-sekhar commented 4 weeks ago

I found the root cause on my end and no longer need to use the fix above. We were using FIREBASE_TOKEN to run the emulators on our CI, which is now deprecated. If I switch to using a service account and GOOGLE_APPLICATION_CREDENTIALS instead, I don't see the issue any more.