firebase / firebase-functions

Firebase SDK for Cloud Functions
https://firebase.google.com/docs/functions/
MIT License
1.03k stars 202 forks source link

[Error] V2 functions deployment fails with error in cloud run container service container fails to start #1131

Closed algoflows closed 2 years ago

algoflows commented 2 years ago

node:

16

firebase-functions:

3.21.2

firebase-tools:

v11.0.1

firebase-admin:

Steps to reproduce

Deploying a single v2 firebase cloud function

// FIREBASE V2 FUNCTION
const { onRequest } = require("firebase-functions/v2/https");

module.exports = onRequest(
  { timeoutSeconds: 1200, region: ["europe-west1"] },
  (_req, res) => {
    res.status(200).send("OK");
  }
);

Expected behavior

Deploy the gen2 firebase cloud function successefully and error free

Actual behavior

Deployment fails, v2 functions are unusable as they simply don't deploy.

Screenshot 2022-06-02 at 01 18 34

Were you able to successfully deploy your functions?

NO

run > firebase deploy --only functions:function-name

result:

Screenshot 2022-06-02 at 01 17 26
❯ firebase deploy --only functions:webhooks-aifi-v2-checkout

=== Deploying to 'cornershop-app'...

i  deploying functions
i  functions: ensuring required API cloudfunctions.googleapis.com is enabled...
i  functions: ensuring required API cloudbuild.googleapis.com is enabled...
✔  functions: required API cloudbuild.googleapis.com is enabled
✔  functions: required API cloudfunctions.googleapis.com is enabled
i  functions: preparing codebase default for deployment
i  functions: preparing functions directory for uploading...
i  functions: packaged /Users/seank/Projects/capgemini/cornershop-app/functions (55.5 MB) for uploading
i  functions: packaged /Users/seank/Projects/capgemini/cornershop-app/functions (55.5 MB) for uploading
i  functions: ensuring required API cloudscheduler.googleapis.com is enabled...
✔  functions: required API cloudscheduler.googleapis.com is enabled
i  functions: ensuring required API artifactregistry.googleapis.com is enabled...
i  functions: ensuring required API run.googleapis.com is enabled...
i  functions: ensuring required API eventarc.googleapis.com is enabled...
i  functions: ensuring required API pubsub.googleapis.com is enabled...
i  functions: ensuring required API storage.googleapis.com is enabled...
✔  functions: required API eventarc.googleapis.com is enabled
✔  functions: required API run.googleapis.com is enabled
✔  functions: required API pubsub.googleapis.com is enabled
✔  functions: required API storage.googleapis.com is enabled
✔  functions: required API artifactregistry.googleapis.com is enabled
i  functions: generating the service identity for pubsub.googleapis.com...
i  functions: generating the service identity for eventarc.googleapis.com...
✔  functions: functions folder uploaded successfully
i  functions: creating Node.js 16 function webhooks-aifi-v2-checkout(europe-west1)...
Could not create or update Cloud Run service webhooks-aifi-v2-checkout, Container Healthcheck failed. Cloud Run error: The user-provided container failed to start and listen on the port defined provided by the PORT=8080 environment variable. Logs for this revision might contain more information.

Logs URL: https://console.cloud.google.com/logs/viewer?project=cornershop-app&resource=cloud_run_revision/service_name/webhooks-aifi-v2-checkout/revision_name/webhooks-aifi-v2-checkout-00001-qur&advancedFilter=resource.type%3D%22cloud_run_revision%22%0Aresource.labels.service_name%3D%22webhooks-aifi-v2-checkout%22%0Aresource.labels.revision_name%3D%22webhooks-aifi-v2-checkout-00001-qur%22
For more troubleshooting guidance, see https://cloud.google.com/run/docs/troubleshooting#container-failed-to-start

Functions deploy had errors with the following functions:
        webhooks-aifi-v2-checkout(europe-west1)
i  functions: cleaning up build files...

Error: There was an error deploying functions
Screenshot 2022-06-02 at 01 21 31
google-oss-bot commented 2 years ago

I found a few problems with this issue:

gustavopch commented 2 years ago

Same issue happened here. I'm using a MacBook Pro M1 and this page says:

Note: If you build your container image on a ARM based machine, then it might not work as expected when used with Cloud Run.

So I guess I'll have to stick with Functions v1 for now.

Source code:

import { onRequest } from 'firebase-functions/v2/https'

export const helloworld = onRequest(
  {
    region: 'us-east1',
  },
  (request, response) => {
    response.status(200).send({ hello: 'world' })
  },
)

Output when deploying:

i  deploying functions
i  functions: ensuring required API cloudfunctions.googleapis.com is enabled...
i  functions: ensuring required API cloudbuild.googleapis.com is enabled...
✔  functions: required API cloudfunctions.googleapis.com is enabled
✔  functions: required API cloudbuild.googleapis.com is enabled
i  functions: ensuring required API artifactregistry.googleapis.com is enabled...
i  functions: ensuring required API run.googleapis.com is enabled...
i  functions: ensuring required API eventarc.googleapis.com is enabled...
i  functions: ensuring required API pubsub.googleapis.com is enabled...
i  functions: ensuring required API storage.googleapis.com is enabled...
✔  functions: required API eventarc.googleapis.com is enabled
✔  functions: required API artifactregistry.googleapis.com is enabled
✔  functions: required API run.googleapis.com is enabled
✔  functions: required API storage.googleapis.com is enabled
✔  functions: required API pubsub.googleapis.com is enabled
i  functions: preparing ./build/functions directory for uploading...
i  functions: packaged ./build/functions (56.65 KB) for uploading
i  functions: packaged ./build/functions (55.54 KB) for uploading
i  functions: ensuring required API cloudscheduler.googleapis.com is enabled...
✔  functions: required API cloudscheduler.googleapis.com is enabled
✔  functions: ./build/functions folder uploaded successfully
i  functions: creating Node.js 16 function helloworld(us-east1)...

Functions deploy had errors with the following functions:
        helloworld(us-east1)
i  functions: cleaning up build files...

Error: There was an error deploying functions
error Command failed with exit code 2.

Google Cloud Logging:

Could not create or update Cloud Run service helloworld, Container Healthcheck failed. Cloud Run error: The user-provided container failed to start and listen on the port defined provided by the PORT=8080 environment variable. Logs for this revision might contain more information.

Versions: firebase-functions: v3.18.1 firebase-tools: v10.2.2

algoflows commented 2 years ago

@gustavopch It's wild considering how many companies and users use firebase functions. Would have thought someone from the team at google would have at least replied by now.

gustavopch commented 2 years ago

@algoflows Yeah, 6 days already. I know v2 is not under global availability and SLA, but still...

algoflows commented 2 years ago

I posted on GCP functions framework issues board a while back there was at least some interaction. This lack of inactivity leaves me feeling like we've bet on the wrong stack here..

Fingers crossed we get an update and fix soon.

trex-quo commented 2 years ago

I have also experienced this issue. I believe it only happened when I erroneously modified the underlying Cloud Run instance after making an initial successful deployment, after which I was unable to successfully update the function.

It may be worth trying to post this on the firebase-tools Github since the deployment comes from the command line tools. Their team is also more responsive from what I've seen.

algoflows commented 2 years ago

@firebase-ops anyone on the dev team going to support this issue? Any news, updates or feedback?

paul-paliychuk commented 2 years ago

Also happened to me on intel chip

mehdi-abbad commented 2 years ago

I stumbled upon the same issue as @algoflows and after rereading the logs a couple of times, it occurred to me that I was getting the configuration of the function by calling functions.config which is not supported in functions v2. The error message is pretty clear on that:

Detailed stack trace: Error: functions.config() is no longer available
in Cloud Functions for Firebase v2. Please see the latest documentation
for information on how to transition to using environment variables

I hope this helps.

taeold commented 2 years ago

Hi. I'm sorry I missed this issue.

I re-read the issue thread, and common issue I see is that there are, to one degree or another, some initialization code that contribute to container instance not starting properly on cloud run.

For example, I can replicate the issue

Note that this isn't chip dependent - the container image deployed to Google Cloud Functions v2 is built on Cloud Build and is not affected by your development machine. Here's an example:

const functions = require("firebase-functions");
const { onRequest } = require("firebase-functions/v2/https");

const foo = functions.config("hello") // This will throw a runtime error on v2

module.exports = onRequest(
  { timeoutSeconds: 1200, region: ["europe-west1"] },
  (_req, res) => {
    res.status(200).send("OK");
  }
);

Note the use of functions.config - we do not support using functions config in v2 environment. Instead, we recommend that you use environment variables (https://firebase.google.com/docs/functions/config-env).

That being said, this is a very bad footgun. I will brainstorm with the team how we can improve the error message so that it's clear exactly what's going on during a function deploy.

algoflows commented 2 years ago

Legendary, and thanks for the response. Will investigate migrating from functions config to env vars as suggested.

Yes agree, clearer error messages would go a long way in helping isolate the issue 👍🏻

clementdevos commented 2 years ago

Hi,

I've just stumbled upon this issue while trying to use both v1 and v2 functions in the same repository. It might be bad code design but as deploying a functions packs the whole repository, it crashes upon functions loading when using functions.config() globaly exported.

This is the code that crashes our deployment.

import * as AlgoliaClient from 'algoliasearch';
import * as functions from 'firebase-functions';

console.log('init algolia', functions.config().algolia.app_id);
export const algolia = AlgoliaClient(functions.config().algolia.app_id, functions.config().algolia.sk, {
    timeouts: {
        connect: 8 * 1000,
        read: 100 * 1000,
        write: 60 * 1000
    }
});

The documentation here (https://firebase.google.com/docs/functions/config-env#environment_configuration ) is not clear about the fact that functions.config() used somewhere in v2 function environment will crash with this message (https://github.com/firebase/firebase-functions/pull/930)

We'll update the code to make it work then !

taeold commented 2 years ago

@clementdevos Thanks for the feedback!

Missing documentation is a clear oversight by us. I'll see if we can add a snippet about this behavior for v2 runtime.

Tangent - on separating v1 and v2 functions; I think you might find the relatively-new codebase feature useful there. Let us know if we an help w/ anything!

taeold commented 2 years ago

Closing the issue as I think the original issue has been identified and resolved.

jonas-ecoverse commented 1 year ago

I got the same issue although I used v2 functions env reference like this. image But I am not sure if it is because we have v1 and v2 in one repo. @taeold , having v1 and v2 together in one repo will impact that issue?

taishi55 commented 1 year ago

replace your functions.config() with process.env 🥳

and use secrets with .env file (you can do this by firebase functions:config:export)

matija2209 commented 1 year ago

I'm having the same issue.

I'm using params to define variables which I access with process.env.BACKEND_ADDRESS.

const { defineString } = require('firebase-functions/params');
const BACKEND_ADDRESS = defineString('BACKEND_ADDRESS');

I continue to receive the same error no matter what.

Could not create or update Cloud Run service add-document, Container Healthcheck failed. Revision 'add-document-00003-mep' is not ready and cannot serve traffic. The user-provided container failed to start and listen on the port defined provided by the PORT=8080 environment variable.

The code works locally just fine.

Blunderchips commented 1 year ago

I'm having the same issue.

I'm using params to define variables which I access with process.env.BACKEND_ADDRESS.

const { defineString } = require('firebase-functions/params');
const BACKEND_ADDRESS = defineString('BACKEND_ADDRESS');

I continue to receive the same error no matter what.

Could not create or update Cloud Run service add-document, Container Healthcheck failed. Revision 'add-document-00003-mep' is not ready and cannot serve traffic. The user-provided container failed to start and listen on the port defined provided by the PORT=8080 environment variable.

The code works locally just fine.

Also getting this error, we have not changed any of the default config.

asoomar commented 1 year ago

I'm also getting the same error. I am not using firebase.config but a .env file. Runs completely fine locally

matija2209 commented 1 year ago

@asoomar if you're running only functions make sure that your deploy script in package.json has a --only functions flag.

"deploy": "firebase deploy --only functions"

asoomar commented 1 year ago

@matija2209 I checked for this and it already had "deploy": "firebase deploy --only functions". I was able to fix the error by using gen 1 firestore trigger instead of gen 2. I read in another issue that for some reason functions that are deployed to Central1 do not work with gen 2 triggers.

Blunderchips commented 1 year ago

All I am getting from this issue is that firebase functions gen 2 is not ready for production.

BilalAtique commented 10 months ago

I was not using config(), so the above solution does not apply to me. What worked for me was deleting the lib function and deploying again.

umrashrf commented 8 months ago

At Riamuapp.com, I was using V1 onCall functions and firebase.config() was working fine but as soon as I added V2 functions to an existing file containing V1 functions to keep backward compatibility, I got this error. But I changed my code to use dotenv module and fall back to function.config in case dotenv doesn't work.

LipJ01 commented 7 months ago

If index.ts is V2 then any V1 functions using firebase.config() will prevent all V2 functions from deploying.

[or so it would seem on my firebase project today]

My fix:

const { defineInt, defineString } = require('firebase-functions/params');

See https://firebase.google.com/docs/functions/config-env?gen=2nd.

Naeem113 commented 4 months ago

I stumbled upon the same issue as @algoflows and after rereading the logs a couple of times, it occurred to me that I was getting the configuration of the function by calling functions.config which is not supported in functions v2. The error message is pretty clear on that:

Detailed stack trace: Error: functions.config() is no longer available
in Cloud Functions for Firebase v2. Please see the latest documentation
for information on how to transition to using environment variables

I hope this helps.

Thanks it works for me

nickjuntilla commented 3 months ago

Wow this is awful. It's July 10 2024 and I've spent the last 24 hours on this trying to upgrade a project. Not only is the error message completely misleading:

" Could not create or update Cloud Run service test, Container Healthcheck failed. Revision 'test-00001-dah' is not ready and cannot serve traffic. The user-provided container failed to start and listen on the port defined provided by the PORT=8080 environment variable. Logs for this revision might contain more information."

The error has nothing to do with a container or a port. It also says NOWHERE in the docs that using functions.config anywhere OUTSIDE of the v2 function will still cause deployment to fail... ?!?!? This is what you guys say:

https://firebase.google.com/docs/functions/2nd-gen-upgrade "2nd gen functions drop support for functions.config in favor of a more secure interface for defining configuration parameters declaratively inside your codebase. With the new params module, the CLI blocks deployment unless all parameters have a valid value, ensuring that a function isn't deployed with missing configuration."

This heavily implies that we should not use functions.config inside of v2 functions. How on Earth would we know that we also can't use functions.config OUTSIDE of the v2 functions?

In the email you sent out you wrote: "Cloud Functions (1st generation), the original version, used whenever targeting the firebase-functions/v1 SDK. Cloud Functions (2nd generation), a new version built on Cloud Run and Eventarc providing an enhanced feature set, which has been generally available since 2022. This is used whenever you need to target the firebase-functions/v2 SDK."

This implies we may mix the use of these 2 in the same project.

It took me many hours and much google-foo to find this thread. Now I have to go refactor my entire (very large) project to change the configuration style in the entire thing before I can finally test if I can deploy a v2 function finally. It is very misleading for you guys to make it seem like we can migrate over one function at a time, when in fact trying to do this leads to this undocumented error. I've always been a huge proponent of firebase, but this is the kind of thing that makes people look for alternatives.

Please consider a note on your migration page warning us about his behavior. https://firebase.google.com/docs/functions/2nd-gen-upgrade

NNN-kakimoto commented 1 month ago

I also reproduced it. (nodejs22, express.js, typescript) I originally thought I was running on functions gen1 and had migrated to gen2, but in fact the deployment was failing. I quit importing from firebase-functions/v1.

// from
- import { logger } from 'firebase-functions/v1';
- import * as logger from 'firebase-functions/logger';
// to
+ import { logger } from 'firebase-functions/v2';

Deployment now succeeds. I note this in case it helps anyone. (Machine translation JP-EN)