firebase deploy returns an exit code == 0 even if a function failed to be deployed

[Philmod]

[REQUIRED] Environment info

Version: 13.7.1

OS: Ubuntu

[REQUIRED] Test case

Deployment with a failing function update.

[REQUIRED] Steps to reproduce

firebase deploy returns an exit code of 0 even if there is a failure like:

# firebase deploy
⚠  functions: failed to update function projects/philmod-godds/locations/us-central1/functions/authentication-delete
Failed to update function projects/philmod-godds/locations/us-central1/functions/authentication-delete

# echo $?
0

[REQUIRED] Expected behavior

Returns an exit code != 0 if it failed.

[REQUIRED] Actual behavior

Return an exit code == 0 even in case of a failure.

[google-oss-bot]

This issue does not seem to follow the issue template. Make sure you provide all the required information.

[Philmod]

This issue does not seem to follow the issue template. Make sure you provide all the required information.

fixed

[aalej]

Hey @Philmod, thanks for filing this issue. I’m currently unable to reproduce the behavior you mentioned. I tried firebase deploy using the ff:

functions/index.js

const functions = require("firebase-functions/v1");
const logger = require("firebase-functions/logger");

exports.authDelete = functions.auth.user().onDelete((user) => {
 logger.log("---- user1 ----");
 logger.debug(user);
});

functions/.env - setting PATH causes the function update to fail

PATH="some/path"

And an error is being raised, which returns a non-zero exit code(2).

i  functions: updating Node.js 18 (1st Gen) function authDelete(us-central1)...
⚠  functions: failed to update function projects/PROJECT_ID/locations/us-central1/functions/authDelete
Failed to update function projects/PROJECT_ID/locations/us-central1/functions/authDelete

Functions deploy had errors with the following functions:
        authDelete(us-central1)
i  functions: cleaning up build files...

Error: There was an error deploying functions

Having trouble? Try again or contact support with contents of firebase-debug.log
USER$ echo $?
2

To get a better understanding of what’s causing the failed update to not raise an error, could you provide code snippets of the functions you’re trying to deploy, and additional details on your project’s setup such as env variables being set?

[Philmod]

Hey @aalej,

I'm using Google Cloud Build to deploy the firebase functions, using this docker image. I wonder if that's the reason the exit code is swallowed?

I don't think the function in itself has anything special. No specific env variables set either.

[nick-fields]

I'm also seeing this intermittently with 13.4.1 and 13.10.2 when deploying v1 functions. I just today had 6 functions fail to deploy, 2 that said "Waiting to retry", and 1 with no result whatsoever, and the deploy command exited with code 0. The 6 that failed were due to quota errors, same with the ones waiting to retry, the 1 that just died I suspect the same.

Edit: I just saw it again minutes later while debugging this further. This time it occurred when CI was deploying the same functions that I was attempting to deploy locally. Possibly a concurrency issue, and the CLI doesn't always exit cleanly when a newer deploy is triggered?

What can I provide that would be helpful here?

[nick-fields]

Another update to this, it happens frequently:

After sifting through our deploy logs, this happens quite frequently and in isolated environments with guaranteed no other deploys ongoing. It seems to happen to different functions, and always when at least 1 retry warning exists in the log.

We haven't noticed because we blindly retry deploys on error (thanks annoyingly low deployment quotas), as well as when the Deploy complete! message is not found in the output.

[dimavedenyapin]

Every 2nd deployment fails because of this problem for v1 functions. How are you solving currently?

[ripperdoc]

I have a similar issue. I run this command firebase deploy --only functions:playRequest -P $GCLOUD_PROJECT --non-interactive deploying just one function, but when two CI jobs ran closely together, one of them got a warning that it didn't deploy successfully:

i  functions: functions folder uploaded successfully
i  functions: updating Node.js 18 (2nd Gen) function playRequest(us-central1)...
⚠  functions: HTTP Error: 409, unable to queue the operation
⚠  functions:  failed to update function <removed>

I can understand that it could get a 409, but the big issue is that firebase deploy doesn't return an error, and allows the rest of the CI to continue. I guess I will need to add a manual check that Deploy complete! is there...

[devth]

I'm seeing the same behavior. Makes for a very non-dependable CI/CD. GitHub Actions shows it passes when the deploy actually failed.

here's my run in github actions ci.yaml:

          firebase deploy --force --only functions:pushNotifications,functions:smsNotifications,functions:emailNotifications -m 'Dev deploy for action 10597112401'
          echo $?

Failure with an unexpected 0 exit code:

✔  functions: required API cloudscheduler.googleapis.com is enabled
✔  functions: . folder uploaded successfully
i  functions: updating Node.js 20 (1st Gen) function pushNotifications-pushNotifications(us-central1)...
i  functions: updating Node.js 20 (1st Gen) function emailNotifications-emailNotifications(us-central1)...
i  functions: updating Node.js 20 (1st Gen) function smsNotifications-smsNotification(us-central1)...
⚠  functions: failed to update function projects/***/locations/us-central1/functions/pushNotifications-pushNotifications
Failed to update function projects/***/locations/us-central1/functions/pushNotifications-pushNotifications
Done in 23.68s.
+ echo 0
0

[intotecho]

I also get 0 returned after this failure with firebase deploy --only functions --project PROJECT -c firebase-cd.json

[2024-09-11T03:19:43.937Z] <<< [apiv2][body] POST https://cloudfunctions.googleapis.com/v1/projects/PROJECT/locations/europe-west3/functions {"error":{"code":403,"message":"Caller is missing permission 'iam.serviceaccounts.actAs' on service account PROJ_NR-compute@developer.gserviceaccount.com. Grant the role 'roles/iam.serviceAccountUser' to the caller on the service account PROJECT_NR-compute@developer.gserviceaccount.com. You can do that by running 'gcloud iam service-accounts add-iam-policy-binding PROJECT_NR-compute@developer.gserviceaccount.com --member MEMBER --role roles/iam.serviceAccountUser' where MEMBER has a prefix like 'user:' or 'serviceAccount:'. Details and instructions for the Cloud Console can be found at https://cloud.google.com/functions/docs/reference/iam/roles#additional-configuration. Please visit https://cloud.google.com/functions/docs/troubleshooting for in-depth troubleshooting documentation.","status":"PERMISSION_DENIED"}}
⚠  functions: failed to create function projects/PROJECT/locations/europe-west3/functions/firebaseAuthBeforeCreate 
Failed to create function projects/PROJECT_NR/locations/europe-west3/functions/firebaseAuthBeforeCreate

Another error that also returns 0 (success)

 Could not find image for function projects/PROJECT/locations/REGION/functions/firebaseAuthBeforeCreate

[mdinos]

Same behaviour for me in v13.13.3 with v1 functions.

// index.js

import { default as myfunction } from './myfunction.js'

export { myfunction }

// myfunction.js

import * as functions from 'firebase-functions'

export default functions
  .runWith({
    // service account doesn't exist, causes deployment to fail
    serviceAccount: `non-existent-sa@****.iam.gserviceaccount.com`,
  })
  .pubsub.schedule('0 9 * * 1,3,5')
  .onRun(() => {
    console.log('hello')
  })

$ firebase deploy -f --non-interactive
...
⚠  functions: failed to create function projects/****/locations/****/functions/myfunction

Then...