search

firebase / firebase-tools

The Firebase Command Line Tools
MIT License
4.03k stars 941 forks source link

Can't deploy all functions consistently. Fails most of the time. #3919

Closed MarwanZeitoun closed 1 year ago

MarwanZeitoun commented 2 years ago

Hello;

I have a project with 169 cloud functions and need to set up automatic deployment to firebase in GitLab, but the deployment is always failed.

I tried with various node versions, also with various access tokens, but always got the same error. Locally one of the developers is able to deploy without any issue. But locally in my environment, it fails. Even when replicating the versions of node and firebase.

here is my .yml file ` image: node:14.16.1 before_script:

first warnings like this with many functions ⚠ functions: got "Quota Exceeded" error while trying to update projects/xproject/locations/us-central1/functions/xxxxx. Waiting to retry... and then got this one Functions deploy had errors with the following functions: function1(us-central1) function2(us-central1) .... etc Because there were errors creating or updating functions, the following functions were not deleted To delete these, use firebase functions:delete and not all functions throw the error only a few functions and finally got this one

i functions: cleaning up build files... Cleaning up project directory and file based variables 00:01 ERROR: Job failed: exit code 1

Thanks in advance

abeisgoat commented 2 years ago

Hey we're aware of issues when deploying over 100 functions, it's best to deploy the functions you changed specifically instead of the whole lot when you're at this scale.

MarwanZeitoun commented 2 years ago

Hey we're aware of issues when deploying over 100 functions, it's best to deploy the functions you changed specifically instead of the whole lot when you're at this scale.

usually, we deploy only changed functions, but we need to find a way to set up CI/CD on this project, also when we deploy partially the config changes are still undeployed!

ggirotto commented 2 years ago

@abeisgoat I'm also facing a quota issue, although I'm trying to deploy just 20 functions. Can you confirm if it's related or it's another issue? The output that I have in Functions dashboard is:

{"@type":"type.googleapis.com/google.cloud.audit.AuditLog","status":{"code":8,"message":"Build failed: Quota exceeded for quota metric 'Build Create requests' and limit 'Build Create requests per minute' of service 'cloudbuild.googleapis.com' for consumer 'project_number:<NUMBER>'."},"authenticationInfo":{"principalEmail":"<EMAIL>"},"serviceName":"cloudfunctions.googleapis.com","methodName":"google.cloud.functions.v1.CloudFunctionsService.UpdateFunction","resourceName":"projects/ticker-dev-4d59e/locations/southamerica-east1/functions/<FUNCTION_NAME>"}

Deploying the functions individually works as expected, but trying to deploy all of them at the same time outputs me the error above.

yendiscart commented 2 years ago

I have the same issue.

Error: Functions did not deploy properly.

Having trouble? Try firebase [command] --help
mac@bogon functions % firebase deploy --only "functions:sendMessageNotification,functions:unsubscribeUserFromBroadcast,functions:setStatusCount,functions:resubscribeUserToBroadcasts,functions:addUserToGroup,functions:deviceIdChanged,functions:sendMessageToBroadcast,functions:participantRemoved,functions:groupInfoChanged,functions:participantAdded,functions:getVirgilJwt,functions:sendUnDeliveredNotifications,functions:unsubscribeUserFromTopicOnDelete,functions:saveUidOnLogin,functions:sendNewCallNotification,functions:deleteMessageForGroup,functions:indexNewCall,functions:groupAdminChanged,functions:deleteMessageForBroadcast,functions:indexNewGroupCall,functions:groupEvents,functions:getTime,functions:sendMessagesForGroups,functions:deleteMessage,functions:indexPKToken,functions:subscribeToBroadcast" 

=== Deploying to 'ichat-daac0'...

i  deploying functions
i  functions: ensuring required API cloudfunctions.googleapis.com is enabled...
i  functions: ensuring required API cloudbuild.googleapis.com is enabled...
✔  functions: required API cloudfunctions.googleapis.com is enabled
✔  functions: required API cloudbuild.googleapis.com is enabled
i  functions: preparing functions directory for uploading...
i  functions: packaged functions (73.82 KB) for uploading
✔  functions: functions folder uploaded successfully
i  functions: updating Node.js 10 function sendMessageNotification(us-central1)...
i  functions: updating Node.js 10 function sendMessagesForGroups(us-central1)...
i  functions: updating Node.js 10 function participantAdded(us-central1)...
i  functions: updating Node.js 10 function groupEvents(us-central1)...
i  functions: updating Node.js 10 function addUserToGroup(us-central1)...
i  functions: updating Node.js 10 function unsubscribeUserFromTopicOnDelete(us-central1)...
i  functions: updating Node.js 10 function participantRemoved(us-central1)...
i  functions: updating Node.js 10 function groupAdminChanged(us-central1)...
i  functions: updating Node.js 10 function groupInfoChanged(us-central1)...
i  functions: updating Node.js 10 function deleteMessageForGroup(us-central1)...
i  functions: updating Node.js 10 function deleteMessageForBroadcast(us-central1)...
i  functions: updating Node.js 10 function deleteMessage(us-central1)...
i  functions: updating Node.js 10 function saveUidOnLogin(us-central1)...
i  functions: updating Node.js 10 function getTime(us-central1)...
i  functions: updating Node.js 10 function setStatusCount(us-central1)...
i  functions: updating Node.js 10 function subscribeToBroadcast(us-central1)...
i  functions: updating Node.js 10 function resubscribeUserToBroadcasts(us-central1)...
i  functions: updating Node.js 10 function sendMessageToBroadcast(us-central1)...
i  functions: updating Node.js 10 function unsubscribeUserFromBroadcast(us-central1)...
i  functions: updating Node.js 10 function sendUnDeliveredNotifications(us-central1)...
i  functions: updating Node.js 10 function sendNewCallNotification(us-central1)...
i  functions: updating Node.js 10 function indexNewCall(us-central1)...
i  functions: updating Node.js 10 function indexNewGroupCall(us-central1)...
i  functions: updating Node.js 10 function indexPKToken(us-central1)...
i  functions: updating Node.js 10 function deviceIdChanged(us-central1)...
i  functions: updating Node.js 10 function getVirgilJwt(us-central1)...
⚠  functions: got "Quota Exceeded" error while trying to update projects/ichat-daac0/locations/us-central1/functions/subscribeToBroadcast. Waiting to retry...
⚠  functions: got "Quota Exceeded" error while trying to update projects/ichat-daac0/locations/us-central1/functions/setStatusCount. Waiting to retry...
⚠  functions: got "Quota Exceeded" error while trying to update projects/ichat-daac0/locations/us-central1/functions/groupInfoChanged. Waiting to retry...
⚠  functions: got "Quota Exceeded" error while trying to update projects/ichat-daac0/locations/us-central1/functions/participantAdded. Waiting to retry...
⚠  functions: got "Quota Exceeded" error while trying to update projects/ichat-daac0/locations/us-central1/functions/indexNewGroupCall. Waiting to retry...
⚠  functions: got "Quota Exceeded" error while trying to update projects/ichat-daac0/locations/us-central1/functions/getVirgilJwt. Waiting to retry...
⚠  functions: got "Quota Exceeded" error while trying to update projects/ichat-daac0/locations/us-central1/functions/deleteMessageForGroup. Waiting to retry...
⚠  functions: got "Quota Exceeded" error while trying to update projects/ichat-daac0/locations/us-central1/functions/unsubscribeUserFromBroadcast. Waiting to retry...
i  functions: updating Node.js 10 function subscribeToBroadcast(us-central1)...
i  functions: updating Node.js 10 function setStatusCount(us-central1)...
i  functions: updating Node.js 10 function groupInfoChanged(us-central1)...
i  functions: updating Node.js 10 function participantAdded(us-central1)...
i  functions: updating Node.js 10 function indexNewGroupCall(us-central1)...
i  functions: updating Node.js 10 function getVirgilJwt(us-central1)...
i  functions: updating Node.js 10 function deleteMessageForGroup(us-central1)...
i  functions: updating Node.js 10 function unsubscribeUserFromBroadcast(us-central1)...
⚠  functions: got "Quota Exceeded" error while trying to update projects/ichat-daac0/locations/us-central1/functions/groupInfoChanged. Waiting to retry...
⚠  functions: got "Quota Exceeded" error while trying to update projects/ichat-daac0/locations/us-central1/functions/participantAdded. Waiting to retry...
⚠  functions: got "Quota Exceeded" error while trying to update projects/ichat-daac0/locations/us-central1/functions/setStatusCount. Waiting to retry...
⚠  functions: got "Quota Exceeded" error while trying to update projects/ichat-daac0/locations/us-central1/functions/getVirgilJwt. Waiting to retry...
⚠  functions: got "Quota Exceeded" error while trying to update projects/ichat-daac0/locations/us-central1/functions/subscribeToBroadcast. Waiting to retry...
⚠  functions: got "Quota Exceeded" error while trying to update projects/ichat-daac0/locations/us-central1/functions/deleteMessageForGroup. Waiting to retry...
⚠  functions: got "Quota Exceeded" error while trying to update projects/ichat-daac0/locations/us-central1/functions/indexNewGroupCall. Waiting to retry...
⚠  functions: got "Quota Exceeded" error while trying to update projects/ichat-daac0/locations/us-central1/functions/unsubscribeUserFromBroadcast. Waiting to retry...
i  functions: updating Node.js 10 function groupInfoChanged(us-central1)...
i  functions: updating Node.js 10 function participantAdded(us-central1)...
i  functions: updating Node.js 10 function setStatusCount(us-central1)...
i  functions: updating Node.js 10 function getVirgilJwt(us-central1)...
i  functions: updating Node.js 10 function subscribeToBroadcast(us-central1)...
i  functions: updating Node.js 10 function deleteMessageForGroup(us-central1)...
i  functions: updating Node.js 10 function indexNewGroupCall(us-central1)...
i  functions: updating Node.js 10 function unsubscribeUserFromBroadcast(us-central1)...
⚠  functions: got "Quota Exceeded" error while trying to update projects/ichat-daac0/locations/us-central1/functions/indexNewGroupCall. Waiting to retry...
i  functions: updating Node.js 10 function indexNewGroupCall(us-central1)...

Functions deploy had errors with the following functions:
    sendMessageNotification(us-central1)
    saveUidOnLogin(us-central1)
    groupEvents(us-central1)
    sendNewCallNotification(us-central1)
    indexNewCall(us-central1)
    addUserToGroup(us-central1)
    resubscribeUserToBroadcasts(us-central1)
    deviceIdChanged(us-central1)
    deleteMessage(us-central1)
    sendUnDeliveredNotifications(us-central1)
    sendMessagesForGroups(us-central1)
    deleteMessageForBroadcast(us-central1)
    unsubscribeUserFromTopicOnDelete(us-central1)
    indexPKToken(us-central1)
    sendMessageToBroadcast(us-central1)
    groupAdminChanged(us-central1)
    getTime(us-central1)
    participantRemoved(us-central1)
    participantAdded(us-central1)
    subscribeToBroadcast(us-central1)
    groupInfoChanged(us-central1)
    getVirgilJwt(us-central1)
    unsubscribeUserFromBroadcast(us-central1)
    deleteMessageForGroup(us-central1)
    setStatusCount(us-central1)
    indexNewGroupCall(us-central1)

To try redeploying those functions, run:
    firebase deploy --only "functions:sendMessageNotification,functions:saveUidOnLogin,functions:groupEvents,functions:sendNewCallNotification,functions:indexNewCall,functions:addUserToGroup,functions:resubscribeUserToBroadcasts,functions:deviceIdChanged,functions:deleteMessage,functions:sendUnDeliveredNotifications,functions:sendMessagesForGroups,functions:deleteMessageForBroadcast,functions:unsubscribeUserFromTopicOnDelete,functions:indexPKToken,functions:sendMessageToBroadcast,functions:groupAdminChanged,functions:getTime,functions:participantRemoved,functions:participantAdded,functions:subscribeToBroadcast,functions:groupInfoChanged,functions:getVirgilJwt,functions:unsubscribeUserFromBroadcast,functions:deleteMessageForGroup,functions:setStatusCount,functions:indexNewGroupCall"

To continue deploying other features (such as database), run:
    firebase deploy --except functions

Error: Functions did not deploy properly.
mac@bogon functions %
milo- commented 2 years ago

This is also happening for us pretty consistently, 3-5 will fail with quota exceeded when deploying 60 functions. We're using firebase-tools 9.23.0.

yendiscart commented 2 years ago

HOW do you solve this issue? Tried to deploy one by one and still getting errors.

To try redeploying those functions, run: firebase deploy --only "functions:sendMessageNotification,functions:groupAdminChanged,functions:deleteMessageForGroup,functions:setStatusCount,functions:groupInfoChanged,functions:deleteMessageForBroadcast,functions:unsubscribeUserFromBroadcast,functions:unsubscribeUserFromTopicOnDelete,functions:saveUidOnLogin,functions:indexNewGroupCall,functions:sendMessagesForGroups,functions:addUserToGroup,functions:participantRemoved,functions:participantAdded,functions:groupEvents,functions:deleteMessage,functions:getVirgilJwt,functions:sendMessageToBroadcast,functions:indexPKToken,functions:sendNewCallNotification,functions:subscribeToBroadcast,functions:indexNewCall,functions:sendUnDeliveredNotifications,functions:deviceIdChanged,functions:resubscribeUserToBroadcasts,functions:getTime"

To continue deploying other features (such as database), run: firebase deploy --except functions

Error: Functions did not deploy properly. mac@bogon functions % firebase deploy --only "functions:getTime"

=== Deploying to 'ichat-daac0'...

i deploying functions i functions: ensuring required API cloudfunctions.googleapis.com is enabled... i functions: ensuring required API cloudbuild.googleapis.com is enabled... ✔ functions: required API cloudbuild.googleapis.com is enabled ✔ functions: required API cloudfunctions.googleapis.com is enabled i functions: preparing functions directory for uploading... i functions: packaged functions (73.93 KB) for uploading ✔ functions: functions folder uploaded successfully i functions: updating Node.js 10 function getTime(us-central1)...

Functions deploy had errors with the following functions: getTime(us-central1)

To try redeploying those functions, run: firebase deploy --only "functions:getTime"

To continue deploying other features (such as database), run: firebase deploy --except functions

Error: Functions did not deploy properly. mac@bogon functions % functions % firebase deploy --only "functions:resubscribeUserToBroadcasts" mac@bogon functions % firebase deploy --only "functions:resubscribeUserToBroadcasts"

=== Deploying to 'ichat-daac0'...

i deploying functions i functions: ensuring required API cloudfunctions.googleapis.com is enabled... i functions: ensuring required API cloudbuild.googleapis.com is enabled... ✔ functions: required API cloudfunctions.googleapis.com is enabled ✔ functions: required API cloudbuild.googleapis.com is enabled i functions: preparing functions directory for uploading... i functions: packaged functions (73.93 KB) for uploading ✔ functions: functions folder uploaded successfully i functions: updating Node.js 10 function resubscribeUserToBroadcasts(us-central1)...

Functions deploy had errors with the following functions: resubscribeUserToBroadcasts(us-central1)

To try redeploying those functions, run: firebase deploy --only "functions:resubscribeUserToBroadcasts"

To continue deploying other features (such as database), run: firebase deploy --except functions

Error: Functions did not deploy properly.

Having trouble? Try firebase [command] --help

joehan commented 2 years ago

Hey all - looks like there are a few different issues here, so I'll take them one at a time:

ggirotto commented 2 years ago

@joehan The original problem that relates with the large number of functions is the same as I'm facing? Because I have a similar issue that outputs a quota error although I'm trying to deploy around 20 functions only. I'm facing:

{"@type":"type.googleapis.com/google.cloud.audit.AuditLog","status":{"code":8,"message":"Build failed: Quota exceeded for quota metric 'Build Create requests' and limit 'Build Create requests per minute' of service 'cloudbuild.googleapis.com' for consumer 'project_number:<NUMBER>'."},"authenticationInfo":{"principalEmail":"<EMAIL>"},"serviceName":"cloudfunctions.googleapis.com","methodName":"google.cloud.functions.v1.CloudFunctionsService.UpdateFunction","resourceName":"projects/ticker-dev-4d59e/locations/southamerica-east1/functions/<FUNCTION_NAME>"}

inlined commented 2 years ago

For clarification, are these failures longstanding issues or regressions as of a recent version of firebase-tools? We did some major refactoring recently and it's possible we have a regression for large codebases

yendiscart commented 2 years ago

My error was solved by upgrading my billing or plan to Blaze before I could deploy functions.

HOW do you solve this issue? Tried to deploy one by one and still getting errors.

To try redeploying those functions, run: firebase deploy --only "functions:sendMessageNotification,functions:groupAdminChanged,functions:deleteMessageForGroup,functions:setStatusCount,functions:groupInfoChanged,functions:deleteMessageForBroadcast,functions:unsubscribeUserFromBroadcast,functions:unsubscribeUserFromTopicOnDelete,functions:saveUidOnLogin,functions:indexNewGroupCall,functions:sendMessagesForGroups,functions:addUserToGroup,functions:participantRemoved,functions:participantAdded,functions:groupEvents,functions:deleteMessage,functions:getVirgilJwt,functions:sendMessageToBroadcast,functions:indexPKToken,functions:sendNewCallNotification,functions:subscribeToBroadcast,functions:indexNewCall,functions:sendUnDeliveredNotifications,functions:deviceIdChanged,functions:resubscribeUserToBroadcasts,functions:getTime"

To continue deploying other features (such as database), run: firebase deploy --except functions

Error: Functions did not deploy properly. mac@bogon functions % firebase deploy --only "functions:getTime"

=== Deploying to 'ichat-daac0'...

i deploying functions i functions: ensuring required API cloudfunctions.googleapis.com is enabled... i functions: ensuring required API cloudbuild.googleapis.com is enabled... ✔ functions: required API cloudbuild.googleapis.com is enabled ✔ functions: required API cloudfunctions.googleapis.com is enabled i functions: preparing functions directory for uploading... i functions: packaged functions (73.93 KB) for uploading ✔ functions: functions folder uploaded successfully i functions: updating Node.js 10 function getTime(us-central1)...

Functions deploy had errors with the following functions: getTime(us-central1)

To try redeploying those functions, run: firebase deploy --only "functions:getTime"

To continue deploying other features (such as database), run: firebase deploy --except functions

Error: Functions did not deploy properly. mac@bogon functions % functions % firebase deploy --only "functions:resubscribeUserToBroadcasts" mac@bogon functions % firebase deploy --only "functions:resubscribeUserToBroadcasts"

=== Deploying to 'ichat-daac0'...

i deploying functions i functions: ensuring required API cloudfunctions.googleapis.com is enabled... i functions: ensuring required API cloudbuild.googleapis.com is enabled... ✔ functions: required API cloudfunctions.googleapis.com is enabled ✔ functions: required API cloudbuild.googleapis.com is enabled i functions: preparing functions directory for uploading... i functions: packaged functions (73.93 KB) for uploading ✔ functions: functions folder uploaded successfully i functions: updating Node.js 10 function resubscribeUserToBroadcasts(us-central1)...

Functions deploy had errors with the following functions: resubscribeUserToBroadcasts(us-central1)

To try redeploying those functions, run: firebase deploy --only "functions:resubscribeUserToBroadcasts"

To continue deploying other features (such as database), run: firebase deploy --except functions

Error: Functions did not deploy properly.

Having trouble? Try firebase [command] --help

inlined commented 2 years ago

Will close soon if nobody is still experiencing issues.

42ae commented 2 years ago

Unfortunately I am also experiencing these issue often (if not each time) trying to deploy all my GCF at once

⚠ functions: got "Quota Exceeded" error while trying to update *********** Waiting to retry...

I know it's best to deploy only the group of functions that have been updated and leave the other ones untouched, but sometimes when updating environment configuration it is required to deploy all cloud function to allow the new configuration to propagate properly.

Please advise on this issue, it would be great to find a better workaround to allow better integration with CI too.

Thanks!

aurimas-fertilemind commented 2 years ago

Experiencing same issue while deploying 62 functions.

⚠  functions: got "Quota Exceeded" error while trying to update projects/<redacted>/locations/us-central1/functions/<redacted>. Waiting to retry...

2 functions fails to deploy, thus I assume there should be somewhere 60 function deployment limit?

As @42ae mentioned previously, sometimes bulk deployment is needed in order for all functions to be updated. Looking forward hearing for solutions to this problem.

vishen commented 2 years ago

We are also experiencing something similar with about 80 functions where 5-6 regularly fail. We are currently on version 10.1.4.

functions: got "Quota Exceeded" error while trying to update projects/***/locations/northamerica-northeast1/functions/<function>. Waiting to retry...
trex-quo commented 2 years ago

I am frequently experiencing this issue as well while deploying over 100 functions. We deploy in groups and have created some parsing logic for our CI system, however it would be very convenient to have these auto-retry if we pass the deployment quota, or at least spit out a command after failure to deploy the rest of the functions (currently it just lists them).

The current text says "waiting to retry" while we exceed the quota, so that was the behavior I was expecting, however it is not what ended up happening.

codin-play commented 2 years ago

I can consistently get this error even when deploying a group of only 5 functions. The error message literally shows nothing useful and it has been a real pain. I can deploy each one individually which results in deployments for a full stack to take hours. Any ideas from folks on how to overcome this would be really appreciated.

$ firebase --version
10.5.0

$ gcloud --version
Google Cloud SDK 369.0.0
alpha 2022.01.14
beta 2022.01.14
bq 2.0.72
core 2022.01.14
gsutil 5.6

Deployment failure:
Build failed: {"metrics":{},"error":{"buildpackId":"","buildpackVersion":"","errorType":"OK","canonicalCode":"OK","errorId":"","errorMessage":""},"stats":[{"buildpackId":"google.utils.archive-source","buildpackVersion":"0.0.1","totalDurationMs":199,"userDurationMs":198},{"buildpackId":"google.nodejs.npm","buildpackVersion":"0.9.0","totalDurationMs":19905,"userDurationMs":19841},{"buildpackId":"google.nodejs.functions-framework","buildpackVersion":"0.9.4","totalDurationMs":3764,"userDurationMs":3744},{"buildpackId":"google.utils.label","buildpackVersion":"0.0.2","totalDurationMs":0,"userDurationMs":0}],"warnings":null,"customImage":false}

EDIT: 4/7/2022

This issue seemed to only be happening in Github Actions CI on a cloud runner. It seemed perhaps the runner was running out of memory. I modified the runner to a self-hosted runner and the issues seemed to go away. Hopefully this helps someone else.

inlined commented 2 years ago

We will prioritize looking into this soon. There will be a three pronged strategy:

  1. Fixing the "invalid source token" problem, which occurs when function deploys take a very long time
  2. Looking at tuning the rate limiter to avoid quota errors
  3. Improving deploy code so that it can recognize and skip functions that don't need redeployment

Hopefully this will help you all when we can get these improvements ready.

trex-quo commented 2 years ago

This is encouraging to hear; thank you for the update!

olve commented 2 years ago

We will prioritize looking into this soon.

@inlined any updates on this?

inlined commented 2 years ago

We've currently launched (but not well documented or added wizzard support for) codebases. With codebases you can have multiple directories of Cloud Functions. Simply create a firebase.json fragment that looks a bit like:

"functions" : [
  {
    "source": "functions",
    "codebase": "main"
  }, {
    "source": "otherfunctions",
    "codebase": "other"
  }
]

With this, you can now call firebase deploy functions:main or firebase deploy functions:other. Next, we're working on recognizing that a codebase hasn't been modified since last deployment and we'll skip those functions altogether.

sluramod commented 2 years ago

@inlined is it possible there is a regression and https://github.com/firebase/firebase-tools/issues/2606 / https://github.com/firebase/firebase-tools/pull/3246 no longer works? Is it possible to restore 'retry on quota' functionality?

Kopunk commented 2 years ago

We've currently launched (but not well documented or added wizzard support for) codebases. With codebases you can have multiple directories of Cloud Functions. Simply create a firebase.json fragment that looks a bit like:

"functions" : [
  {
    "source": "functions",
    "codebase": "main"
  }, {
    "source": "otherfunctions",
    "codebase": "other"
  }
]

With this, you can now call firebase deploy functions:main or firebase deploy functions:other. Next, we're working on recognizing that a codebase hasn't been modified since last deployment and we'll skip those functions altogether.

@inlined Sorry, but I don't see how introducing codebases aims to solve this issue (what if a single codebase has more functions than the quota limit allows?). While it might become useful in some cases it certainly won't be helpful for me.

Wouldn't it be better if there was an option to deploy only modified functions? (--only-changed?) Usually a single push only affects a few functions so deploying only those would help.

gustavopch commented 2 years ago

I'm getting "Invaild source token" every time for the same 2 functions when deploying 24 Node.js v16 functions at once. If I deploy then individually, there's no error. I'm using firebase-tools@11.1.0.

gustavopch commented 2 years ago

I've sent a bug report through https://firebase.google.com/support/troubleshooter/report/bugs and they answered pointing to this part of the documentation:

When deploying large numbers of functions, you may exceed the standard quota and receive HTTP 429 or 500 error messages. To solve this, deploy functions in groups of 10 or fewer. Link: https://firebase.google.com/docs/functions/manage-functions?authuser=0#deploy_functions

But @taeold said in another issue that:

The Firebase CLI is tuned to deploy in batches of 40 functions at any given time, so I suspect issuing a single firebase deploy command to deploy all your 80+ functions should be much better than your expectation. Link: https://github.com/firebase/firebase-tools/issues/4266#issuecomment-1067381741

They seem to contradict each other. What am I missing?

gustavopch commented 2 years ago

So, I was trying to deploy 24 functions at once and 2 failed. I've experimented more and realized that if I deploy exactly those 2 problematic functions, deployment happens just fine. If I deploy the 2 problematic functions + another function, the 2 problematic functions fail again while the other function is successfully deployed.

EDIT: Just upgraded to firebase-tools@11.4.0 and deploying the 24 functions at once works again!

inlined commented 2 years ago

Scanning through some bugs and came across this one. I wanted to give a few updates:

  1. Regression/Fix. I'm glad that the regression seems fixed already and sorry that we had a regression.
  2. Invalid source token. We have not started a fix here. If someone wants to try fixing this, it requires some cleverness but not knowledge of the codebase
  3. Quotas You may find some (seemingly) contradictory advice about running w/ --only and groups of 10 functions or that we may run your functions in batches. The Firebase CLI already batches your functions for deploy thanks to the ThrottlerQueue class; this is basically a JS equivalent of a threadpool and makes sure we only have N functions deploying concurrently. Then with backoffs we approximate the most calls you can make in a project within quota limits. It's a pipe dream to have an improved request queue that is quota aware so it can do smarter backoffs, but we may not get there for a long time. Manually deploying only a few functions at a time is even more conservative than normal. You are not supposed to run that command several times in parallel, since that removes all benefits of fewer deploys. In the future, Codebases will also help you limit the number of functions that are deployed.
  4. Codebases feature

Wouldn't it be better if there was an option to deploy only modified functions? (--only-changed?) Usually a single push only affects a few functions so deploying only those would help.

@Kopunk Unfortunately, a more accurate implementation of --only-changed that isn't the codebases feature would be about as complicated as writing a JavaScript interpreter (which means that a true implementation of --only-changed is probably also impossible due to the Halting Problem). Any non-trivial cloud function leans on utility methods. Any change to any one of these utility methods will change some amount of code that can only be determined by running through a program.

Codebases is a compromise. If you agree to put firewalls between parts of your code (e.g. directories) we can instead do a file diff and notice nothing changes. Sometimes you may have shared code and it truly isn't possible to separate your code cleanly (e.g. there are shared interfaces). For the first several years of its existence, Firebase solved this problem using an internal-API node module. I've heard that Yarn workspaces can solve this quite eloquently as well. Many teams at Google use them.

willhackett commented 2 years ago

We've got a large number of functions (~150) we deploy regularly. We get maybe 1 - 2 failed function deploys per run due to the "Quota Exceeded" error.

I thought firebase-tools was meant to manage this & retry, but we're not seeing this logic succeed.

Is there any way to work around this without having to rearchitect our entire production application? I understand the codebases pattern, but this won't work for our use case given all the functions are camel-case named.

taeold commented 2 years ago

@willhackett I'd love to understand what you mean by camlcased name functions better - can you elaborate why codebases doesn't fit your application?

I'm planning to review https://github.com/firebase/firebase-tools/pull/4967 soon, but my feeling is that fighting the GCF quota limit isn't fool-proof way, especially since quota is different depending on your project's state, location, history, etc.

Our thesis is that codebase with feature to automcatically skip deployment of function that doesn't require a re-deploy is more promising and I'd love to understand in what ways we could improve codebase support to work more developers' setups.

gustavopch commented 2 years ago

@taeold Suppose I want to have very granular updates. If a single function changed, I want to redeploy just that one function and no other. Would it make sense to have 1 function per codebase? Any possible issues? Or do you think that would be over-engineering in the first place?

taeold commented 2 years ago

@gustavopch I think somewhere between the 2 extremes (1 codebase 100 functions vs 100 codebase 1 function) would help. For example, you might group function in a codebase by trigger type. Or by their business domain. In the ideal case, when you are apply a bug fix or a new feature, it would affect a single codebase. But I suspect that our reality is a bit messier than that.

You can give "skip deploying a codebase that's doesn't require one" right now by using the latest Firebase CLI and enabling the priview:

firebase --open-sesame skipdeployingnoopfunctions

We are making some last pushes to get this feature cleaned up, an when its ready you see it be mentioned in our documentations. Would love to hear from advanced users like yourselves what you think.

gustavopch commented 2 years ago

@gustavopch Oh, interesting. Currently, I have a script that uses ESBuild to bundle each function independently and generate a hash, so when I'm deploying, I only deploy functions whose hashes changed. Is skipdeployingnoopfunctions similar to that? Can you explain what logic it performs to detect what to deploy and what not?

taeold commented 2 years ago

@gustavopch Nice - I've been experimenting with esbuild too and liked the results (possible cold start improvements!)

Yes, very much similar to what you described. In addition, we check that the secrets associated with your function - if there is a new version of a secret linked to your function, we mark it as re-deploy.

TheIronDev commented 2 years ago

skipdeployingnoopfunctions generates a hash from:

If a file in a codebase changes (add a space, etc.), then the generated hash for that file (and as a result the subsequent codebase) will receive a new hash.

gustavopch commented 2 years ago

@TheIronDev @taeold Thanks for clarifying.

I think somewhere between the 2 extremes (1 codebase 100 functions vs 100 codebase 1 function) would help.

What problems could I find if I put each function in a separate codebase (1 function = 1 codebase)?

taeold commented 2 years ago

@gustavopch That's an interesting stress case! One edge case you might consider is that a Cloud Function deploy is actually a 2 step process:

1) Create container image from your source code 2) Deploy your container image to GCF

Since we don't want to create the same container image for every function you have, we have special logic to generate 1 container image per codebase^ and to re-use it for all functions in the codebase.

So if you have 100 functions in 1 codebase, your deploy will call out to Cloud Build once to generate and reuse a single container image. If you have 100 codebases with 1 functions each, you will call out to Cloud Build 100 times. That probably is going to hit your Cloud Build quota in some way.

^ I'm lying here a bit. We actually generate 1 container image per codebase-memory pair for technical reasons.

gustavopch commented 2 years ago

@taeold If I have 100 functions (each in an isolated codebase) and deploy a change that affects only 5 functions, will it consume quota for 100 containers or for 5 containers?

google-oss-bot commented 2 years ago

Hey @MarwanZeitoun. We need more information to resolve this issue but there hasn't been an update in 7 weekdays. I'm marking the issue as stale and if there are no new updates in the next 3 days I will close it automatically.

If you have more information that will help us get to the bottom of this, just add a comment!

s11richard commented 2 years ago

We have about 97 functions, and consistently get Quota errors when deploying them, which then requires multiple retries to fix our deployment pipeline. This problem obviously gets worse as we add more functions.

It would be great if there was some guidance on how to manage a project that has 100+ functions. Or is that not supported/expected?

It seems like there used to be some retry capabilities. Having those reinstated would be very helpful. Is that on the roadmap at all? Is there any other fix to this issue?

hefgi commented 2 years ago

It would be great if there was some guidance on how to manage a project that has 100+ functions.

Could anyone confirm the following please?

From my understanding, the best practice for a scalable project, is to "group" the functions in folders or repositories (or codebases). The idea is to have < 30 functions per "group" in order to avoid the "Quota Exceeded" problem. Then we could deploy manually or using CI/CD individually for each "group".

Here's the doc from Firebase: https://firebase.google.com/docs/functions/organize-functions Here's a tutorial: https://codeburst.io/organizing-your-firebase-cloud-functions-67dc17b3b0da

taeold commented 2 years ago

@s11richard @hefgi Thanks for reaching out!

While I can see improvements in trying to improve experience in deploying large amount of functions with a single deploy command, it's always going to be hard because Google Cloud Platform's quota is going to be dynamic and different for every user. We also noticed that it often changes over time and can be different for each project for the same user.

Our current function deploy impl. is an attempt to navigate the quota as best as we could, and while I'm pretty sure that it's an improvement over our past impl., it's obvious from this issue that we still have much to go to cover everyone's use cases.

Lately, we've been experimenting from another direction:

Is there a way to reduce the number of functions that's deployed on each deployment?

Say make a code change that affects runtime behavior of a subset of your functions. Do we really need to deploy all of the functions?

@hefgi's suggestion of breaking your functions into codebases is part of an answer. But it still means you need to manually pick which codebases to deploy on each run.

@TheIronDev recently worked on a new feature to automatically detect and skip deployment of functions in a codebase without any changes.

I'd strongly recommend you give codebases and "no-op deploy" a try. In the coming weeks, we'll try to publish more public guidance on how we think we can improve reliability of deploys for projects with many many functions.

dtran320 commented 2 years ago

Thanks so much for all the advice @taeold.

Unfortunately for us, refactoring to move to codebases isn't feasible at the moment.

Looking at the changelog for firebase-functions 11.13.0, could you help clarify what the change from #5032's flipping of the skip no-op deploys flag to true does? Does that ONLY apply to diff-ing the hashes for entire codebases and only applying a skip no-op deploy for a set of codebase functions, or is it meant to hash each individual function and skip the no-op function deploy for each individual function? We updated our fork of the firebase-action for Github to use firebase-tools 11.13.0 but I believe I'm still seeing it deploy every function even though they should all be no-ops. Thanks so much!

TheIronDev commented 2 years ago

@dtran320

Looking at the changelog for firebase-functions 11.13.0, could you help clarify what the change from #5032's flipping of the skip no-op deploys flag to true does?

That enables the no-op deploy detection by default.

Does that ONLY apply to diff-ing the hashes for entire codebases and only applying a skip no-op deploy for a set of codebase functions, or is it meant to hash each individual function and skip the no-op function deploy for each individual function?

Applying skip deployment of no-op deploys is applied to a set of functions within a codebase.

We updated our fork of the firebase-action for Github to use firebase-tools 11.13.0 but I believe I'm still seeing it deploy every function even though they should all be no-ops.

That shouldn't be the case..... diffing codebases occur by checking:

  1. the codebase's folder
  2. environment variables
  3. secret versions

If you want to check, I'd suggest doing thee following from a single box

  1. make sure you're on > firebase-tools@11.13.0
  2. run firebase deploy
  3. run firebase deploy again (from the same box)

You should see:

gustavopch commented 2 years ago

If I have 100 functions (each in an isolated codebase) and deploy a change that affects only 5 functions, will it consume quota for 100 containers or for 5 containers?

@TheIronDev What about this?

dtran320 commented 2 years ago

Thanks @taeold!

If you want to check, I'd suggest doing thee following from a single box

make sure you're on > firebase-tools@11.13.0 run firebase deploy run firebase deploy again (from the same box)

Thought this might be an issue with our CI, so deployed locally several times in a row (without changes), and when they finally go through without the quota error, still haven't been able to get anything to successfully display Skipping the deploy of unchanged functions...

% firebase --version
11.13.0
firebase deploy --only functions

=== Deploying to <REDACTED>...

i  deploying functions
✔  functions: Finished running predeploy script.
i  functions: ensuring required API cloudfunctions.googleapis.com is enabled...
i  functions: ensuring required API cloudbuild.googleapis.com is enabled...
i  artifactregistry: ensuring required API artifactregistry.googleapis.com is enabled...
✔  artifactregistry: required API artifactregistry.googleapis.com is enabled
✔  functions: required API cloudfunctions.googleapis.com is enabled
✔  functions: required API cloudbuild.googleapis.com is enabled
i  functions: preparing codebase default for deployment
i  functions: preparing functions directory for uploading...
i  functions: packaged /path/to/functions (7.41 MB) for uploading
i  functions: ensuring required API cloudscheduler.googleapis.com is enabled...
✔  functions: required API cloudscheduler.googleapis.com is enabled
✔  functions: functions folder uploaded successfully
i  functions: updating Node.js 14 function <REDACTED>(us-central1)...
i  functions: updating Node.js 14 function <REDACTED>(us-central1)...
...
✔  functions[<REDACTED>(us-central1)] Successful update operation.
✔  functions[<REDACTED>(us-central1)] Successful update operation.
...
Function URL (<REDACTED>(us-central1)): https://<REDACTED>
Function URL (<REDACTED>(us-central1)): https://<REDACTED>
...
i  functions: cleaning up build files...

✔  Deploy complete!

Nothing changes in subsequent deploys

firebase deploy --only functions

=== Deploying to <REDACTED>...

i  deploying functions
✔  functions: Finished running predeploy script.
i  functions: ensuring required API cloudfunctions.googleapis.com is enabled...
i  functions: ensuring required API cloudbuild.googleapis.com is enabled...
i  artifactregistry: ensuring required API artifactregistry.googleapis.com is enabled...
✔  artifactregistry: required API artifactregistry.googleapis.com is enabled
✔  functions: required API cloudfunctions.googleapis.com is enabled
✔  functions: required API cloudbuild.googleapis.com is enabled
i  functions: preparing codebase default for deployment
i  functions: preparing functions directory for uploading...
i  functions: packaged /path/to/functions (7.41 MB) for uploading
i  functions: ensuring required API cloudscheduler.googleapis.com is enabled...
✔  functions: required API cloudscheduler.googleapis.com is enabled
✔  functions: functions folder uploaded successfully
i  functions: updating Node.js 14 function <REDACTED>(us-central1)...
i  functions: updating Node.js 14 function <REDACTED>(us-central1)...
...
✔  functions[<REDACTED>(us-central1)] Successful update operation.
✔  functions[<REDACTED>(us-central1)] Successful update operation.
...
Function URL (<REDACTED>(us-central1)): https://<REDACTED>
Function URL (<REDACTED>(us-central1)): https://<REDACTED>
...
i  functions: cleaning up build files...

✔  Deploy complete!

Also tried a deploy with --debug, but didn't see anything that stuck out. If anyone else has gotten this to work, tips would be greatly appreciated!

mike-4040 commented 2 years ago

We have about 150 functions.

% firebase -V
11.14.0

During the first deploy, 33 functions failed. Second attempt: seen some Skipped (No changes detected), but only a handful (10). Mostly firebase was trying to deploy the same function again even after a successful deployment.

alexlouden commented 2 years ago

We have just over 100 functions, and similarly have had a lot of issues with deploy reliability/quota issues (including exhausting the daily quota limit). We are currently using a custom deploy script to deploy them in batches of 20 functions at a time using --only - this was the only way to make continuous deployments reliable, however deploys take 30~45 minutes. (If there's interest I can tidy up and share this script!)

I've updated to 11.14.0, re-run the deployment on our CI server twice (no filesystem/env changes) and I'm not seeing any skipped functions - all are re-deploying. We're not using codebases at the moment - just a single "default" codebase.

The two things that are preventing us from migrating to codebases are:

  1. We have some common modules/shared code - for managing config, firestore setup and getters/setters, firebase auth helpers, internal utils. We could split these out using yarn workspaces as mentioned above, but this adds extra developer overhead and complexity for testing/mocking/etc. I'd prefer a solution that just re-deployed all functions if a specific folder of shared code changed, or ideally operated on a per-function basis - e.g. by hashing each bundled & tree-shaken function? (the ESBuild approach mentioned above sounds great to me!)

  2. Since the names of HTTP functions form the public function URL (https://us-central1-myproject.cloudfunctions.net/myFunctionName), we can't rename these functions since the URLs are used by external services and in native client apps. (We now give new HTTP functions nice URLs using firebase hosting rewrites, but that doesn't help with existing URLs that are in use). If we had control over the function URLs or if we could maintain the current URLs (/myFunctionName rather than /mycodebase-myFunctionName) then we could get around this.

alexlouden commented 2 years ago

I was chatting to some friends who use AWS - looks like lambda uses the esbuild approach: https://github.com/aws/aws-cdk/tree/main/packages/%40aws-cdk/aws-lambda-nodejs Could be useful for reference ☺️

gustavopch commented 2 years ago

On a high level:

The problem is how to identify what functions need to be redeployed when dependencies in package.json change. Ideally, bumping a dependency that's used by a single function shouldn't cause all the other functions to be redeployed. Maybe use regex to identify the imports/requires in the output produced by ESBuild.

gustavopch commented 2 years ago

I'd be happy enough if somehow I could inform firebase-tools about the hashes that I generated by myself.

firebase deploy hashes:{"hosting":{"mysite":"abc123"},"functions":{"fn1":"bcd234","fn2":"cde345"}}
mike-4040 commented 2 years ago

@gustavopch:

The problem is how to identify what functions need to be redeployed when dependencies in package.json change. Ideally, bumping a dependency that's used by a single function shouldn't cause all the other functions to be redeployed. Maybe use regex to identify the imports/requires in the output produced by ESBuild.

I believe ESBuild can be configured to include node_modules, I will be slower, but this way all dependencies will be in the bundles and the hash will change of dependency change.