Closed wojtek-iwarranty closed 1 day ago
For those stuck with this for, you can deploy in 2 steps reliably when you do a manual deploy, rather than running the job over and over, hoping it succeeds eventually:
First, do a firebase deploy --only functions
If it fails (and it probably will) it's going to fail on some, succeed on hopefully most. At the end of the run, it's going to produce a list of functions that failed to deploy like this:
- Error Failed to update function bioCreated in region us-central1
- Error Failed to update function messageCreated in region us-central1
- Error Failed to update function groupInvitationUpdated in region us-central1
Extract that list of functions into a list and re-run deploy like this:
firebase deploy --only "functions:bioCreated, functions:messageCreated, functions:groupInvitationUpdated"
and it should succeed.
Hey everyone, we're still waiting for the single builds feature to finish rolling out globally - once we do, we'll update the CLI to re-enable the feature by default. The Cloud team has been proceeding carefully with the release given that we've had issues with the feature in the past. A rough estimate would be at least another week until the global rollout is complete.
Are there any workarounds we can implement ourselves other than breaking the function deploys into groups of 5? Is there a way for us to increase the timeouts?
Unfortunately, users are unable to modify the Cloud Build queue TTL. We're considering increasing the TTL for all GCF 2nd gen functions, and will update if we decide to move forward with that.
If we built a single function and deployed it would the next deploy be able to use the first function build cache?
You are essentially describing the Single Builds feature. :)
Each deployment consumes builder minutes as far as I'm aware, which costs us. There should be no extra costs incurred due to a bug.
To clarify, build-minutes are not incurred for the time that a build is in the queue.
Now some of my attempts are returning this, will we have an update for the expired problem?
Error: Cloud Runtime Config is currently experiencing issues, which is preventing your functions from being deployed. Please wait a few minutes and then try to deploy your functions again."
This is a separate issue and has been mitigated. https://github.com/firebase/firebase-tools/issues/7341
You can also request a quota increase for the number of concurrent builds via the Google Cloud Console. Truly sorry for the continued deployment issues. I'm confident that the single builds feature will mitigate the build expiration issue, but it has caused issues in the past and we'd like to make sure we do it right this time.
Now my 2nd gen CFs are deploying perfectly. Something has been fixed or improved... so thank you so much.
Still failing randomly with expired builds for me here... I've had a few instances of builds successfully completing but it's far from being all fixed as far as I can tell, at least as far as my projects go.
I've also requested and been granted quota increases and it doesn't really seem to make the problem go away.
Still failing randomly with expired builds for me here... I've had a few instances of builds successfully completing but it's far from being all fixed as far as I can tell, at least as far as my projects go.
I've also requested and been granted quota increases and it doesn't really seem to make the problem go away.
Same, I think the people who think it's fixed are re-running cached builds or something...
I am using the latest version of Firebase Tools, and in my case, the problem still persists.
If I try to deploy again, the cache activates and indicates that the functions are updated. However, when I inspect the source code of the functions, the previous version with the old bugs is still there.
We urgently need to update a large set of functions, and it is proving to be very challenging to deploy them bit by bit.
Hey folks, some of you may have already experienced improved build performance due to the single builds feature being re-enabled. The feature is almost fully rolled out, but some of you may continue to encounter the build expiration issue until after the holidays.
Note that for firebase-tools >= 13.8.1, you must enable the feature explicitly by running firebase experiments:enable functionsv2deployoptimizations
. Again, single builds hasn't completed its global rollout so some of you may still fail to build and deploy all your functions; but once the rollout is complete (likely at the beginning of next week), we will cut a new release of firebase-tools that will enable the feature again by default.
Thank you for the update. To be clear, are you recommending that developers using the latest CLI should enable the experiment to have the best chance of resolving this issue?
I currently have it disable and usually a single function fails to deploy each time. Would this be resolved by enabling the experiment?
Thank you for the update. To be clear, are you recommending that developers using the latest CLI should enable the experiment to have the best chance of resolving this issue?
I currently have it disable and usually a single function fails to deploy each time. Would this be resolved by enabling the experiment?
Go ahead and try it, for me it seems to work, just ran a deployment set of more than 50 functions, all deployed, no timeouts. Some seem to be in invalid state, still, but at least I can get 40~ OK deployments this way.
Go ahead and try it, for me it seems to work, just ran a deployment set of more than 50 functions, all deployed, no timeouts. Some seem to be in invalid state, still, but at least I can get 40~ OK deployments this way.
What do you mean by invalid state? I'm a bit loathed to turn this on if you're saying 20% of your functions do not deploy properly or do not work after deploying.
With the experiement disabled I'm getting 1/50 failing to deploy, and it's usually a different one each time.
Go ahead and try it, for me it seems to work, just ran a deployment set of more than 50 functions, all deployed, no timeouts. Some seem to be in invalid state, still, but at least I can get 40~ OK deployments this way.
What do you mean by invalid state? I'm a bit loathed to turn this on if you're saying 20% of your functions do not deploy properly or do not work after deploying.
With the experiement disabled I'm getting 1/50 failing to deploy, and it's usually a different one each time.
Invalid state as in, the CLI reports that the function has been "deployed", but google cloud states that the deployment failed.
FYI - Upgraded to CLI version 13.13.0
Deployed 47 functions in one step with no errors.
I'm using the firebase experiments:enable functionsv2deployoptimizations
flag.
Single builds has now been fully rolled out - to enable the feature in the CLI, run firebase experiments:enable functionsv2deployoptimizations
. We'll be cutting a release next week to enable the feature again by default. Thanks for your patience, I'll continue to monitor this thread for problems but I expect that the vast majority of you will see significant improvements in your deploy performance.
Definitely working better with that experiment enabled, but I'm still seeing the expired message for 21 of my 98 V2 functions. Using firebase-tools@13.13.0
Is 98 simply too many for GCP to handle at once?
Still having problems. Experiment turned on, just had 8 out of 22 fail.
I'm still experiencing this with functionsv2deployoptimizations enabled. 8/51 functions "Build failed with status: EXPIRED"
Some feedback:
I have not upgraded the Firebase Tools version, I'm running v13.10.0
and I just deployed - having previously enabled the functionsv2deployoptimizations
patch weeks ago when this issue first arrived - and successfully deployed all my functions which were previously failing:
[REQUIRED] Environment info
firebase-tools: 13.8.0 or 13.10.2
Platform: ubuntu-latest, via GitHubAction
[REQUIRED] Test case
run
firebase deploy --force --only functions,firestore
with a codebase containing 55 2nd Cloud Functions. In my case it is ran via action w9jds/firebase-action@v13.10.2 (or w9jds/firebase-action@v13.8.0)[REQUIRED] Steps to reproduce
Same as test case
[REQUIRED] Expected behavior
I would expect this deployment to go through since it is below the deployment quota of 80 functions per minute.
[REQUIRED] Actual behavior
Some of the functions fail to deploy with build with the following error:
Build failed with status:
EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted].
The build logs are empty and don't contain any useful information other than the build expiring.I can see in Cloud Build console that some of the builds failed:
This happens for about 15 of the 55 functions. Here is a truncated output of the full deploy:
It's actually reopening for #5967 ...