charifield
commented
5 months ago Following from previous ticket. Encountering the same issues
Closed wojtek-iwarranty closed 4 months ago
charifield
commented
5 months ago Following from previous ticket. Encountering the same issues
gersonlimadev
commented
5 months ago The same errors are happening here to me
SherpaMiguel
commented
5 months ago Blocking issue since last week.
toddmotto
commented
5 months ago It amazes me how this is going to badly wrong again @joehan the previous 'fix' suggested by running firebase experiments:enable functionsv2deployoptimizations now does not work, what are we supposed to do to deploy vital function updates?
raymond-tetteh
commented
5 months ago Experiencing the same issues. Can't deploy my functions.
onn-software
commented
5 months ago Same here, 51 functions, and it worked for years. Now I have 'Build failed with status: EXPIRED and message: An unexpected error occurred'.
Edit: Also, Gitlab build went from a successful deploy in ±5 minutes, to a failing deploy in ±12 minutes. Which might be correlated to the EXPIRED error.
halafi
commented
5 months ago same issue, also sometimes deployment is skipped when there are changes to be deployed, thinking to --force deploy each function manually
Endran
commented
5 months ago We encounter the same. Breaking up the build in doesn't work either, each of the groups takes forever.
firebase deploy --only functions:group1
firebase deploy --only functions:group2
firebase deploy --only functions:group3
firebase deploy --except functionsEach function group contains around 10 functions, and takes over 10 minutes, where it used to deploy all in a few minutes.
The last successful deploy was on May 30th, first failing deploy June 2nd. What did the Firebase/GCP team change in the meantime?
wesleygonalv
commented
5 months ago We are having the same issue: we have over 100 functions, but we can only deploy a few of them.
We considered breaking them into smaller groups, but as @Endran mentioned, it unfortunately did not solve the problem.
We have been experiencing this issue since the date mentioned by others and are still waiting for updates.
aalej
commented
5 months ago Hey everyone, thanks for raising this and apologies for the issue this has caused. Our engineering team has been notified about this issue.
While waiting for our engineering team’s response, I’m currently trying to replicate this by deploying ~60 v2 functions, but so far I haven’t been able to encounter the same error. I would like to ask if anyone could share the ff:
functionsv2deployoptimizations enabled?europe-west1, us-central1, etc.onRequest, beforeUserSignedIn, onDocumentWritten, etc.toddmotto
commented
5 months ago @aalej Thanks for the update.
functions.auth.user().onCreate and the remainder V2 - lots of CRUD document ops i.e. onDocumentCreated, onDocumentUpdated and then the rest are custom functions. wesleygonalv
commented
5 months ago Hi @aalej, about your questions.
Do you have the experiment functionsv2deployoptimizations enabled?
As far as I know, we're not using it. Is there a command to check?
Which region are you deploying the function to?
europe-west3
What functions are being deployed? E.g. onRequest, beforeUserSignedIn, onDocumentWritten, etc.
We have
onCall,onRequestandonMessagePublished, all functions for v2
charifield
commented
5 months ago @aalej Thanks for looking into this:
onRequest functions, and about 3 onValueWritten functions for the RTDaalej
commented
5 months ago Thanks for the additional information folks! @wesleygonalv, you can check if functionsv2deployoptimizations by running:
firebase experiments:describe functionsv2deployoptimizationsHi @aalej! Thanks for your help. More info:
wesleygonalv
commented
5 months ago @aalej Thanks for the support,
We are not using functions v2 deploy optimizations. Is there anything else I can help you with?
wojtek-iwarranty
commented
5 months ago Hi @aalej thx for your comment, it gives us hope 😉
- Do you have the experiment
functionsv2deployoptimizationsenabled?Name: functionsv2deployoptimizations Enabled: yes- Which region are you deploying the function to? E.g.
europe-west1,us-central1, etc.
europe-west1, us-central1 and australia-southeast1 (we have the same code and problem in all these regions)
- What functions are being deployed? E.g.
onRequest,beforeUserSignedIn,onDocumentWritten, etc.
45 x onCall (http) + 10 x onDocumentWritten (firestore)
wojtek-iwarranty
commented
5 months ago Hi @aalej , maybe this will help you somehow. After last weekend the new "success" number for Per project mutation requests per minute per region is around 40 (when before it was 55 so all our functions).
tbusot
commented
5 months ago Region: us-central-1
Expiremental Optimizations:
Name: functionsv2deployoptimizations
Enabled: yes
Description: Reuse build images across funtions to increase performance and reliaibility of deploys. This has been made an experiment due to backend bugs that are temporarily causing failures in some regions with this optimization enabledFunction Breakdown:
35 onDocumentCreated, onDocumentUpdated, and onDocumentWritten
2 onSchedule (Adding these is when I started seeing the issues, whether it was causal or coincidence is unknown as now even when I remove them, I still get the errors.)
1 onCreate Auth
4 onCall
1 Expo set of API calls.
When I run firebase deploy --only functions manually, I generally get about 8 functions failed to deploy with the rest succeeding and I can then firebase deploy --only functions:failedFunctionName deploy them successfully.
My CI/CD pipe on Github Actions is completely non-functioning as the whole job fails 100% of the time.
Endran
commented
5 months ago Do you have the experiment functionsv2deployoptimizations enabled?
No, we are using firebase-tools@13.0.2.
Which region are you deploying the function to? E.g. europe-west1, us-central1, etc.
europe-west1
What functions are being deployed? E.g. onRequest, beforeUserSignedIn, onDocumentWritten, etc.
11x onMessagePublished 19x onCall 3x onRequest 1x v1 auth onCreate 1x v1 auth onDelete 13x onDocumentWritten 4x onDocumentCreated 1x onDocumentUpdated 1x onDocumentDeleted 5x onSchedule
aalej
commented
5 months ago Hey folks, just a quick update here. I’m able to reproduce the issue. Thanks for all the information you’ve provided! I’ll mark this issue as reproducible.
Endran
commented
5 months ago Any update on this? It's blocking our releases for a week now, which impacts business. Some relevant logs from the build.
...
success Installed "firebase-tools@13.11.1" with binaries:
- firebase
Done in 22.28s.
$ firebase experiments:enable functionsv2deployoptimizations
Enabled experiment
functionsv2deployoptimizations
$ firebase experiments:describe functionsv2deployoptimizations
Name: functionsv2deployoptimizations
Enabled: yes
...
49x functions: updating Node.js 18 (2nd Gen) function ...
2x functions: updating Node.js 18 (1st Gen) function auth-...
...
12x: Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: ...
...
Error: There was an error deploying functions:
12x: Error: Failed to update functionPlease keep us posted on progress on this blocking issue.
SherpaMiguel
commented
5 months ago Any update on this? It's blocking our releases for a week now, which impacts business. Some relevant logs from the build.
We have the same problem. Yesterday, we deployed to production, which blocked the application for more than 15 minutes. I hope the engineering team is aware of the severity of the bug.
wesleygonalv
commented
5 months ago @SherpaMiguel
We are going through the same thing, and one of the temporary solutions we found was to deploy 4 or 5 functions at a time, we had problems in the production environment and the solution to upload the fixes was this.
firebase deploy --only "functions:functionName, ..."This command can be run on other cloud build runs at the same time 👍
Edit: For example, you can use this command to send the 12 functions that were left out of the deploy
...
12x: Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: ...
...
Error: There was an error deploying functions:
12x: Error: Failed to update function
omgovich
commented
5 months ago Having the same issue (Build failed with status: EXPIRED and message...).
We can't deploy anything (it always fails) and just loosing money because of this bug.
functionsv2deployoptimizations doesn't help.
We deploy 38 functions to us-central1. Most of them are v2 onCall.
Interestingly, our other project (older and bigger), which has 85 functions, has no issues with deployment. Most of the functions there are v1. So I assume the issue is related to v2.
curtis-jotson
commented
5 months ago I'm going to just +1 this. We've been fighting this for weeks. It started in our CD pipeline and now is doing it even with our manual deploys. This might already be known or figured out, but I'll share what I know so far. I am by no means a GCP/Firebase expert or team member so take this for what it is.
Firebase deploy uses Cloud Build to actually build the function containers that run on GCP. These builds have a default (I think) queueTtl configuration. This configuration controls how long a function build sits in the build queue before the build itself expires. It's either configured for or defaulting to 5/6 minutes. Per the Cloud Build documentation, the way I understand it, the default worker pool that actually builds the functions should be limited to 30 concurrent builds (https://cloud.google.com/build/docs/private-pools/private-pools-overview). What we're seeing in the GCP console is that it's only running 5 at a time. We have ~30 functions that are taking ~1-2 minutes per build. At a concurrency of 5, the ones batched last are expiring because they're sitting in the queue for longer than the 5/6 minutes that queueTtl is set to.
In addition to all this, when we do get a failing deploy, subsequent deploys using the functionality that only deploys the functions that need to be updated says all functions are up to date. However upon checking functionally, they are not and have not been deployed.
So my thinking on possibilities of things that have changed are:
queueTtl value has changedAs far as I can tell (with my limited Google-Fu), with firebase deploy we are unable to configure the queueTtl value or the runner type that the builds run on in an effort to speed them up or extend the queue timeout.
cedvdb
commented
5 months ago
- Do you have the experiment
functionsv2deployoptimizationsenabled?- Which region are you deploying the function to? E.g.
europe-west1,us-central1, etc.- What functions are being deployed? E.g.
onRequest,beforeUserSignedIn,onDocumentWritten, etc.
It is successful when forcing a second run in github actions
AvishJ03
commented
5 months ago Facing same issue since a week. Any fixes yet? Tried with both functionsv2deployoptimizations enabled and disabled, not working for both. The region is us-central1 and we are usng onRequest for v2 functions
SherpaMiguel
commented
5 months ago More than a week since the issue was reported again. It's starting to become unacceptable.
toddmotto
commented
5 months ago @SherpaMiguel
Yeah, this is really poor. Considering a fix was previously published and worked absolutely fine, and now broken again with zero fix you have to question what they are even doing.
Endran
commented
5 months ago I tried to contact Firebase Support at https://firebase.google.com/support/troubleshooter/contact?authuser=0 . But they also just mention the same dead end about functionsv2deployoptimizations , and go quiet after the first response. Please try to contact them as well, in hopes the priority is raised.
al4nnw
commented
5 months ago Adding a +1 here.
So far, I've seen this issue in the past and searching google I got back to #5967, but this time going back to 13.8.0 didn't fixed it.
Also, as @onn-software said, the build times went from success in 6 minutes to a failure at 12, which could be related to these expiration errors.
mkdior
commented
5 months ago
- Just throwing it out there but it's possible we've added an external dependency that is extending the build time drastically
I really doubt that, the occurrences of this issue exploded during a specific week, hence the activity in this issue, I think something changed internally that started to cause this. I doubt this many people introduced something affecting their build times drastically in the same period of time.
Wish we could introduce project wide defaults for GCP, allowing us to set timeouts for the cloud builder for example. Anyways, in essence, +1, sick and tired of this issue.
Somebody might want to go through the changelogs and see if something was introduced to any related service on cloud platform in the beginning of June, as as @al4nnw mentioned, early August 2023 as indicated in #5967 .
hasbean
commented
5 months ago +1. I'm not sure how Firebase team doesn't see this as a top priority. I've been stuck with undeployed functions for weeks now.
toddmotto
commented
5 months ago I'm baffled too that this isn't 1) a high priority and 2) already fixed within a day or so. This is a fundamental building block of Firebase and with no update in days and days it's difficult to question the reliability of the platform.
It's also not like this is a new 'bug' that needs tracking down, refactoring and deploying, they literally already fixed it - and broke it again.
cedvdb
commented
5 months ago @toddmotto it's may be high priority, but we don't know for sure since there is no label on this issue.
The only thing we can do is +1 the top comment and contact their support to increase the priority. I'd also like some reassurance that this is high priority.
ecartaxo
commented
5 months ago Hi everyone,
We are facing the same problem with deploying our Cloud Functions.
Is there any update or known workaround for this issue? Any help would be greatly appreciated.
Thank you!
mandalornl
commented
5 months ago @ecartaxo As a workaround you can try deploying in small batches. This has worked for me. I'd suggest 5 functions at a time. The only caveat is, it'll take forever to complete when you have many many many functions to deploy :sleeping: But you can easily automate the process if you know your way around e.g. Node.js.
CarlosDeucherMoretoCont
commented
5 months ago +1 I'm facing the same issue here implementing our 88 Cloud Functions ):
gersonlimadev
commented
5 months ago I'm facing the same issue for almost 2 weeks now, it's crazy
blidd-google
commented
5 months ago Hey folks, really sorry to hear about all the build-related errors you're seeing. We've been investigating the issue and reached out to our Cloud partner teams to understand why builds are expiring at an abnormally high rate.
We recently had to disable the single builds feature, which allowed multiple functions to re-use builds and thus significantly reduces the number of builds per function deployment. I suspect that the rollback has compounded the severity of the expiration problem you all are facing, and we're actively working on re-enabling the feature.
Again, sorry to hear about these deployment issues; we appreciate your patience, and I'll keep this thread updated on our progress, both for our investigation into the underlying issue causing the number of concurrent builds / queue TTLs to decrease significantly and for our effort to safely re-enable single builds.
kdawgwilk
commented
5 months ago Are there any workarounds we can implement ourselves other than breaking the function deploys into groups of 5? Is there a way for us to increase the timeouts? If we built a single function and deployed it would the next deploy be able to use the first function build cache?
toddmotto
commented
5 months ago 2 weeks ago this issue was opened... unable to deploy still, lost for words a bit on this.
cedvdb
commented
5 months ago Deploying usually works when running the deployment multiple times back to back. So you can make a shell script that runs firebase deploy 3 times. It will take longer but you'll be able to deploy
mkdior
commented
5 months ago Deploying usually works when running the deployment multiple times back to back. So you can make a shell script that runs firebase deploy 3 times. It will take longer but you'll be able to deploy
Each deployment consumes builder minutes as far as I'm aware, which costs us. There should be no extra costs incurred due to a bug.
wesleygonalv
commented
5 months ago Now some of my attempts are returning this, will we have an update for the expired problem?
Error: Cloud Runtime Config is currently experiencing issues, which is preventing your functions from being deployed. Please wait a few minutes and then try to deploy your functions again."
GabiGrin
commented
5 months ago Same here
Endran
commented
5 months ago Ahhh.. I have a high prio issue I need to push, and instead of 10 minutes build and 5 minutes deploy (front end and backend) it takes 10 + 27 minutes to go live. I broke up everything in smaller batches, but it's taking ages. Its been 3 weeks now since our last normal deploy, please fix it asap.
wesleygonalv
commented
4 months ago We have a similar case @Endran,
we had to deploy more than once to correct production problems, with the risk of losing the partner company :(
MNITD
commented
4 months ago Hey @blidd-google
It's a week since your last message. Is there any bad or good news regarding the resolution of the issue?
Also, I believe many people will be happy to hear an estimate on when we can expect at least a workaround (as it was with downgrading the CLI to 13.8.0)
[REQUIRED] Environment info
firebase-tools: 13.8.0 or 13.10.2
Platform: ubuntu-latest, via GitHubAction
[REQUIRED] Test case
run
firebase deploy --force --only functions,firestorewith a codebase containing 55 2nd Cloud Functions. In my case it is ran via action w9jds/firebase-action@v13.10.2 (or w9jds/firebase-action@v13.8.0)[REQUIRED] Steps to reproduce
Same as test case
[REQUIRED] Expected behavior
I would expect this deployment to go through since it is below the deployment quota of 80 functions per minute.
[REQUIRED] Actual behavior
Some of the functions fail to deploy with build with the following error:
Build failed with status:
EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted].The build logs are empty and don't contain any useful information other than the build expiring.I can see in Cloud Build console that some of the builds failed:
This happens for about 15 of the 55 functions. Here is a truncated output of the full deploy:
It's actually reopening for #5967 ...