Deployment error with no details after updating to Node 8 and firebase-tools@^4.0.0

[jpreynat]

Version info

^4.0.0

Platform Information

Ubuntu 14.04.5 LTS (on Travis CI)

Steps to reproduce

firebase deploy --except hosting with lots of functions (in our case 52) running on Node 8 runtime.

Expected behavior

Before we updated firebase-tools to version 4 and higher (we had the exact same behavior for versions 4.0.0, 4.0.1 and 4.0.2) and using Node 8, updating correctly all of our cloud functions never failed. We expect it to continue since this happens when deploying our application in production.

Actual behavior

Since this update, there is always at least one or two functions that fail deploying with the following error:

⚠  functions[<our-function-name>(us-central1)]: Deployment error.
Build failed: Build error details not available

During the last deployment running with firebase-tools@4.0.2, 4 functions failed to update.

[laurenzlong]

Hi, I think the issue is likely due to the number of functions you are deploying, and not due to the firebase-tools version (you can of course test this by downgrading). Please deploy a subset of your functions at once instead of all of them, see https://firebase.google.com/docs/cli/#deploy_specific_functions

[mlake]

getting the same error and only deploying 12 functions. Never had these deployment issues until upgrading to node 8.

..if there's a limit to the number you can deploy, shouldn't it be documented somewhere?

[itaydressler]

Same here. Getting it when deploying only one function with node 8.

[laurenzlong]

Ok interesting, thanks for the additional datapoints. So it seems like it's not due to the new versions of firebase-tools per se, but due to deploying to Node 8 vs Node 6. (So using the same firebase-tools but deploying to Node 6 will reduce errors). I've filed a bug internally with the team responsible for the runtime.

[0x80]

This is a pretty annoying bug. My function deployments fail 9 out of 10 times now it seems 😞. Glad to hear it's on peoples radar though. I'm moving back to Node.js 6 for now.

[ahaverty]

@jpreynat @0x80 Does the entire deployment fail or some functions deploy and some fail? (Investigating whether we should upgrade our 100+ functions to Node v8)

[jpreynat]

@ahaverty In my case, this is only 3 or 4 functions on 52 that fail. But I'd rather have the deployment fail completely so all our application functions are always on the same version.

[ahaverty]

@jpreynat I agree, we're running deploys via CI, and could live with complete fails but not partial. Thanks for confirming, I'm going to hold off upgrading until this is resolved.

[jpreynat]

This is still happening and it affects our staging and production environments. We just redeployed the whole application a minute ago, and one of the function failed deploying for no apparent reason.

As @ahaverty also states, the CLI should at least rollback all functions that were successfully deployed to prevent instability in the whole application. @laurenzlong Do you have any update about the reason of the failure and an ETA for a fix release? We are about to deploy a critical update to production and need to be sure that it is either fully or not deployed at all, so we can relaunch it and the app won't crash.

[mbleigh]

I would recommend deploying your functions in small groups or individually to minimize potential bad partial deploy states. Rollback is not a feature on the near-term horizon.

On Thu, Aug 16, 2018, 10:07 AM Johan Preynat notifications@github.com wrote:

This is still happening and it affects our staging and production environments. We just redeployed the whole application a minute ago, and one of the function failed deploying for no apparent reason.

As @ahaverty https://github.com/ahaverty also states, the CLI should at least rollback all functions that were successfully deployed to prevent instability in the whole application. @laurenzlong https://github.com/laurenzlong Do you have any update about the reason of the failure and an ETA for a fix release? We are about to deploy a critical update to production and need to be sure that it is either fully or not deployed at all, so we can relaunch it and the app won't crash.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/firebase/firebase-tools/issues/853#issuecomment-413557599, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAD_gn6e0hkZXlbiM_hv-oBf2T8_Iujks5uRXySgaJpZM4Vsn0E .

[lukepighetti]

I can confirm that I am also having this issue with Node 8. I will switch back to Node 6.

After switching to Node 8...

1. ~3% invocation failure
2. ~4x increased time to deploy
3. ~80% deploy failure due to 2 of 24 functions failing to deploy

https://stackoverflow.com/questions/52056953/functions-issues-with-node-8-lts/52058796#52058796

I switched to Node 8 to gain access to the spread operator so I could build a library like this. I'm not sure if this is contributing to my lack of success with Node 8.

https://stackoverflow.com/questions/43486278/how-do-i-structure-cloud-functions-for-firebase-to-deploy-multiple-functions-fro/52056785#52056785

[NEO97online]

I'm having the same issue on firebase-tools v4.2.1 using Node 8.

Downgrading back to Node 6 allowed me to deploy.

[mulhoon]

@michaelauderer Same happening here. Seems to be flakey over the past few hours so it might be a cloud issue.

[urkopineda]

@michaelauderer @mulhoon Same here with Node 8.

[yuliankarapetkov]

I have the same issue, but it's weird because these very same functions (4 out of 4) used to deploy successfully yesterday.

[mulhoon]

@yuliankarapetkov Yes, mine were working fine yesterday. FYI I've been installing node modules with yarn on node v8.9.4, then switching to node v9.4.0 and deploying. Seemed to be going well.

[gautier-gdx]

Same here, but I had this working this morning, I think for our case, this is related to the GCF status right now: https://status.firebase.google.com/incident/Functions/18034 @mulhoon @yuliankarapetkov @urkopineda @michaelauderer

[jojonarte]

woa I thought I'm the only one having this issue. How did you guys solve this?

[jpreynat]

@jojonarte There is no quick fix currently, the current massive errors are due to the Firebase platform encountering a service disruption as stated by @gautier-gdx. However, we still have problems deploying all our functions at once with Node 8 when Firebase is up.

[lukepighetti]

@jojonarte the only resolution I found was to refactor back to Node 6, deploy an empty index.js to delete all functions (otherwise they remain Node 8 functions), and then deploy the Node 6 index.js

[yuliankarapetkov]

Yes, the problem is that if you have a lot of functions that make use of async/await (or any other Node 8 features), you'll have to spend tremendous amount of time refactoring them.

[lumyus]

Same here. Node 8 tried to deploy 13 functions at once. This issue did not occur yesterday with the same amount of functions.

[nandenjin]

Same problem here. It occurs when tried to update two exist Node 6 functions to Node 8 in this 12 hours. Using firebase-tools@4.2.1, Solved by deleting exist functions by deploying empty index.js and re-deploying with configuration "engines": { "node": "6" } inside package.json as it was described previous comments.

[deremer]

Any updates on this? This issue is happening for us still as of 9/25. We have 47 functions and anywhere from 0-5 on average will fail to deploy.

[mulhoon]

It's working for me, but this really isn't great for firebase functions.

Does anyone have any good techniques for using a dev environment? e.g. a blue-green deployment to reduce downtime. At the moment I deploy to the live functions, so a deployment failure leads to real app downtime. Thanks

[lukepighetti]

You could blue-green with two Firebase apps.

[kevendra]

+1 Same problem

[radixdev]

Still ongoing.

Build failed: Build error details not available

[thechenky]

Hi all, thank you for these reports. We are aware of this issue and there is ongoing work in the pipeline to mitigate this pain point. I cannot share timelines, but please know we hear you, we know this issue is frustrating, and we're prioritizing work that will reduce these errors.

Internal bug reference: 117124663

[lukepighetti]

Late is better than never. Thanks for acknowledging the issue.

[sboyd]

Since this is a known issue it'd probably be helpful if it's mentioned in this blog post https://firebase.googleblog.com/2018/08/cloud-functions-for-firebase-config-node-8-timeout-memory-region.html where it talks about the steps required to upgrade, but doesn't mention that you probably shouldn't.

[jpreynat]

Actually, everything seems fine now. I think that Google has stabilized the Cloud Function Node 8 runtime since I opened the issue. However, I’ll let the Firebase team close the issue when they find it relevant.

[sboyd]

I just switched over to the lastest version of firebase tools, Node 8, and firebase functions last night and unfortunately ran into the same issues mentioned here.

[mbifulco]

Same - this seems to have cropped up for me on a deployment last night as well 😭

[Bonitis]

We have been using node 8 on 32 functions since it was available and only started having this issue last night. Deploying fails for between 1 and 5 functions, and they seem to be different every time.

[jpreynat]

Doesn’t seem to match your timing, but there was an issue with Cloud Functions deployments on March 11th.

[sboyd]

I suppose it brings some comfort knowing that maybe it was a "one off", but had I been deploying to production that would have really made a mess. Some were successful others weren't and it was so much slower than the node 6 deployment.

[jpreynat]

Sure, but this kind of platform problem could also occur when deploying on the Node 6 runtime. However, what I think firebase-tools is critically missing, that our team has been pushing for a long time and the community should really engage into is transactional deployments (see #699).

As described in the linked issue, we’ve set some best practices in place in our deployment process to prevent critical errors and have been lucky enough that our production deployments always ended well. But we clearly shouldn’t be worried everytime we are releasing our app.

[sboyd]

I can't disagree with that at all it "could" occur when deploying on Node 6, but in over the year + since my application has been in production and running on Node 6 I just hadn't seen it. I'll take a look at #699 for sure! Have a great evening @jpreynat

[jpreynat]

Thanks. For information, we opened this issue because it happened for our deployments on Node 6 too. Anyways, good to see people wanting and pushing improvements on this matter too! Good evening to you too @sboyd

[ahaverty]

Hi @thechenky, has there been any update? This bug is starting to affect our project as other libraries start dropping node v6 support, we're having to do a lot of workarounds staying on node v6, with this bug being the only blocker from upgrading to v8.

[thechenky]

Hi @ahaverty sorry to hear you're still having this issue. I've re-read through this thread and am trying to make sure I have the right understanding of the issue. Does this problem affect only Node 8 deployments? And does this issue go away sometimes or is it always present? I have been unable to recreate this with Node 8 deployments, as the functions I've tried to deploy with Node 8 seem to deploy okay.

The bug that I referenced earlier in https://github.com/firebase/firebase-tools/issues/853#issuecomment-456701673 that should make this situation better is more of an infrastructure change in how we build and deploy the function that should make deployments faster. I'm curious now to see whether the issues folks have been seeing are isolated to the Node 8 runtime. I will bring this up with the relevant teams and see if we can get more information on this. Thanks for everyone's patience!

[thechenky]

Can someone confirm if this is still happening with the latest firebase-tools and firebase-functions?

Also, if anyone is willing to share their code, we can see if we can deploy that function in Node8 to see whether this is something source-related.

From the graphs we are seeing similar deployment success rates for Node 6 and Node 8, so we think this might be impacting specific projects (maybe some projects got in a bad state) - does this occur on a new project that is created? There have been a couple Cloud Build outages that would affect the build step in the deployment of a Node 8 function, but that would not explain constant inabilities to deploy Node 8 functions. I would advise to help us investigate further, that people who are still having consistent issues deploying with Node 8 open a support ticket (report deploy related functions issues at https://support.google.com/firebase/contact/support?page=/functions/deploy) so that we can dig deeper into your specific project and take a look at what's going on.

[u007]

im using node v11 lol, updated firebase-tools, initial pubsub.schedule didnt deployed, rerun works now

[thechenky]

@u007 that's great to hear that your issue was resolved! Is anyone else still experiencing these issues (with updated firebase-functions and firebase-tools)?

[Adrian-Samuel]

Hey @thechenky, I did:

Build failed: {"cacheStats": [{"status": "MISS", "hash": "{{hashCode}}", "type": "docker_layer_cache", "level": "global"}, {"status": "MISS", "hash": "{{hashCode}}", "type": "docker_layer_cache", "level": "project"}]}

I waited about 5 mins as I read this thread online, deployed again and it worked fine.

Is there a reason why the build might still fail at times?

[thechenky]

@Adrian-Samuel this may have been a transient error - hopefully you don't run into this again. If you do, please open a new issue.

I'm going to close this out as it looks like the original issue has been resolved. Please feel free to open a new issue if you encounter any other problems.

[afuggini]

I am having this problem out of the blue, without having changed anything on my end. I believe something changed recently and now I am unable to deploy. My package.json declares node 8, and I am running node 8 on my computer.

[samtstern]

@afuggini if you're having deploy errors please file a new issue or write in to Firebase support.

[janakaud]

If you are a newbie hitting this issue (like myself), double-check your function definitions.

In my case, there were some issues with function import paths in the root-level index.js, and I was getting this exact same error upon deployment.

This was on raw Cloud Functions, but it would probably be applicable for any other platforms built on the same - like Firebase Functions.