Deployment error with no details after updating to Node 8 and firebase-tools@^4.0.0

jpreynat commented 6 years ago

Version info

^4.0.0

Platform Information

Ubuntu 14.04.5 LTS (on Travis CI)

Steps to reproduce

firebase deploy --except hosting with lots of functions (in our case 52) running on Node 8 runtime.

Expected behavior

Before we updated firebase-tools to version 4 and higher (we had the exact same behavior for versions 4.0.0, 4.0.1 and 4.0.2) and using Node 8, updating correctly all of our cloud functions never failed. We expect it to continue since this happens when deploying our application in production.

Actual behavior

Since this update, there is always at least one or two functions that fail deploying with the following error:

⚠  functions[<our-function-name>(us-central1)]: Deployment error.
Build failed: Build error details not available

During the last deployment running with firebase-tools@4.0.2, 4 functions failed to update.

chuan-qin commented 4 years ago

Hey @thechenky, I did:

Build failed: {"cacheStats": [{"status": "MISS", "hash": "{{hashCode}}", "type": "docker_layer_cache", "level": "global"}, {"status": "MISS", "hash": "{{hashCode}}", "type": "docker_layer_cache", "level": "project"}]}

I waited about 5 mins as I read this thread online, deployed again and it worked fine.

Is there a reason why the build might still fail at times?

Still experiencing exactly the same issue today. (2020 - 01 - 07)

luanch commented 4 years ago

Same here. But it doesn't worked after waiting for some time...

sulfurcode commented 4 years ago

Hey @thechenky, I did: Build failed: {"cacheStats": [{"status": "MISS", "hash": "{{hashCode}}", "type": "docker_layer_cache", "level": "global"}, {"status": "MISS", "hash": "{{hashCode}}", "type": "docker_layer_cache", "level": "project"}]} I waited about 5 mins as I read this thread online, deployed again and it worked fine. Is there a reason why the build might still fail at times?

Still experiencing exactly the same issue today. (2019 - 01 - 07)

Yep I have the same issue as well

vazkir commented 4 years ago

Hey @thechenky, I did: Build failed: {"cacheStats": [{"status": "MISS", "hash": "{{hashCode}}", "type": "docker_layer_cache", "level": "global"}, {"status": "MISS", "hash": "{{hashCode}}", "type": "docker_layer_cache", "level": "project"}]} I waited about 5 mins as I read this thread online, deployed again and it worked fine. Is there a reason why the build might still fail at times?

Still experiencing exactly the same issue today. (2019 - 01 - 07)

*2020-01-07 probably;) And yes I am too experiencing this for some time now..

lunaroyster commented 4 years ago

Yep, just experienced this, and landed here from search results

Build failed: {"cacheStats": [{"status": "MISS", "hash": "{{hashCode}}", "type": "docker_layer_cache", "level": "global"}, {"status": "MISS", "hash": "{{hashCode}}", "type": "docker_layer_cache", "level": "project"}]}

I think it's a different issue?

sulfurcode commented 4 years ago

Google just confirmed they are having deployment issues with cloud functions

cristianovagos commented 4 years ago

Same problem here. Bad time for that tho

Build failed: {"cacheStats": [{"status": "MISS", "hash": "{{hash}}", "type": "docker_layer_cache", "level": "global"}, {"status": "HIT", "hash": "{{hash}}", "type": "docker_layer_cache", "level": "project"}]}

zohaib-sarwar-confiz commented 4 years ago

Hi,

I am also getting a similar error

Deployment error. Build failed: {"cacheStats": [{"status": "MISS", "hash": "{hashCode}", "type": "docker_layer_cache", "level": "global"}, {"status": "HIT", "hash": "{hashCode}", "type": "docker_layer_cache", "level": "project"}]}

sulfurcode commented 4 years ago

From Google Support

"We're currently having an issue with Cloud Functions deployment. Our engineering team is currently working on this to put back the services the soonest possible time. I'll get back to you with more updates.

Apologies for the inconvenience this may have caused you."

hafiz-mohsin-confiz commented 4 years ago

I have just checked the google services on the following link that they have claimed all are up. https://status.cloud.google.com/ We have to spend three hours for issue investigation. Please update the confirm status with some URL here Thanks,

sulfurcode commented 4 years ago

I have just checked the google services on the following link that they have claimed all are up. https://status.cloud.google.com/ We have to spend three hours for issue investigation. Please update the confirm status with some URL here Thanks,

They've updated the page now

jpinilloslr commented 4 years ago

I managed to deploy all my functions after a lot of retries but it's still unstable though

WadhahEssam commented 4 years ago

😖

Kian-Aghaei commented 4 years ago

We still have the issue, both on app engine and cloud function

WadhahEssam commented 4 years ago

looks like they are fixing something

taronaeo commented 4 years ago

Same here. Also having the same problem:

!  functions[createUserAccount(us-central1)]: Deployment error.
Build failed: {"cacheStats": [{"status": "MISS", "hash": "e03d8d6f6bf22fcb1cf50f4e12b3e3a3b59954fade707db62e79562c9dbac3ef", "type": "docker_layer_cache", "level": "global"}, {"status": "HIT", "hash": "e03d8d6f6bf22fcb1cf50f4e12b3e3a3b59954fade707db62e79562c9dbac3ef", "type": "docker_layer_cache", "level": "project"}]}

Functions deploy had errors with the following functions:
        createUserAccount

with the following code:

await database.collection('members').where('Email', '==', user.email).limit(1).get()
    .then(snapshot => {
      snapshot.forEach(doc => {
        console.log(doc.id, '=>', doc.data())
      })
    }).catch(err => console.error(err))

Hritik14 commented 4 years ago

I happened to have a backup file in the functions directory. Removing that file seemed to fix the deploy.

taronaeo commented 4 years ago

I happened to have a backup file in the functions directory. Removing that file seemed to fix the deploy.

Backup file as in functions/lib?

juank11memphis commented 4 years ago

I can confirm that removing the folder functions/lib fixed the issue for me :)

coolcool1994 commented 4 years ago

Is it safe to delete functions/lib ? Does it get regenerated every time with firebase deploy?

taronaeo commented 4 years ago

Is it safe to delete functions/lib ? Does it get regenerated every time with firebase deploy?

It is completely safe to delete that folder. Whenever you run firebase deploy, it will automatically create that folder with the contents inside.

coolcool1994 commented 4 years ago

Deleting functions/lib didn't fix anything for me. Still getting functions[getVerificationCode(us-central1)]: Deployment error. Build failed: {"cacheStats": [{"status": "MISS", "hash": ...

Error: Functions did not deploy properly.

taronaeo commented 4 years ago

Deleting functions/lib didn't fix anything for me. Still getting functions[getVerificationCode(us-central1)]: Deployment error. Build failed: {"cacheStats": [{"status": "MISS", "hash": ...

Error: Functions did not deploy properly.

Your best bet is to keep re-trying the request. I've had the same issue as well but all I needed to do is keep running firebase deploy until it worked.

P.S. It might take a while for it go through -- not sure what Google's Team is up to tbh.

coolcool1994 commented 4 years ago

I have one function that doesn't get deployed/updated even after retrying over and over. But just that one function. I guess that's fine for me now since I am not testing that function.

El-Fitz commented 4 years ago

https://status.cloud.google.com/ still pretends everything is up But last I tried, none of my functions deployed 😬

samtstern commented 4 years ago

The status page currently says:

We've received a report of an issue with Google Cloud Functions and Google App Engine deployments failing for some customers.

The team is investigating and will post an issue on the dashboard if it's warranted as they learn more. In the meantime if you're having a deployment issue with Cloud Functions please write in to Firebase or Google Cloud support! Thank you all for your patience.

guzzz-12 commented 4 years ago

Same here. Any attempt to deploy even only the default helloWorld firebase function throws the same error: Deployment error. Build failed: {"cacheStats": [{"status": "MISS", "hash": "...", "type": "docker_layer_cache", "level": "global"}

bytewiz commented 4 years ago

Same here. Any attempt to deploy even only the default helloWorld firebase function throws the same error: Deployment error. Build failed: {"cacheStats": [{"status": "MISS", "hash": "...", "type": "docker_layer_cache", "level": "global"}

I am seeing this too

stuntmobile commented 4 years ago

Same here. Deployment error. Build failed: {"cacheStats": [{"status": "MISS", "hash": "hash", "type": "docker_layer_cache", "level": "global"}, {"status": "MISS", "hash": "hash", "type": "docker_layer_cache", "level": "project"}]}

davaya commented 4 years ago

https://status.cloud.google.com/incident/cloud-functions/20001

evoships commented 4 years ago

After a whole night of trying to find my mistake on what went wrong finally i was able to deploy my functions. Worked for me 👍

El-Fitz commented 4 years ago

AND IT WORKS !

MikeWeiZhou commented 4 years ago

Sorry to bring this topic back up. But I do have the same problem.

Everytime I deploy, anywhere from 1 - 5 functions would fail with message like this:

There isn't a function that consistently fails to deploy, it seems like random functions fail at different times. Maybe 1 in 12 tries I might get all the functions to deploy at once.

Any hints as to any work arounds besides deploying multiple times for failed functions? And I appreciate all the work thats put into this tool. TY

pragati27gupta commented 4 years ago

Build failed: {"cacheStats": [{"status": "MISS", "hash": "d77e8c491b10d7f6c295d44d41963365d20c4b63b5a98586acce25c13826ff34", "type": "docker_layer_cache", "level": "global"}, {"status": "HIT", "hash": "d77e8c491b10d7f6c295d44d41963365d20c4b63b5a98586acce25c13826ff34", "type": "docker_layer_cache", "level": "project"}]}

Facing the same issue today. Was working well till yesterday.

taronaeo commented 4 years ago

Those error codes are the exact ones we received previously when Google's Cloud Function systems had an internal error of themselves.

My best solution for you is either:

Keep attempting to deploy until it works (should take not more than 14 retries)
Contact Google about the issue and let them check their infrastructure for issues

selvinfehric commented 4 years ago

I am still getting the same error about HIT and MISS on cache stats. This is happening whole day.

Guys, please report deploy issues here, so they might resolve it finally. @pragati27gupta, @taronaeo, @acoyfellow, @MikeWeiZhou https://firebase.google.com/support/troubleshooter/functions/deploy

RWOverdijk commented 4 years ago

So, firebase uses docker. That's fun to know.

Thanks for reaching out! Your request has been submitted and we'll reach out to you soon.

@selvinfehric has the right idea.

jaceq commented 4 years ago

btw. I opened issue at cloudfunctions issue tracker about that: https://issuetracker.google.com/issues/154260223 Please, comment / star the issue to get more traction there.

Cdik commented 4 years ago

Same here!

emadalam commented 4 years ago

Same here, we have the latest version of firebase tools running on a GitLab runner and this keeps happening. It's such a big annoyance to keep retrying. Our whole CI/CD pipeline is broken due to this, causing such a waste of developer time, tracking the jobs and clicking retry to eternity.

Worst of it, the Firebase Support team hasn't been helpful at all and have pointed to some bunch of random docs, totally ignoring the randomness of function deployment errors.

oviliz commented 4 years ago

Happens to us too, when locally on Ubuntu 16.04 with firebase-tools 7.12.1 or via Azure DevOps Task with the latest 8.1.1. Ridiculous that this issue is closed and nobody (Google side) seems to do care less.

neilpoulin commented 4 years ago

this is happening to us, too. We've been battling it for nearly a year. We only have about 30 functions (of all types, https, firestore triggers, pub sub subscribers, auth triggers). It mostly happens when we deploy via CI/CD (circleci), but it happens sometimes when we deploy from our local machines, too (although we do it so infrequently it's hard to say).

I'm running node 8, because of a bug firebase had a few months ago with node 10. Typically, probably 8 or 9 times out of 10, one or two of our functions will just fail to deploy. It seems to be a random one each time - never a consistent failure. This happens without making any changes to our functions code - we may just be deploying a front-end change (via hosting, but we want to keep versions in sync across files via hosting and the API). So, we know it's not our code.

I finally bit the bullet today and spend half of the day breaking out our "large number" (~30) of functions into smaller groups that deploy on their own, with each group having less than 10. THIS SITLL DOES NOT WORK. I've had several failures in a row. in my first group of functions, which only has 6 items. This "group" is actually declared using the firebase deploy --only functions:func1,functions:func2,...functions:func6 syntax, not a group defined in the main export.

chenlevy commented 4 years ago

Same here Deployment error. Anywhere between 1-5 different functions fail to deploy: Build failed: {"cacheStats": [{"status": "MISS", ...

mbleigh commented 4 years ago

Anyone experiencing this error please +1 and weigh in on this from the public Cloud issue tracker: https://issuetracker.google.com/issues/154260223

On Sat, Apr 25, 2020, 4:49 AM chenlevy notifications@github.com wrote:

Same here Deployment error. Anywhere between 1-5 different functions fail to deploy: Build failed: {"cacheStats": [{"status": "MISS", ...

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/firebase/firebase-tools/issues/853#issuecomment-619367267, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAAH7RZHEUCY2EAR2ABU3DROLE23ANCNFSM4FNSPUCA .

erykos111 commented 4 years ago

I have found that, when creating new project, for my previous one I didn't have the "google analytics" enabled, and every single of my deployments was failing. I just tested it with brand new project and everything works, when I do enable google-analytics

mbleigh commented 4 years ago

Whether Analytics is enabled or not will not affect this error, that's likely just been a coincidence.

On Sat, Apr 25, 2020, 12:17 PM erykos111 notifications@github.com wrote:

[image: image] https://user-images.githubusercontent.com/47335304/80288700-94ee1680-8731-11ea-8f57-66598893458e.png

I have found that, when creating new project, for my previous one I didn't have the "google analytics" enabled, and every single of my deployments was failing. I just tested it with brand new project and everything works, when I do enable google-analytics

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/firebase/firebase-tools/issues/853#issuecomment-619427952, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAAH7TAOV2IOQG7NUDBXFTROMZMHANCNFSM4FNSPUCA .

taronaeo commented 4 years ago

As stated in the above comment, enabling Google Analytics does not change the failed deployment of functions. What you have probably done is created a new project in a server that does not have their Cloud Functions server failing to operate.

dudemeister commented 4 years ago

Just ran into the same issue - deployments were working fine until we got these, reliably on two functions - started sometime today:

⚠ functions[legacyFeedRedirect(us-central1)]: Deployment error. Build failed: {"cacheStats": [{"status": "MISS", "hash": "526b4459b8b6ae9cc7de4bbed53b2d24a8c5fcf1737f07f27a44e4029d472306", "type": "docker_layer_cache", "level": "global"}, {"status": "HIT", "hash": "526b4459b8b6ae9cc7de4bbed53b2d24a8c5fcf1737f07f27a44e4029d472306", "type": "docker_layer_cache", "level": "project"}]} ⚠ functions[updateUserOnboardingFlags(us-central1)]: Deployment error. Build failed: {"cacheStats": [{"status": "MISS", "hash": "526b4459b8b6ae9cc7de4bbed53b2d24a8c5fcf1737f07f27a44e4029d472306", "type": "docker_layer_cache", "level": "global"}, {"status": "HIT", "hash": "526b4459b8b6ae9cc7de4bbed53b2d24a8c5fcf1737f07f27a44e4029d472306", "type": "docker_layer_cache", "level": "project"}]}

jpreynat commented 4 years ago

To anyone that still encounter deployment errors:

After first fixing our deployments by switching on Node 10 and running with the latest version of firebase-tools, we recently (about 2 weeks ago) encountered errors again with an opaque Build error: details not available. We were completely stuck because this happened constantly for all of our functions on all our environments.

After investigating, we noticed that the build time for cloud functions had been reduced on GCP from 10 minutes to only 5 minutes.

To improve build time, GCP uses caching to re-install node_modules based on the hash of either the package-lock.json or the yarn.lock files, depending on which you are using. But if your dependencies are updated and this file changes, GCP has to re-install all of them, which was taking too long in our case, especially since we have many native modules that need to be rebuilt. I think this is what we can see in the Node 8 error details with the "status": "MISS".

We ended up building our own tooling to compile our functions separately, bundling all non-native modules in our main function code using webpack, and having a trimmed package.json file for each function with only the missing dependencies for each of them.

Now our deployments work well every time, and take 1 to 2 minutes max per function.

nes123 commented 4 years ago

+1

firebase / firebase-tools