4 out of 13 builds fail with: Build failed with status: EXPIRED and message: An unexpected error occurred

berengamble commented 2 months ago

Environment info

firebase-tools: 13.13.1

Platform: Mac OS Sonoma 14.0

Test cases

1. firebase experiments:disable functionsv2deployoptimizations firebase deploy --only functions:main with a codebase containing 13 2nd generation Cloud Functions. 2 firebase experiments:enable functionsv2deployoptimizations firebase deploy --only functions:main with a codebase containing 13 2nd generation Cloud Functions.

Steps to reproduce

Same as test cases

Expected behavior

All functions successfully deployed.

Actual behavior

In both cases, after running the above commands and waiting about 10 minutes, 4-5 of the functions will complete successfully fairly early in the build, then after a few more minutes, the rest fail with errors like this, then the messages saying a couple more successfully deployed, all together in one big chunk of logs at the end.:

Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: https://console.cloud.google.com/cloud-build/builds;region=us-central1/90e2bdef-d529-4dbe-be9e-355a18fc835e?project=1033623241793. For more details see the logs at https://console.cloud.google.com/cloud-build/builds;region=us-central1/90e2bdef-d529-4dbe-be9e-355a18fc835e?project=1033623241793.

The confusing thing is, in GCP Console, there's a ⚠️ sign on each function that errored and this is shown: Function is active, but the last deployment failed.

There is nothing useful in the build logs, as outlined in these two tickets: https://github.com/firebase/firebase-tools/issues/5967 & ~~https://github.com/firebase/firebase-tools/pull/5969~~ https://github.com/firebase/firebase-tools/issues/7268

google-oss-bot commented 2 months ago

This issue does not seem to follow the issue template. Make sure you provide all the required information.

Nushio commented 2 months ago

This was seemingly fixed in https://github.com/firebase/firebase-tools/issues/7268 , however the fix seems to be a slow rollout.

I can confirm that on my end, I haven't had any issues since around July 4, but maybe the fix is still rolling out?

berengamble commented 2 months ago

Ah yes, that was the issue I meant to reference above. I've updated my original comment.

Despite these two issues being resolved, I am still experiencing this problem.

halafi commented 2 months ago

I still have the issue https://github.com/firebase/firebase-tools/issues/7268

Name: functionsv2deployoptimizations Enabled: yes

tools v13.13.1

aalej commented 2 months ago

Hey folks, sorry to hear you’re still encountering this issue. I’m discussing this with our engineering team to see what could be causing the issue.

In the meantime, to unblock you from updating your functions, could you try deploying using firebase deploy --only functions:<function_name>, or firebase deploy --only functions:<codebase>:<function_name> if you're using codebases, for the functions which encountered an error.

Function is active, but the last deployment failed.

The function is still running and processing requests, however updating the function's code with the latest deployment failed. This means the function is still running the older version of the code.

berengamble commented 2 months ago

I can confirm that deploying each function individually does result in a successful deployment. While this is a workaround, this level of hand-holding for each deploy is exactly optimal. Any idea when this will be fixed?

berengamble commented 1 month ago

Any progress on this? It's still happening for me on every. single. deploy. I'm using 13.14.1 now.

Ekhogecko commented 1 month ago

This is also happening for our project of ~80 functions deployed across v1 and v2

firebase / firebase-tools