Flaky Cloud Function v2 deploy - `There was an error retrieving the Firestore database`

[kaueDM]

[REQUIRED] Environment info

firebase-tools: 13.15.2

Platform: Windows

[REQUIRED] Test case

Try to deploy Cloud Functions V2 (mix between callables, https and Firestore triggers)

firebase deploy --only functions

Currently I have more than 70 cloud functions in my project. Doesn't matter if I'm trying to deploy 1 or all of them, 80% of the time I get the message:

Failed to make request to https://firestore.googleapis.com/v1/projects/<MY PROJECT>/databases/(default)
There was an error retrieving the Firestore database. Currently, the database id is set to (default), make sure it exists.

After many retries, the deploy works. So we can exclude any code issues in the Cloud Functions. This has been happening in the last weeks and lately it has been impacting my productivity a lot. Sometimes I have to retry for 20 minutes to get a working deploy.

I also traced and pinged the IP reported on the debug and it is working normally.

[REQUIRED] Steps to reproduce

See above

I also tried uninstall & install firebase-tools, have installed every single update and reauthenticated multiple times, without any change.

[REQUIRED] Expected behavior

Successful deploy or at least an useful error message.

[REQUIRED] Actual behavior

Relevant part:

[debug] [2024-08-16T18:54:10.636Z] *** [apiv2] error from fetch(https://firestore.googleapis.com/v1/projects/<MY PROJECT>/databases/(default), {"headers":{},"method":"GET"}): FetchError: request to https://firestore.googleapis.com/v1/projects/<MY PROJECT>/databases/(default) failed, reason: connect ETIMEDOUT <IP ADDRESS>:443
[info] There was an error retrieving the Firestore database. Currently, the database id is set to (default), make sure it exists.
[debug] [2024-08-16T18:54:10.637Z] FetchError: request to https://firestore.googleapis.com/v1/projects/<MY PROJECT>/databases/(default) failed, reason: connect ETIMEDOUT <IP ADDRESS>:443
    at ClientRequest.<anonymous> (C:\Users\<MY USER>\AppData\Roaming\npm\node_modules\firebase-tools\node_modules\node-fetch\lib\index.js:1501:11)
    at ClientRequest.emit (node:events:517:28)
    at ClientRequest.emit (node:domain:489:12)
    at TLSSocket.socketErrorListener (node:_http_client:501:9)
    at TLSSocket.emit (node:events:517:28)
    at TLSSocket.emit (node:domain:489:12)
    at emitErrorNT (node:internal/streams/destroy:151:8)
    at emitErrorCloseNT (node:internal/streams/destroy:116:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
[error] 
[error] Error: Failed to make request to https://firestore.googleapis.com/v1/projects/<MY PROJECT>/databases/(default)
[debug] [2024-08-16T18:54:10.638Z] *** [apiv2] error from fetch(https://firestore.googleapis.com/v1/projects/<MY PROJECT>/databases/(default), {"headers":{},"method":"GET"}): FetchError: request to https://firestore.googleapis.com/v1/projects/<MY PROJECT>/databases/(default) failed, reason: connect ETIMEDOUT <IP ADDRESS>:443
[info] There was an error retrieving the Firestore database. Currently, the database id is set to (default), make sure it exists.
[debug] [2024-08-16T18:54:10.639Z] *** [apiv2] error from fetch(https://firestore.googleapis.com/v1/projects/<MY PROJECT>/databases/(default), {"headers":{},"method":"GET"}): FetchError: request to https://firestore.googleapis.com/v1/projects/<MY PROJECT>/databases/(default) failed, reason: connect ETIMEDOUT <IP ADDRESS>:443
[info] There was an error retrieving the Firestore database. Currently, the database id is set to (default), make sure it exists.

[aalej]

Hey @kaueDM, sorry to hear you’re encountering this issue and thanks for reaching out to us. Based on the ETIMEDOUT error message, it looks like this could be a network related issue. Would you be able to try connecting to a different network to see if the behavior would change?

Also, I’m currently not able to reproduce the issue. Could you let us know how many Firestore trigger functions are in your codebase? I understand that the behavior is inconsistent, having a similar codebase to the one you have might help us with reproducing the issue.

[kaueDM]

Hey @aalej, thanks for the reply. I've already tried a different connection but the inconsistency persists. As I said in the original report, the connection seems OK since Test-NetConnection (powershell) returns a positive feedback. My current project contains 75 Cloud Functions V2 from multiple types: Firestore Triggers, HTTPS, Callables and Tasks. I started noticing the issue around ~50 CFs, but not that flaky as seem on the last weeks.

[aalej]

I discussed this with our team, currently it’s still pretty difficult to determine what specifically is causing the ETIMEDOUT error. I understand that you were able to ping the IP without any issues, but from what we can gather, generally ETIMEDOUT is usually caused by network related issues like intermittent connection. Though it may be possible that the Firebase CLI might be using a misconfigured proxy setting.

While we try to investigate what could be causing this, if you have any proxy setup, could you try temporarily disabling those to see if there would be any changes in behavior during deployment?