v2 functions stopped working upon upgrade to v12.5.0

[mazenkhalil]

[REQUIRED] Environment info

firebase-tools: v12.5.0

Platform: macOS

Using the latest firebase-tools version (12.5.0) causes runtime error in v2 functions.

Downgrading to v12.4.8 solves the issue, but had to delete the whole function beforehand.

stacktrace

[google-oss-bot]

This issue does not seem to follow the issue template. Make sure you provide all the required information.

[atsjo]

same for me, also noticed that combining v1 and v2 functions will fail v1 functions during deploy... but didn't need to remove functions to fix this, just changed the my functions a bit (to not skip during deploy) and redeployed with 12.4.8 fixed the issue.

This is caused by https://github.com/firebase/firebase-tools/pull/6275 which is to avoid building separate images for functions inside same source code, but ends up breaking execution during runtime. There is no error in cloud build or during deploy (except when combining v1 and v2 functions)

Functions deploy had errors with the following functions:
        createUser(europe-west3)
        deleteUser(europe-west3)

Platform: windows

[aalej]

Thanks for reporting this issue @mazenkhalil, and thanks @atsjo for the additional information. So far, I am able to reproduce the error when trying to deploy a combination of v1 and v2 functions. I’ll notify our team about this so that they can take a look. However, I did not encounter an issue when deploying just v2 functions and when invoking a v2 function.

@mazenkhalil, let me know if I may be wrong but from my understanding you are encountering the issue during v2 function runtime. I’d like to ask if you could provide a sample code snippet of the function you’re deploying so that we can reproduce the issue on our end?

[cabello]

We noticed unexpected errors as well. We added log.info at the beginning of each Cloud Function and some of them were crashing before being called:

Error: TypeError: res.on is not a function

We learned by deleting the function and deploying it fresh it would work again, but next deployment would cause the issue.

The other error we saw Request body is missing data.

We only have v2 functions.

We use a single build with functions being in separate files:

// file: index.ts
export { foo } from "./foo"
export { bar } from "./bar"
export { qux } from "./qux"

Our current tsconfig.json:

{
  "compilerOptions": {
    "target": "ES2017",
    "module": "CommonJS",
    "skipLibCheck": true,
    "noImplicitReturns": true,
    "noUnusedLocals": true,
    "outDir": "dist",
    "sourceMap": true,
    "strict": true
  },
  "compileOnSave": true,
  "include": ["src"]
}

I suspected the resulting build code so I tried different targets and module resolutions but that crashed when deploying. The code looked modern but Firebase could not analyze the source code:

Functions codebase could not be analyzed successfully. It may have a syntax or runtime error

Rolling back to 12.4.X fixed the issue, we no longer have functions misbehaving.

[atsjo]

It works with v2 if I only deploy 1 function, but with several they fail runtime. I have a combination of firestore triggers, http and a scheduled function. All with newest deps and a esm bundle with functions running on node 18. I see different errors like the ones above, and my scheduled function end up with this:

Request has incorrect Content-Type.
Error: Invalid request, unable to process.
    at entryFromArgs (/workspace/node_modules/.pnpm/firebase-functions@4.4.1_firebase-admin@11.10.1/node_modules/firebase-functions/lib/logger/index.js:130:19)
    at Object.error (/workspace/node_modules/.pnpm/firebase-functions@4.4.1_firebase-admin@11.10.1/node_modules/firebase-functions/lib/logger/index.js:116:11)
    at /workspace/node_modules/.pnpm/firebase-functions@4.4.1_firebase-admin@11.10.1/node_modules/firebase-functions/lib/common/providers/https.js:405:24
    at /workspace/node_modules/.pnpm/firebase-functions@4.4.1_firebase-admin@11.10.1/node_modules/firebase-functions/lib/common/providers/https.js:394:25
    at cors (/workspace/node_modules/.pnpm/cors@2.8.5/node_modules/cors/lib/index.js:188:7)
    at /workspace/node_modules/.pnpm/cors@2.8.5/node_modules/cors/lib/index.js:224:17
    at originCallback (/workspace/node_modules/.pnpm/cors@2.8.5/node_modules/cors/lib/index.js:214:15)
    at /workspace/node_modules/.pnpm/cors@2.8.5/node_modules/cors/lib/index.js:219:13
    at optionsCallback (/workspace/node_modules/.pnpm/cors@2.8.5/node_modules/cors/lib/index.js:199:9)
    at corsMiddleware (/workspace/node_modules/.pnpm/cors@2.8.5/node_modules/cors/lib/index.js:204:7)"

everything works fine when I switch back to 12.4.8

[iampava]

Lol, I thought I was going crazy. My functions suddenly weren't working. Reverted back to 12.14.8 seems ok

[milo-]

Yep. This just took down our app when our CI environment picked up 12.5

[blidd-google]

Hi all, question — are you able to successfully deploy multiple v2 functions? I've discovered an issue with mixed v1 / v2 deployments, but I'd like to confirm that multiple v2 functions are still deploying.

[iampava]

I have multiple v2 functions. They were deploying allright, but when running I encountered all sorts of weird errors... 🤷‍♂️

[atsjo]

I just tried with 12.5.1, deploying all my functions... It build and deploys both v1 and v2 now, but (at least) the v2 functions still don't work... reverting back to 12.4.8 and redeploying fixes it again...

[patkoeller]

Also running into v2 functions not working due to this error. Deploying 2 onRequest functions, 1 onDocumentWrite, 1 onMessagePublished.

Error when making an API request to one of the onRequest endpoints. It mentions pubsub as the problem area even though that request doesn't involve pubsub.

ERROR 2023-08-23T23:32:52.375298Z TypeError: Cannot read properties of undefined (reading 'data') at func (/workspace/node_modules/firebase-functions/lib/v2/providers/pubsub.js:126:42)
DEFAULT 2023-08-23T23:32:52.375313Z at /workspace/node_modules/@google-cloud/functions-framework/build/src/function_wrappers.js:141:25
DEFAULT 2023-08-23T23:32:52.375319Z at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

Reverting to before 12.5.0 fixes the problem.

[blidd-google]

To confirm — after upgrading to v12.5.1 (which includes the patch to fix deployments that include both v1 and v2 functions), you are still running into runtime errors? Apologies for the inconvenience caused, and thanks for your patience as we try to get to the bottom of this issue.

[JJ11teen]

I have multiple v2 functions (no v1 functions), and they stopped working with v12.5.0. The logs when calling some of the functions looked like this: And for other the wrong function would get called entirely (I'd get exceptions from paths that didn't exist in the function being called.)

I did not explicitly try v12.5.1, but given I have no v1 functions I don't think that fix is enough/applicable. Reverting to v.12.4.8 & deleting & redeploying the functions fixed the issue for me.

[milo-]

@blidd-google can you delete the 12.5.0 and 12.5.1 from npm? There are some critical issues here that are affecting production environments.

[atsjo]

12.5.1 also has the problem, yes... maybe this is only a problem in some regions, since you struggle to reproduce...? I use europe-west3...

[Wackymax]

I've had similar problems. Been breaking my brain trying to figure it what I was doing wrong with the migration of firebase functions from V1 to V2 😅. From what I could tell my https function was executing in place of other functions like the firebase triggers. What is strange is that some functions would work and others wouldn't. It almost looked like functions alphabetically before my http function would work and functions after that would be overwritten by the http function and it would execute instead.

[alexistm96]

I have a similar problem, all of the sudden none of my cloud functions work, im on 12.5.1, when trying to run "X" function, i get back an error thrown by "Y" function, which is weird because i am not doing anything to run "Y" function.

[trevor-rex]

This is the error I received for 4/~60 of my functions (v2 functions) when trying a full deploy this morning (on 12.5.1).

⚠  functions: HTTP Error: 400, Invalid Request. Violations: only Firebase Functions are supported for source tokens.

The functions that this broke for were using a serverless VCP connector, I am not sure if that is the cause? However I did see a few functions using the same VCP connector succeed in deploying. They were a mix of onCall and onRequest triggers.

[al4nnw]

And for other the wrong function would get called entirely (I'd get exceptions from paths that didn't exist in the function being called.)

@JJ11teen Dude, I was debugging this for the past 4 hours. I started to hallucinate out of madness.

For me, my app was calling a function named addUser, but the passwordReset function was the one that got called and actually sent the email.

I found out that all functions were pointing to this passwordReset, which was the first one exported in my index.ts file. After commenting it out, all functions started to point to the second function being exported.

[joehan]

Hey all, apologies for the disruption. We're in the process of rolling back this change in v12.5.2, which should be out shortly. We're also working with the Google Cloud Functions team to debug this issue and ensure a smooth rollout at a later date.

If you run into this issue, the quickest way to resolve it is to upgrade to 12.5.2, then delete and redeploy any affected functions.

[joehan]

Going to close this as fixed for now with the release of 12.5.2, but @blidd-google and I will be keeping an eye on it. If you are still running into similar issues after upgrading, please let us know and we'll investigate ASAP.

[mazenkhalil]

Thank you for the efforts. v12.5.2 is working as expected.

[arndtvoges]

🤯 This just caused us 2.5 million (!!!) function executions in less than 16 hours because the wrong function was called, trying to update itself.

What's your plan to avoid bugs of this magnitude in the future?

[joehan]

@arndtvoges Very sorry to hear how this issue impacted you. Please file a support ticket for this (https://firebase.google.com/support/troubleshooter/contact) - they should be able to get you help with any unexpected bills due to this.

[thuytrinh]

Has this issue reoccurred in version v12.7.0?

I recently upgraded from version v12.4.8 to v12.7.0, and after deploying, I encountered failures in calls to all onCall functions, resulting in the following error:

TypeError: res.on is not a function
2023-10-14 14:39:41.215
    at /workspace/node_modules/firebase-functions/lib/common/providers/https.js:392:17
2023-10-14 14:39:41.215
    at new Promise (<anonymous>)
2023-10-14 14:39:41.215
    at /workspace/node_modules/firebase-functions/lib/common/providers/https.js:391:16
2023-10-14 14:39:41.215
    at /layers/google.nodejs.functions-framework/functions-framework/node_modules/@google-cloud/functions-framework/build/src/function_wrappers.js:141:25
2023-10-14 14:39:41.215
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

Similar failures also occured in all onRequest functions:

Bad signature TypeError: Cannot read properties of undefined (reading 'stripe-signature')
    at /workspace/lib/stripe-invoicing/stripe-functions.js:30:90
    at /workspace/node_modules/firebase-functions/lib/v2/trace.js:16:28
    at /layers/google.nodejs.functions-framework/functions-framework/node_modules/@google-cloud/functions-framework/build/src/function_wrappers.js:141:25
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

However, after I reverted to v12.4.8, such failures were gone.

[milo-]

@thuytrinh Yep, currently no fix available https://github.com/firebase/firebase-tools/issues/6414

[chrisgjarrett]

@thuytrinh Just adding to this, we are also seeing a similar stack trace when deploying with Firebase tools 12.7. We rolled back to 12.4.8 to address this

[cabello]

A selfish piece of advice, make sure your CI/CD pins the dependency to a working version and wait a couple weeks before moving on to the latest release, we did that due to the last incident, before that our CI was always installing the latest release.

[SherpaMiguel]

Problem solved after upgrading to 1.8.0. :)