NextJS - Extreme latency and out of memory cloud functions on Firebase after upgrading next to >13.4.12

[heliohm]

[REQUIRED] Environment info

firebase-tools: 12.5.2

Platform: MacOS 13.2.1 , MBP Pro M2 Pro,

[REQUIRED] Test case

https://github.com/heliohm/next-firebase-debug-latency

[REQUIRED] Steps to reproduce

• Enable web frameworks experiment for firebase, create new firebase project
• Deploy repository to the Firebase project
• Deployed example available temporarily at \<github repository name>.web.app at endpoints /api/ping, /api/ping-auth (loading FB Auth) and /api/hello (default NextJS route from create-next-app)

[REQUIRED] Expected behavior

NextJS SSR cloud function latency is typically 1-200ms after cold start

[REQUIRED] Actual behavior

• API responses from NextJS cloud run function will always take several seconds, also with minInstances > 0.
• The minimum example latency is 1-3 seconds, the real application takes 5-10 seconds for every API call.
• Frequent out of memory messages (not always) in SSR cloud function logs in google cloud logs unless memory is set to 1GiB for the real project (for which the SSR part is very small)
• Downgrading to next.js 13.4.12 makes the problem go away.
  • Behaviour in firebase emulator (real project emulates hosting+auth+firestore) is good

[heliohm]

I edited the firebase-tools frameworks/index.js build script and added the Google Cloud Profiler and Trace Agent before deploying again.

The tracing is set up according to this guide: https://nextjs.org/docs/app/building-your-application/optimizing/open-telemetry.

Findings are:

1. Profiling shows that for most of the delay, the cloud function is just sitting idle.
2. Heap usage is <10MB worst case
3. Tracing shows the API endpoints at <1ms and the whole execution round at seconds as expected
4. Logs show that cloud function runs out of memory almost every call, but not all.
5. After increasing memory to 1GiB in firebase.json and re-deploying - latency is just as bad, but out of memory log messages are gone and profiler now only shows Idle for 600ms and nothing more.

[heliohm]

Addition: GCP tripped the billing budget set up throughout development without any usage beyond a developer. Metrics Explorer show Cloud Run Revision -> Container -> Billable Instance Time going at 16 mins/min (whatever that means) and ramping up almost 10$/day, with zero active instances. Seems something is hanging beyond the user's control here?

[mrjackyliang]

Not really a resolution to this issue, but I've also spent hours figuring this out. The cloud function memory is being exhausted, even if I increase it.

It's also annoying to realize that the documentation stated that the latest supported Next.js version is 13.4.7, and it wasn't even shown during deployment.

https://firebase.google.com/docs/hosting/frameworks/nextjs

[emarcotte]

Might be related to https://github.com/FirebaseExtended/firebase-framework-tools/pull/122.