Project deployed to fly.io runs out of memory

[Josehower]

We are currently having some issues with memory usage in the example repos deployed to fyl.io. The interesting part is the maintainer's solution is a scale to more memory (which at the same time would mean upgrade to a paid plan).

Possible Root Causes:

• Memory leak in Next.js

• Fly.io specific problem

  • https://community.fly.io/t/out-of-memory-restarts/1629

• Some of our code is not setting things up correctly

Alternatives when a solution has not been found/ is out of range (for example. A bug in Next.js):

• Code Updates that reduce the amount of memory used
• Use Postgres Cluster instead of a custom solution
• Use a different solution to deploy Bootcamp apps

Actions taken and further research:

Project updates to improve the memory usage:

• Using React cache in the database queries
• Moving the project to fly.io V2

Memory tests and data gathered:

After testing with scaling the machine to 512MB, the memory usage goes only up to 285MB which exceeds the limit of 256 MB just for a small amount.

Processes in fly.io VM sorted by memory usage:

Mem: 220200K used, 6252K free, 40428K shrd, 1212K buff, 62304K cached
CPU:   0% usr   0% sys   0% nic  99% idle   0% io   0% irq   0% sirq
Load average: 0.00 0.00 0.00 2/101 640
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
  613   597 root     S    20.3g9381%   0   0% /usr/local/bin/node /app/node_modules/.pnpm/next@13.4.0_@babel+core@7.21.0_react-dom@18.2.0_react@18.2.0_sass@1.62.1/node_modules/next/dist/compiled/jest-worker/processChild.js  
  597   586 root     S    10.2g4757%   0   0% node /app/node_modules/.bin/../next/dist/bin/next start
  514     1 root     S     696m 314%   0   0% /.fly/hallpass
  586   513 root     S     330m 149%   0   0% node /usr/local/bin/pnpm start
  625   613 root     S     262m 118%   0   0% /usr/local/bin/node /app/node_modules/.pnpm/next@13.4.0_@babel+core@7.21.0_react-dom@18.2.0_react@18.2.0_sass@1.62.1/node_modules/next/dist/compiled/jest-worker/processChild.js  
  624   613 root     S     262m 118%   0   0% /usr/local/bin/node /app/node_modules/.pnpm/next@13.4.0_@babel+core@7.21.0_react-dom@18.2.0_react@18.2.0_sass@1.62.1/node_modules/next/dist/compiled/jest-worker/processChild.js  
  535   530 postgres S     167m  76%   0   0% postgres: autovacuum launcher
  536   530 postgres S     167m  76%   0   0% postgres: logical replication launcher
  532   530 postgres S     166m  75%   0   0% postgres: background writer
  531   530 postgres S     166m  75%   0   0% postgres: checkpointer
  534   530 postgres S     166m  75%   0   0% postgres: walwriter

This seems to show Postgres database plays a relevant role in memory usage

Local Memory Tests

In a local environment Next.js is using around 210MB of RAM only to render a page, which i think it may be the reason is falling in fly.io

The following screenshot was printed from process.memoryUsage() inside of each of the project pages:

The app was also tested under different ram conditions using the Node flag --max-old-space-size

NODE_OPTIONS=--inspect NODE_OPTIONS=--max-old-space-size=256 pnpm dev

• 256MB the app is stable
• 220MB the app is consistently failing

Conclusions summary

1. Multiple resources point to Next.js is using more memory since the latests updates. Some of them even suggest a Memory Leak.
2. Fly.io and local environment RAM usage are very similar using around ~230MB of memory on idle. This in addition to the Postgres setup in Fly.io environment may cause the 256MB of memory being insufficient and causing the app to crash.
3. Updates to the project such as using Fly V2 architecture and React cache don't reflect any significant changes in memory usage.

Next steps based on research:

Since we can't completely discard the idea of being Postgres in the same machine as the app is causing the memory to be insufficient more tests should be performed. This may lead into a potential solution being the use of Postgres from an external machine in the environment.

1. Create and deploy without database setup
2. Test if this reduce the amount of RAM needed for the project

[karlhorky]

Some updates (done in the Security Vulnerability Examples repo - deployed version):

• upgrading to the latest canary for all of the memory improvements that the Next.js team has made (commit)
• setting the start command to reduce RSS memory footprint (commit)
• setting swap_size_mb = 512 in fly.toml (commit)

Let's continue the discussion about how we want to move forward here: https://github.com/upleveled/courses/issues/1732