still-ex / still

A composable Elixir static site generator
https://stillstatic.io
ISC License
252 stars 10 forks source link

Improve runtime memory usage #162

Closed gabrielpoca closed 3 years ago

gabrielpoca commented 3 years ago

All of this started because my server got killed by the OS for using too much memory. I did a quick test and after 3 minutes of working on the website, it was already using 1GB of memory:

Screenshot 2021-08-17 at 08 35 21

This is insane and there's a couple of reasons for it. With these changes, I'm now running under 100MB:

  1. The Collections process is saving every file and its contents. This is wasteful because the Collections only cares about files that have a "tag". It also doesn't care about the "content" of these files. On top of that, the "profiler" file was also being indexed and the metadata on that one is huuuuuuuge.
  2. We have a cached_source_file in every Node. This is useful to make helpers like import_js_file run fast, but we don't need to save the "content" in there because we never use it. Removing it saves a ton of memory as well.
  3. The profiler is the big offender because its memory usage is increasing with every refresh. But it's even more interesting: disabling the profiler also makes the whole server run faster! This makes sense because every render/compile reports to the profiler.

While I was working on this, I realized that we are writing to disk too many times. On dev, if we are compiling files on a per-request basis, there's no point in saving to disk just to read it from the disk immediately after. We can just serve it from memory! That's why introduce a new run type compile_dev.