All of this started because my server got killed by the OS for using too much memory. I did a quick test and after 3 minutes of working on the website, it was already using 1GB of memory:
This is insane and there's a couple of reasons for it. With these changes, I'm now running under 100MB:
The Collections process is saving every file and its contents. This is wasteful because the Collections only cares about files that have a "tag". It also doesn't care about the "content" of these files. On top of that, the "profiler" file was also being indexed and the metadata on that one is huuuuuuuge.
We have a cached_source_file in every Node. This is useful to make helpers like import_js_file run fast, but we don't need to save the "content" in there because we never use it. Removing it saves a ton of memory as well.
The profiler is the big offender because its memory usage is increasing with every refresh. But it's even more interesting: disabling the profiler also makes the whole server run faster! This makes sense because every render/compile reports to the profiler.
While I was working on this, I realized that we are writing to disk too many times. On dev, if we are compiling files on a per-request basis, there's no point in saving to disk just to read it from the disk immediately after. We can just serve it from memory! That's why introduce a new run type compile_dev.
All of this started because my server got killed by the OS for using too much memory. I did a quick test and after 3 minutes of working on the website, it was already using 1GB of memory:
This is insane and there's a couple of reasons for it. With these changes, I'm now running under 100MB:
cached_source_file
in every Node. This is useful to make helpers likeimport_js_file
run fast, but we don't need to save the "content" in there because we never use it. Removing it saves a ton of memory as well.While I was working on this, I realized that we are writing to disk too many times. On dev, if we are compiling files on a per-request basis, there's no point in saving to disk just to read it from the disk immediately after. We can just serve it from memory! That's why introduce a new run type
compile_dev
.