Massive memory leak

[ffehse]

Describe the bug When loading a project with access to two folder of roughly 40 GB of size total (not only text data), the memory consumption increases with seemingly no end.

To Reproduce Load a large project? Small projects seem to work just fine.

Expected behavior Only allocate the necessary RAM that is needed.

Screenshots

The project is loaded at the beginning of the memory graph and the editor starts using more and more ram

It keeps filling the memory, see the "Committed" amount

• OS: [e.g. Windows]
  • Win 11
• Focus Version: 0.1.10, 0.1.9

Additional context The two folders loaded in the project are of sizes 35.2 GB and 9.5 GB, so even if all the data is loaded, it should not be much larger than 50 GB I believe.

[focus-editor]

Thanks for the report. Memory usage is currently not very optimized, so the overhead can be quite high. It will be improved eventually, but it's not at the top of the list because this editor is not designed with super large projects in mind.

That said, the gigabytes of memory in those folders are most likely binary files, which the editor should ignore. In order to decide whether a file is binary or not, it's currently loading it into memory and checks for zeroes. The memory is then freed, but the operating system might not see it right away. It's not optimal, of course, and should be improved later as well. To avoid doing this it should be possible to list the extensions of the known binary files in the config file so that they can be ignored right away. Can you share your [workspace] section of your project config please?

[ffehse]

The [workspace] section is just two project folders

C:/Felix/Coding/Nanoacademic/qtcad
C:/Felix/Coding/Nanoacademic/poissonfem

I just noticed, that I acidentally allowed a binary file type instead of "ignore"ing it. By fixing this, the memory consumption is not growing indefinitely anymore, but stays at ~43 GB. I will make sure I "ignore" more stuff. But it's still surprising, that the memory doesn't get freed after reading through the folders...

[ffehse]

I found the culprit: There was a data type .msh2 that stores mesh data in a text form. By "ignoring" that filetype, the memory consumption is only about 3 GB!

[focus-editor]

That explains everything. By loading those .msh2 files it would pay the full memory overhead since the files were text files. Perhaps we should add it to the default ignore list.

[Mulej]

We should perhaps have a whitelist of file types that we preload into memory, the rest should be lazy loaded when (if) needed.

[focus-editor]

Loading files into memory is required for fast search. If we don't load them we can't search them as quickly.

I do agree that we should have a way for people to only load the files with extensions they specify, and ignore everything else. If I remember correctly, if we ignore everything in the config right now then it will have precedence over the allow list. We should reconsider that

[rzuf79]

Mac version 0.2.0 grows steadily just by being idly opened, 1MB every about 20 seconds. Noticed this on a very small project (less than 1MB of files) after having the editor opened for a day or two.

[focus-editor]

@cookednick do you know what might be going on on Mac?

Maybe the problem is fixed in the dev version, but I can't build a mac version right now and @ileonte doesn't seem to have the nightlies for mac available either