investigation: Improve memory usage of Search store / backends

lexical-lsp / lexical

Lexical is a next-generation elixir language server

782 stars 77 forks source link

investigation: Improve memory usage of Search store / backends #584

Closed scohen closed 4 months ago

scohen commented 5 months ago

Things to try:

[x] #592
[x] #599
[x] #591
[x] Ensure loading data from the wal to the store doesn't read more than a couple dozen megs at a time
[x] #597
[x] #594
[x] Disable / improve the fuzzy matcher. (This will save 200mb in the lexical project) #596
[x] Compress the ets table

scohen commented 4 months ago

With the all of the above changes applied, the memory usage of lexical running in the lexical project dropped from 1200Mb to 435Mb.

scohen commented 4 months ago

Using streams reduces memory usage from 435Mb to 245Mb

scohen commented 4 months ago

Closing for now. When i started this investigation, Lexical consumed 1.2 gigabytes when running its own code. Now it's sitting comfortably at 330 megabytes.