search

jameslittle230 / stork

🔎 Impossibly fast web search, made for static sites.
https://stork-search.net
Apache License 2.0
2.73k stars 56 forks source link

Limit the indexer's RAM usage #137

Open geekjr opened 3 years ago

geekjr commented 3 years ago

Hi. I have a vm with 0.5GB ram that i will be using to built the stork index. The toml has 7k records. Is there a way to limit how much ram stork can use?

jameslittle230 commented 3 years ago

@geekjr - Not currently, though this would be good to look into!

This might be made easier when sharding the index (#128) is possible. The indexer could index N records and write those records to disk until all the files are indexed, then merge the indexes together at the end.

geekjr commented 3 years ago

How do i merge them? Do i just rerun build?

jameslittle230 commented 3 years ago

Sorry - it's not possible to do this today, I was just thinking through how I might want to implement this.

geekjr commented 3 years ago

Oh ok. Do you think its possible to limit the memory usage using ulimit?

On Tue, Jan 19, 2021 at 6:59 PM James Little notifications@github.com wrote:

Sorry - it's not possible to do this today, I was just thinking through how I might want to implement this.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jameslittle230/stork/issues/137#issuecomment-763223900, or unsubscribe https://github.com/notifications/unsubscribe-auth/AN3OQ5TAIEMQWZ5C6PMWNALS2YMGHANCNFSM4WJX7TGQ .

jameslittle230 commented 3 years ago

There's no special handling for that, so you'd likely see the program panic.

geekjr commented 3 years ago

Yes. That is what happened when I tried.

On Tue, Jan 19, 2021 at 7:11 PM James Little notifications@github.com wrote:

There's no special handling for that, so you'd likely see the program panic.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jameslittle230/stork/issues/137#issuecomment-763232274, or unsubscribe https://github.com/notifications/unsubscribe-auth/AN3OQ5XAYFCQW32FFSVSK43S2YNSRANCNFSM4WJX7TGQ .