search

iggy-rs / iggy

Iggy is the persistent message streaming platform written in Rust, supporting QUIC, TCP and HTTP transport protocols, capable of processing millions of messages per second.
https://iggy.rs
MIT License
1.82k stars 86 forks source link

How can I limit RAM usage in settings? #714

Open pikhovkin opened 6 months ago

pikhovkin commented 6 months ago

How can I limit RAM usage in settings? For example, limit the use of RAM to no more than 1GB, but keep logs on the local storage system (SSD) for six months.

hubcio commented 6 months ago

in server.toml you have cache section: obraz

you can set the size to for example 1 GB, server will use a little more than that, (maybe 2 GB tops)

pikhovkin commented 6 months ago

server will use a little more than that, (maybe 2 GB tops)

@hubcio this is a really helpful comment!

pikhovkin commented 6 months ago

you can set the size to for example 1 GB

I set 512 MB for cache, but it doesn't work for Iggy 0.1.8, the RAM continues to increase. I stopped checking at 3GB RAM.

hubcio commented 6 months ago

@pikhovkin could you please share the logs? local_data/logs

spetz commented 6 months ago

@pikhovkin does it happen immediately when starting the server or maybe after some time? There's also the setting called messages_required_to_save which is essentially responsible for keeping the unsaved messages in memory buffer, after they reach the provided threshold to be eventually saved onto the disk (or they can be saved in the background task, as specified in message_saver configuration). It's possible to have multiple partitions holding lots of messages in the buffer, which might increase the RAM usage. Also, another thing to consider are the indices (you can find the settings called cache_indexes and cache_time_indexes) - by default, a single index value is just 4 bytes, but once you have billions of messages, these could result in higher memory usage as well (unless you simply disable their cache).

hubcio commented 6 months ago

@pikhovkin can you also do this?

image

hubcio commented 6 months ago

@pikhovkin for me, when I disable caching indexes and time indexes and use cache 512MB i get constant 1.3 GB memory usage.

hubcio commented 6 months ago

@pikhovkin FYI I created #724 to address this issue.

hubcio commented 6 months ago

@pikhovkin upon commiting of #658 problem will be partially resolved.

hubcio commented 6 months ago

@pikhovkin could you please check now (server 0.2.5), crates.io (0.2.7)

pikhovkin commented 6 months ago

@pikhovkin could you please check now (server 0.2.5), crates.io (0.2.7)

Thanks @hubcio! I can check it after resolving this issue

hubcio commented 6 months ago

@pikhovkin could you please check now (server 0.2.5), crates.io (0.2.7)

Thanks @hubcio! I can check it after resolving this issue

Alright, I will check it too.

hubcio commented 3 months ago

@pikhovkin it is now solved, can you try?

pikhovkin commented 2 months ago

@hubcio I can’t test it properly yet because of this error .

pikhovkin commented 1 month ago

I still can't start properly https://github.com/iggy-rs/iggy-python-client/issues/20