Closed jefimm closed 1 year ago
Good point. I will add it. Thanks
I dug to add this feature, and for now, it's not possible. The module I use for communication with redisearch is https://github.com/RediSearch/redisearch-go
(official SDK) and it doesn't allow to send an FT.EXPIRE
command to set a ttl.
I think it's not a big deal to submit a PR for adding this feature in the SDK but I don't know how long it will take before the maintainers release a new version.
I added 500k (507457) events in a local redisearch container, here's the details about memory usage:
# Memory
used_memory:1347273984
used_memory_human:1.25G
used_memory_rss:1366188032
used_memory_rss_human:1.27G
used_memory_peak:1347275552
used_memory_peak_human:1.25G
used_memory_peak_perc:100.00%
used_memory_overhead:25562504
used_memory_startup:904848
used_memory_dataset:1321711480
used_memory_dataset_perc:98.17%
allocator_allocated:1347414104
allocator_active:1349976064
allocator_resident:1373999104
total_system_memory:33355489280
total_system_memory_human:31.06G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.00
allocator_frag_bytes:2561960
allocator_rss_ratio:1.02
allocator_rss_bytes:24023040
rss_overhead_ratio:0.99
rss_overhead_bytes:-7811072
mem_fragmentation_ratio:1.01
mem_fragmentation_bytes:18955064
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:165040
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
I don't know your use-case, but if you have 500k events a year, maybe your rules are not optimum and create flood.
I have installed falco + falcosidekick-ui on our test cluster before tuning and adjusting rules and it run out of 1GB disk space in less than a week. This is the reason for this request.
How many events did you get for 1GB? I get your point, it's a really valuable idea and I would like to implement the feature, but the SDK doesn't allow it, here's the function they use to add keys https://github.com/RediSearch/redisearch-go/blob/bd32e0b428fd340ac1bc71812e9e9dcdcc852e52/redisearch/query.go#L417/query.go#L417
redisearch-go sdk uses redisgo as background, I could use it also but it means to deal with 2 different clients, which is not a good pattern. the best should be to submit a PR to redisearch to add the feature
@Issif How to use this setting when setting up sidekick-ui with helm chart? Is there a "ttl" field which can be set inside values.yaml?
The helm chart is not ready as this feature will be in next falcosidekick-ui release. But yes, an helm value will expose it.
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale
.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close
.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle stale
Currently writing data to redis does not set any TTL on the data - this will eventually fill the disk Please add configuration for the expiration of the data written to redis with default of no expiration