Purge cached ZIMs / open entries from kiwix-serve after a certain delay

kiwix / libkiwix

Common code base for all Kiwix ports

https://download.kiwix.org/release/libkiwix/

GNU General Public License v3.0

112 stars 55 forks source link

Purge cached ZIMs / open entries from kiwix-serve after a certain delay #1025

Open benoit74 opened 7 months ago

benoit74 commented 7 months ago

Currently kiwix-serve is caching ZIMs / open entries and its cache size is only limited by number of ZIMs / open entries (if I understood @mgautierfr well).

We could probably benefit from purging the cache from ZIMs / open entries which have not been accessed since a given amount of time, this would probably free a significant amount of memory.

This expectation comes that on library.kiwix.org we have a varnish cache in front of kiwix-serve with a retention of 24h. Varnish cache consumes between 2.5 and 5G of RAM, while kiwix-serve consumes a lot more, and it keeps growing (even if growth is slower and slower as time pass by). For instance today after 3.5 days of uptime, kiwix-serve is already consuming about 10G of RAM.

Would it make any sense / be feasible with a reasonable effort?

kelson42 commented 7 months ago

I would like to come back to the problem description before we talk about any solution.

@mgautierfr Can you confirm please:

Does kiwix-serve runs on the longer term in a stable manner from a memory consumption perspective?
Do we have any working system to prevent memory exhaustion?

mgautierfr commented 7 months ago

Does kiwix-serve runs on the longer term in a stable manner from a memory consumption perspective? Do we have any working system to prevent memory exhaustion?

We have a system to limit what it is stored in term of the number of entries (pretty well resumed in https://github.com/kiwix/k8s/issues/147)

While it is not technically speaking a system to prevent memory exhaustion (we don't reason by memory, and a entry could be really big), on the long term the memory consumption hits a limit (which one ? Can't say) and we should be stable.

kelson42 commented 7 months ago

I have read our two tickets more than twice and what I can say is:

I still don't understand really much how it works today
We have many level of caching: libzim, libkiwix, kiwix-serve?
It seems to not be properly documented!?
The garbage-collection of memory is not explained (all these things should benefit of standards caching methodology... but not sure).
Obviously it's very hard that way to control memory usage
impossible to say a top memory usage
I don't like to have these internal driven (only) with env. variables.

Make a lot of problems and no clue where to start really... documenting things in detail is probably the thing.

kelson42 commented 2 months ago

@veloman-yunkan Would you be able to make a dedicated documentation in the online doc how cache works in libkiwix?