Better documentation on how evicition / TTLs / limits works - HTreeMap

[lkolek]

Based on doc it is hard to understand how eviction / overflow realy works.

• as I understand, at least one of expireAfterGet / expireAfterCreate / expireAfterUpdate is needed for eviction to work,
• positive parameter values in expireAfterGet / expireAfterCreate / expireAfterUpdate purges records after specified time - parameter value (TTL) in ms. This time counts from the last operation on that record.
  • if for example expireAfterGet(100) is specified, and record is first get then updated, will it be purged?
• next step is checking limits (expireStoreSize, expireMaxSize). If the store goes above specified limits, more records will be purged. In that case, expireAfterGet / expireAfterCreate / expireAfterUpdate with no parameter (or -1) is also included.
  • what is an effective order of purging? I can see:
  • Segment (what that meens?),
  • get/update/create queues (in that order)
  • queue records starting from the oldest.
• for above scenarios, last operation (get/create/update) counts,
• after that, compactation could take place.
• what is the unit of expireStoreSize? bytes?
• object removal from htreemap also removes this object from overflow,
• if object is not in htreemap, it will search for it in overflow,

Also, If I understand correctly, overflow works only for evicted (purged) records. Is there a good / prefered way to save all records to overflow htreemap before close?

[jankotek]

Yes, most of this is correct. ExpireStoreSize should be in bytes.

s there a good / prefered way to save all records to overflow htreemap before close?

Clear on memory map moves all records into overflow map.

I need to document this and get Mapdb-JCache plugin running. Will keep this open until its fixed.

[nddipiazza]

Yeah this would be great to have this documentation. I'd be happy to add it.

I have a 12G JVM that I was able to create a 60G mapdb hashmap. no problem.

So then later on in my program I need to sequentially process each entry in this hashmap with multiple threads.

But when reading the store, it causes OOMs when doing the reading.

So I'm in the need of setting up the expires. But it's hard to figure out what each parameter does due to lack of documentation. I have been reading the code and using this PR to figure out what to do. Perhaps I can help out and contribute some documentation.

The most confusing part is probably what he mentions above about how expireAfter[Get/Update/Create] needs to be used in order to activate evictions. And then how they all work together along with the expire store size, expire max size, ...