Eviction issue resulting in OutOfMemoryException

[Pinkolik]

Hello. We use hazelcast 3.12.3 in our application and recently we have encountered OutOfMemoryExcpetion related to hazelcast. After further studying I have found out that map eviction doesn't work as its expected to work.

First, let me tell the use case of hazelcast map in our application. We have a map <String, byte[]> which stores raw bytes of documents by their ids, and we can't clear it by timeout (time to live) for the sake of fault tolerance, so it was decided to evict entries only in case when application is running out of memory. We implemented our own MapEvictionPolicy which compares entries by creation time.

We though that everything worked fine since we saw log messages that eviction policy has been executed but later our application started falling with OutOfMemoryException. After some research I found out that when eviction policy is executed only one entry is evicted from map, the problem is that documents have different sizes. And I think that hazelcast assumes that they have same size so for hazelcast removing one entry means freeing space for another entry, which isn't true in our case.

So if, for example, we fill our map with many small documents until MaxSizeConfig capacity is reached, and then start adding larger documents, used heap size will keep increasing until we hit OutOfMemoryException.

Additional information: Hazelcast version: 3.12.3 Cluster size: same problem as for single member as for many Number of the clients: same as above Version of Java: 1.8 Operating system: tested on Linux and Windows 10

I have a simple test application for recreation of that test case https://github.com/Pinkolik/HazelcastOOME . Thank you for your attention

[ahmetmircik]

Hi @Pinkolik, you can retry by setting below property to some number that fits your case. This forces map to evict more than 1 entry at a time. Can you please test with this workaround?

Config config = new Config();
config.setProperty("hazelcast.map.eviction.batch.size", "2");

[Pinkolik]

Hello @ahmetmircik, thank you for your suggestion, just tested it, still same result. The problem is that we can't assume that N number of entries will equal one new entry because size of a document is always different. I assumed that hazelcast should evict entries from the map until there is enough space for the new entry after putting which there size from MaxSizeConfig won't be exceeded

[Pinkolik]

I'm sorry if I'm bothering you but could you tell me if this problem has a solution that would fit in our application? It's been 8 days since my last comment and nobody has replied yet...

[ahmetmircik]

Hi @Pinkolik,

As far as i see, your test's entries don't have an upper limit and entry sizes always keep growing, eventually this will result with OOME under any condition. IMap eviction happens per partition basis, so it is good to have identical entries.

I uploaded a patched version of 3.12 also, but only for experimental purposes, which checks if a map still needs to remove entries after eviction and does further removals accordingly, just in case if you want to play with it: https://github.com/ahmetmircik/hazelcast/tree/oomeEviction, if it fits your case, we can think on this as an alternative solution.

[Pinkolik]

Thank you! I'll try this out as soon as possible and give you the feedback

[Pinkolik]

Hello @ahmetmircik, I have tested your patched version of hazelcast and it seems to work, but now I've encountered another problem: when size of the map is close to its limit and I try to add a new entry, new entry isn't added. Do you have any suggestion why this could happen? It doesn't happen always but with about 50% chance. I've updated test project so you could see it for yourself.

Also we are wondering if you are going to include this feature in the next releases of hazelcast? It would be nice if there would be some configuration property which could enable eviction algorithm we suggested.

Thank you for your reply

[ahmetmircik]

Hi,

I have tested your patched version of hazelcast and it seems to work, but now I've encountered another problem: when size of the map is close to its limit and I try to add a new entry, new entry isn't added. Do you have any suggestion why this could happen?

When memory is at its limits and the partition which the entry is in has no other entry than you just put, IMap evicts that only 1 entry. This is the reason.

Also we are wondering if you are going to include this feature in the next releases of hazelcast? It would be nice if there would be some configuration property which could enable eviction algorithm we suggested.

Let us think about this. I'll update this issue if we have any improvement.

[taherajna]

Hi @Pinkolik , Just a thought, have you tried using the eviction policy based on max heap space occupied: https://docs.hazelcast.org/docs/3.3-RC3/manual/html/map-eviction.html I believe that will solve your issue.

The exact setting being:

USED_HEAP_SIZE: Maximum used heap size in megabytes for each JVM.
<max-size policy="USED_HEAP_SIZE">4096</max-size>

[Pinkolik]

Hello @taherajna , Yes we have tried that policy too. It had the same issue. For now we came up with a temporary workaround to set eviction time to 10 minutes