daniloarcidiacono / hazelcast-monitor

Hazelcast Monitoring Tool
MIT License
14 stars 5 forks source link

Viewing hazelcast map causes memory overflow #1

Open sandiegoe opened 2 years ago

sandiegoe commented 2 years ago

Hello, we are using your open source hazelcast-monitor and encountered an OutOfMemory error. We have a very large IMap cache with a data volume of about 10 million. In hazelcast we have configured a maximum capacity of 2 million, but when querying the map data through hazelcast-monitor, it will cause memory overflow. . After reading the code, I found that all the data was loaded during the query. I want to confirm whether this is the reason.

image
daniloarcidiacono commented 2 years ago

Hello, yes the monitor will indeed pull each entry of the Map and apply the filtering predicate. In your case I suspect that there is no filter so all 2 million entries are added in the resulting list, causing the OutOfMemoryError.

At the time of writing, I was using Hazelcast 3.9.3 which didn't have any facility to do this type of things, so I chose to implement this manual filter. However, even with enough memory and/or a restrictive filter, I would expect very poor performance on such a large Map, let alone the fact that the query is executed periodically.

A possible improvement would be to upgrade Hazelcast to at least version 3.12 which introduces paginated distributed queries with SQL syntax (see Hazelcast Docs: Distributed Query, and Hazelcast Docs: Querying in Collections and Arrays), which I expect to perform much better.

Last but not least, please keep in mind that hazelcast-monitor is intended as an aid during development rather than a battle-tested tool for production monitoring.

sandiegoe commented 2 years ago

Thank you very much for your answers!

The monitor tool uses true by default to filter data. It is best to add a limit to the number of existing implementations to avoid possible problems

image image