search

thomasnordquist / MQTT-Explorer

An all-round MQTT client that provides a structured topic overview
https://mqtt-explorer.com
Other
3.11k stars 295 forks source link

Memory management #103

Open thomasnordquist opened 5 years ago

thomasnordquist commented 5 years ago

There is the issue of quantity of topics, message history size, message parsing and preview

30.000 topics x 50kb messages => 1.5GB Ram used

Known "bugs"

React Fiber

React fibers memory management is still a little bit mysterious when all topics of a 30.000 topic tree are expanded, there are about 400.000 dom nodes, reacht fiber seems to hold a reference to old nodes when they no longer exist due to React Fiber's "memoization". It is unclear when these references are cleared. When they have finally cleared it appears electron can reclaim almost all of the allocated memory. In this extreme scenario, it is possible to run out of memory. We maybe should prohibit actions like "expand all", when there are too many topics in a tree. It is unlikely one could even fathom this amount of data.

Long term stability

Currently, all topics are stored, the amount of stored messages per topic is limited. However incoming topics names may be generated and may flood the tree over time. This may cause long-term memory issues. It might be wise to forget very old data when a specific threshold is reached.

Detect and handle out-of-memory

Currently, the app would most likely crash or freeze. We could clear the message history ring buffers and drop old topics.

thomasnordquist commented 5 years ago

Memo: research WeakMap and React Fiber, the history Ringbuffer may provoke that react keeps all memoized nodes since the references still exist in the ringbuffer. Idea: Immutables? Clone instances before presenting them. The tree updates the most, "leaks" the most. (Leak in parenthesis because it is not leaking permanently)

thomasnordquist commented 5 years ago

I added the base framework for automated leak tests, based on existing UI-Tests.