svlad-90 / DLT-Message-Analyzer

"Extended search" plugin for the DLT-viewer
GNU General Public License v3.0
77 stars 13 forks source link

[SEARCH_VIEW] Investigate high memory consumption during requests with many regular expression groups #192

Closed svlad-90 closed 5 months ago

svlad-90 commented 11 months ago

How can one reproduce this issue?

  1. Work with the data set that contains ~ 2M messages ( ~ 512 Mb of data )
  2. Work with the regular expression that makes it possible to create a sequence diagram out of those 2M messages. The "UML" checkbox in the "Search view" should become available for all the found messages, after the search is started.
  3. Start the search
  4. Check the memory consumption. It might increase up to > 10 GB

This is a new issue caused by the creation of sequence diagrams for a large number of messages. The plugin was not used in this way before, so it was expected that something like this might happen.

=> It is not a memory leak, but rather a properly allocated, still huge area of memory, that is used for storing the sequence diagram metadata. At least now we have such an assumption, that should be checked.

Use Valgrind massif to locate the high memory consumption location. Then try to eliminate it.

svlad-90 commented 5 months ago

It was identified that the root cause of this issue is located in the highlighting metadata. Any search query that:

will cause a high RAM consumption.

We need to refactor the highlighting functionality to consume less RAM.