ScanContext contains hash maps that tracks the matches found for each pattern. The values in these maps are vectors that can become very large. For performance reasons, instead of completely deallocating those vectors after each scan, we were clearing the vector but retaining its capacity, so that they can be reused in later scans. The problem with that is that these vectors are not freed while the scanner is in use, and this can have a large impact on the process' memory footprint, causing OOM issues.
This PR adds a PatternMatches type that encapsulates all the logic for tracking pattern matches, including a more sophisticated approach that tries to reuse the vectors as much as possible, but frees them when they grow too much.
ScanContextcontains hash maps that tracks the matches found for each pattern. The values in these maps are vectors that can become very large. For performance reasons, instead of completely deallocating those vectors after each scan, we were clearing the vector but retaining its capacity, so that they can be reused in later scans. The problem with that is that these vectors are not freed while the scanner is in use, and this can have a large impact on the process' memory footprint, causing OOM issues.This PR adds a
PatternMatchestype that encapsulates all the logic for tracking pattern matches, including a more sophisticated approach that tries to reuse the vectors as much as possible, but frees them when they grow too much.