This PR adds a new method to the MatcherPool that avoids creating the intermediate list of rules, which should reduce the memory footprint of those operations.
How to test
The automated tests should pass, and should convince you that the method is correct.
(re)publish a rule in CODE with this branch deployed. Does the resulting re-ingest knock over the box?
What does this change?
We publish a rule count when we ingest rules artefact (and run a healthcheck), and that operation needlessly creates another complete list of rules, which is likely a fair amount of memory (there are ~338,000 rules at time of writing.) We seem to be crashing at the point of getting that list (after publication): https://logs.gutools.co.uk/s/editorial-tools/app/discover#/doc/67604290-baa6-11e9-bea2-633437abb232/logstash-ed-tools-2023.10.16?id=C9d6N4sBMaZrWQ-DVxC1
This PR adds a new method to the MatcherPool that avoids creating the intermediate list of rules, which should reduce the memory footprint of those operations.
How to test