search

hashicorp-forge / grove

A Software as a Service (SaaS) log collection framework.
https://hashicorp-forge.github.io/grove/
Mozilla Public License 2.0
130 stars 11 forks source link

Add zip_paths processor. Improve documentation. #39

Closed hcpadkins closed 1 year ago

hcpadkins commented 1 year ago

Overview

This pull-request introduces a new zip_paths processor. This allows to flattening lists of key / values into key / value pairs directly.

Additionally, this pull-request cleans up a few doc-strings and configuration samples.

hcpadkins commented 1 year ago

It should be noted that we run benchmarks of calling jmespath.compile and using the compiled expressions over compiling the expressions each invocation.

This did halve the time spend in the zip_paths processor but with increased complexity of the processors to track compiled expressions. As the size of this test was quite small, using around 8,000 log entries, this speed-up was around 2-seconds of wall clock time.

As a result, this change set has not been integrated into this pull-request. In future, if we find that these processors are beginning to introduce a performance concerns we can look to optimise them and ensure that features such as compile are used for frequently called operations like jmespath and re.