sinkingpoint
commented
2 months ago As a demonstrative, we were chewing 70 cores using go-syslog through the OpenTelemetry Collector, which reduced to 3 when introducing a sync.Pool to store/reuse bufio.Readers
pprof:
At the moment, every parser (one of which is constructed for each message), allocates a new buffer (https://github.com/leodido/go-syslog/blob/develop/octetcounting/scanner.go#L42). In high throughput scenarios, this can chew up a huge amount of CPU allocating memory to parse messages. I would like to propose that we create a new
sync.Poolthat storesbufio.Readers which https://github.com/leodido/go-syslog/blob/develop/octetcounting/scanner.go#L40 can pull from, rather than allocating a new buffer every time.