deadtrickster / prometheus.erl

Prometheus.io client in Erlang
MIT License
341 stars 117 forks source link

Write bigger chunks to ram_file #136

Closed binarin closed 2 years ago

binarin commented 2 years ago

ram_file is implemented as a port driver, and has quiet a significant overhead if every single series is emitted separately.

What happens here is that bigger chunk is collected before writing to ram_file. Chunk is collected using binary append, in such a way that is can benefit from binary append optimization in ERTS.

One nice side-effect of collecting binaries is that a lot of iolists in the hottest code path are no longer created, putting significantly less pressure on the garbage collector.

Observed performance gains can be as high as 20 times (700k metrics emitted by RabbitMQ went from 2 minutes to 5 seconds). Such big difference can be mostly attributed to garbage collection overhead (for some reason it's non-linear in the number of metrics), but profiling shows that 20% improvement is there even without accounting for gc.

deadtrickster commented 2 years ago

@binarin please look at the pipeleines - some minor code style things