grafana / loki

Like Prometheus, but for logs.
https://grafana.com/loki
GNU Affero General Public License v3.0
23.96k stars 3.46k forks source link

perf(blooms): Replace JSON lib for encoding/decoding metas #14767

Closed chaudum closed 2 weeks ago

chaudum commented 2 weeks ago

What this PR does / why we need it:

A lot of CPU time is spent in decoding the meta files, which are JSON files.

See this CPU profile from a trace:

screenshot_20241105_101340

github.com/json-iterator/go is a drop-in replacement for the Go stdlib encoding/json.

Benchmark

$ benchstat old.txt new.txt 
goos: linux
goarch: amd64
pkg: github.com/grafana/loki/v3/pkg/storage/stores/shipper/bloomshipper
cpu: 11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz
                                     │   old.txt    │               new.txt                │
                                     │    sec/op    │    sec/op     vs base                │
_Fetcher_processMetasCacheResponse-8   4.318µ ± 12%   2.095µ ± 49%  -51.49% (p=0.000 n=10)

                                     │   old.txt   │              new.txt               │
                                     │    B/op     │    B/op     vs base                │
_Fetcher_processMetasCacheResponse-8   1301.0 ± 0%   869.0 ± 0%  -33.21% (p=0.000 n=10)

                                     │  old.txt   │              new.txt               │
                                     │ allocs/op  │ allocs/op   vs base                │
_Fetcher_processMetasCacheResponse-8   27.00 ± 0%   15.00 ± 0%  -44.44% (p=0.000 n=10)

Checklist