swift-server / swift-prometheus

Prometheus client library for Swift
https://swiftpackageindex.com/swift-server/swift-prometheus
Apache License 2.0
142 stars 30 forks source link

Add Benchmarks for metrics #106

Open freef4ll opened 11 months ago

freef4ll commented 11 months ago

Motivation

Adding benchmark tests in order to aid development and avoid performance regressions.

Benchmark is added in the same fashion as for swift-nio and swift-certifcates.

Modifications

Benchmarks are added for incrementing a counter, gauge, histogram and an export of 5000 metrics from the esystem.

How to execute benchmark

cd Benchmarks
swift package benchmark

In Xcode can open the project like and use Instruments to profile:

open --env BENCHMARK_DISABLE_JEMALLOC=true Package.swift

Result

Benchmarks are available, and performance difference between v1 is great which only managed to increment 2-3k times per second.

freef4ll commented 11 months ago

Docker scripts have been updated, but having issues with thresholds:

+ export PATH=/root/.tools:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ PATH=/root/.tools:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ cd Benchmarks
+ swift package --disable-sandbox --scratch-path .build/5.8/ --allow-writing-to-package-directory benchmark --format metricP90AbsoluteThresholds --path Thresholds/5.8/
error: 'benchmarks': unknown package 'swift-prometheus' in dependencies of target 'PrometheusBenchmarks'; valid packages are: 'code', 'package-benchmark'

Split the benchmark code into the benchmark definition and separate files for the benchmarks under test

I have done this, and there are 2 implementations for Counter to see on which is the preferred one (one captures the allocation but is slower due to return of the closure).

freef4ll commented 11 months ago

I have fixed the SPM error and thresholds are being generated fine now; just need to decide of which code style implementations to keep.

freef4ll commented 11 months ago

@FranzBusch , could you have a look at the PR code style that is preferred? Counter #2 doesn't capture the allocation of the counter it self and is slower.

FranzBusch commented 11 months ago

@swift-server-bot add to allowlist

FranzBusch commented 11 months ago

@swift-server-bot test this please

freef4ll commented 11 months ago

I'm not sure I understand on what happened with https://ci.swiftserver.group/job/swift-prometheus-soundness-prb/31/ and what other PR state https://ci.swiftserver.group/job/swift-prometheus-soundness-prb/buildTimeTrend ; especially runs 27 and 25 which report the same result.

EDIT: fixed

hassila commented 7 months ago

Anything we can do to help move this forward?

ktoso commented 7 months ago

From my perspective this looks fine, any concerns @FranzBusch @fabianfett ?

ktoso commented 6 months ago

@swift-server-bot test this please

ktoso commented 6 months ago

Failures because of missing baseline: Could not find any matching absolute thresholds at path [Thresholds/swift-5.7.3-RELEASE/], failing threshold check.

@fabianfett I think had some opinions before we merged here?