prombench: Uneven Load generation for `query_range`

prometheus / test-infra

Prometheus E2E benchmarking tool

Apache License 2.0

156 stars 67 forks source link

prombench: Uneven Load generation for `query_range` #629

Open bwplotka opened 10 months ago

bwplotka commented 10 months ago

Despite @bboreham improvements to loadgen (https://github.com/prometheus/test-infra/pull/618) we might need still some love to make the load exactly the same across two instances. For some reason it's uneven, and once uneven situation starts it persists through days:

link1

link2

This causes allocations, CPU use and HTTP latency metrics uncomparable across instances.

Acceptance Criteria

Both instances receives exactly the same load, no matter how much fast they compute it.

bwplotka commented 10 months ago

Also I am curious if it makes sense to load different things in phases. e.g. 1h one API, 1h second API. Would allow us to isolate issues better.