search

elastic / hey-apm

Basic load generation for apm-server built on hey
Apache License 2.0
16 stars 16 forks source link

Occassional benchmark failure in hey-apm-benchmark tests #176

Closed cachedout closed 4 years ago

cachedout commented 4 years ago

Hi @jalvz

We (@elastic/observablt-robots ) are seeing the occasional failure in the hey-apm benchmark tests. Below are a few examples from the past couple of weeks:

https://apm-ci.elastic.co/blue/organizations/jenkins/apm-server%2Fapm-hey-test-benchmark/activity

Could you have a look? No hurry, of course. We would love to find a way to make these Jenkins build more stable. Thanks!

jalvz commented 4 years ago

I've been following for a while, and this seems to work as intended, ie: test failures are because apm-server performance actually decreased.

Many times is legit (code changes) and expected, some other times are unexpected and we act on it, and some other times are probably because of small environmental differences (network latency, etc).

It's not perfect, but we are generally getting out what we need from these tests, so I'll go ahead and close this issue.

If you have any concerns/questions or find something suspicious feel free to reopen!