search

nodejs / Release

Node.js Release Working Group
4.03k stars 570 forks source link

Running benchmarks during release #479

Open BridgeAR opened 5 years ago

BridgeAR commented 5 years ago

We recently ran into some significant performance drop due to some changes in-between releases.

Our current microbenchmarks partially take a very long time but we could start evaluating what microbenchmarks we could run before we release a new version to detect significant performance drops early on.

mhdawson commented 5 years ago

Potentially related: https://github.com/nodejs/benchmarking/issues/293

Also on what day did the changes land which caused the regression? I'm curious if we could have caught it from: benchmarking.nodejs.org ? Throughput does look ~5% lower on 12.x than 10.x in the graphs

MylesBorins commented 5 years ago

The benchmarks we saw regressions on at google cloud were under large CPU load. I have a way to reproduce, but am unsure if our benchmark suite is checking this specific case.

On Thu, Oct 3, 2019, 3:45 PM Michael Dawson notifications@github.com wrote:

Potentially related: nodejs/benchmarking#293 https://github.com/nodejs/benchmarking/issues/293

Also on what day did the changes land which caused the regression. I'm curious if we could have caught it from: benchmarking.nodejs.org ? Throughput does look ~5% lower on 12.x than 10.x in the graphs

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/nodejs/Release/issues/479?email_source=notifications&email_token=AADZYV65QAK3QAW37JXE62LQMZDVBA5CNFSM4I4NG6R2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAJLYYY#issuecomment-538098787, or mute the thread https://github.com/notifications/unsubscribe-auth/AADZYVZMW6XLMOHMPEWBJPDQMZDVBANCNFSM4I4NG6RQ .

targos commented 5 years ago

As discussed during the last meeting, the next step is to make sure we have a CI job that can run in a reasonable amount of time so we can use it for releases.

BridgeAR commented 4 years ago

The CI job that we can use for our microbenchmarks is:

https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks

The only problem with the job is that we have to explicitly name a module that we want to check for. We can not just run all of them. I am not sure but I believe our micro-benchmarks setup does not provide a run all version currently, so we have to fix that first (but I will check about that again).

We also likely have to trim the run time of some of our benchmarks as they sometimes have a very long run time.

BridgeAR commented 4 years ago

I removed it from the agenda as we already discussed it properly and we just have to now improve the way we do this.

mhdawson commented 4 years ago

@BridgeAR from what I understand a run all would take days....

So very much would need to stip down to a subset.