search

nodejs / performance

Node.js team focusing on performance
MIT License
372 stars 7 forks source link

Minimal viable benchmarks #14

Open sheplu opened 1 year ago

sheplu commented 1 year ago

Following the discussion during the first meeting of the team a few topics were raised regarding benchmarks

Some issues were also discussed about benchmarks

I will grab and put the discussion we had yesterday (at least the key points made) so we can have a constructive talk around it

anonrig commented 1 year ago

We should keep in mind that certain benchmarks take a lot of time to execute right now. For example, running all URL benchmarks take 4 hours. (Referencing https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/1222/)

BethGriggs commented 1 year ago

https://github.com/nodejs/Release/issues/479 is old but relevant. If we had a small set of benchmarks that provided a reasonable amount of coverage/value, then we could consider running them as part of the release process (for specific, or even all releases).

The caveat is that they would need to take a similar amount of time as our builds and be reasonably easy to interpret to avoid adding too much time/effort to the already laborious release process. Maybe that could initially be achieved by setting a generous max regression % that if we go below, then we should investigate before shipping the release.

sxa commented 1 year ago

@sheplu I didn't manage to get on the call yesterday but it sounded from the recording like you had information from quite a few sources - are there more reference links that you can add into this issue so we can use this as an initial "hub" for collating information about what we've currently got?

sheplu commented 1 year ago

I was talking a bit linked to this issue https://github.com/nodejs/performance/issues/13

Here is the interesting links

Now that I list them here I was imagining more resources :D

Also a good talk to have is even if we don't benchmark everything every times we should have something to test some of the core feature on some "basic" use case