search

paritytech / bench-bot

ISC License
9 stars 19 forks source link

Perform performance degradation check before deploying #48

Open joao-paulo-parity opened 3 years ago

joao-paulo-parity commented 3 years ago

We should have a workflow in the deploy process where before deploying a new version

  1. Revert the bot's code to a known good commit
  2. Revert the Substrate to a known good commit
  3. Run a benchmark
  4. Update the bot's code
  5. Run the benchmark again
  6. Compare the results

This would avoid confusion such as the one seen in https://github.com/paritytech/bench-bot/pull/47 because we'd know upfront which commit introduced a regression when the bot is updated.

It's relevant for this workflow to be executed directly in the machine, rather than e.g. CI, so that the results can be compared more predictably.

joao-paulo-parity commented 3 years ago

This would be useful for #49 since we don't want to go through #47 again.

At the moment this "sanity degradation check" will have to be performed manually.

ggwpez commented 2 years ago

I like the idea, it's similar to #56.
Instead of running the benchmark twice, the previous run can be saved into a file when executed with --raw.
Making these files available would be a next big step and would allow to compare performance across commits.

There is this #51 refactor issue open. Do you think this should be done first?
@joao-paulo-parity

joao-paulo-parity commented 2 years ago

There is this #51 refactor issue open. Do you think this should be done first?

I consider #68 to be the biggest problem of processbot right now. That is what I would work on if I were available to work on this project. If #68 can't be figured out because of Node.js, then the suggestion of https://github.com/paritytech/bench-bot/issues/51#issuecomment-1009834590 incurs a rewrite (in another language) rather than a refactor. In any case, I would not work on any new feature, including this one, until #68 is dealt with.

My personal priorities are not a blocker for the project and developers are free to work on whatever they find most relevant. If it were up to me then, to answer your question, no, I would not work on this first because I would prioritize something else. If another developer wants to work on this ticket, or any other for that matter, I don't see a problem with that.