search

Joystream / joystream

Joystream Monorepo
http://www.joystream.org
GNU General Public License v3.0
1.43k stars 114 forks source link

Making sure benchmarks don't depend on genesis config values #4384

Open Lezek123 opened 1 year ago

Lezek123 commented 1 year ago

There are values in the genesis config that affect the weight of some extrinsics.

One example of this is the project-token's bloat bond. If its value is 0, then no funds are being transferred when a new project-token account is created. If it's non-zero, there is an additional transfer involved in this case, which means there are additional read and write operations.

As discoverd in https://github.com/Joystream/joystream/pull/4374, a few benchmarks produce different weight functions when those genesis config values are changed.

Why is this a problem?

  1. Benchmarks are currently ran on --dev chain, which has a slightly different genesis config than production chain
  2. The values in the genesis config are just initial configuration values. They are part of the runtime state and they can usually be changed later, for example, via a proposal or by a working group lead.

What are the solutions?

  1. We can run benchmarks with the same or almost the same genesis config as production chain. This will only solve the issue for values that are immutable even though they are part of genesis config, which by itself seems kind of contradictory to me. If I understand correctly, the issue of immutable values being part of genesis config was also raised here: https://github.com/Joystream/joystream/issues/4277
  2. We fix the benchmarks that depend on genesis config values to use worst-case-scenario instead. Given a project-token example, we assume the bloat bond is always non-zero. This is already the case for most of the benchmarks, there are just a few where this rule was not applied (simply because a specific case was overlooked), like content.issue_creator_token.
mnaamani commented 1 year ago

Great write up. Solution 1 is fairly straightforward - https://github.com/Joystream/joystream/pull/4385

But Solution 2 is obviously the correct approach. How long would it take to implement the fixes?

Lezek123 commented 1 year ago

But Solution 2 is obviously the correct approach. How long would it take to implement the fixes?

I think I can fix issues identified in https://github.com/Joystream/joystream/pull/4374 within an hour