Testing framework for runtimes

[Demi-Marie]

Testing runtimes is currently difficult in practice. While it seems to only involve passing a few blocks to the runtime and seeing what the runtime does, there are no convenience functions for this.

[kianenigma]

Okay, this is a bit vague: testing the runtime can be hard but in my opinion ONLY because of the boilerplate needed to set up the externalities. Afterwards, you can perfectly simulate anything that you might need including processing blocks with just single calls to basic meta-modules like system and timestamp.

What is quite annoying is:

• The setup
• No structure: some modules have very basic setup and some have builders.
• Maybe lack of some auxiliary functionalities to be added to the builder pattern. At some point, I was quite keen on checking some piece of code after each test but did not manage to do that for example.

If the issue concerns above then we have another tracking issue for it: paritytech/polkadot-sdk#367

[tomusdrw]

I think there are two separate things being spoken of:

1. Unit testing of runtime modules (which is what @kianenigma is mentioning) - that requires a lot of boilerplate to mock the externalities
2. Testing the entire runtime via submitting blocks (which I believe is what @DemiMarie-parity has in mind).

For the latter we have core/test-client and in node/testing there is an example how you can create an extension trait to de-boileplate common tasks in a particular runtime.

[kianenigma]

ahh, I see. Okay so on top of your recommendation, node/executor also includes a bunch of tests that exactly do: create som blocks -> create an executor -> pass the block to runtime.

I usually use it for e2e tests over the real substrate node runtime. Though at the moment it is using the binaries of node/runtime directly and it cannot be changed.

[tomusdrw]

@kianenigma core/test-client is completely runtime-generic. We have one extension that is using node/runtime (used inside node/executor) and another one that is using test-runtime - in core/test-runtime/client. So it's open to be used with any custom runtime is there, but we might lack some examples and docs how to wire things up.

[Demi-Marie]

@kianenigma @tomusdrw I think the bigger issue is less the lack of facilities, and more the lack of documentation on how to use them. Should I open another issue about developer docs?

[tomaka]

Should I open another issue about developer docs?

The intention is good, but 95% of the Substrate code base lacks documentation on how to use it. I don't think it's useful to open an issue specifically about test-client's docs.

[rphmeier]

The only thing that's lacking IMO is documentation. I'm in agreement that the externalities setup is a little tedious/mysterious for testing of runtime modules.

I can also see some level of property-based testing to be valuable -- for instance, rather than calling runtime functions directly, you call them indirectly where you have to say under what circumstances a function is callable. The specific case I have in mind is whether it's called before initialization or after -- trait implementations for e.g. session are often called before explicit initialization and must initialize or partially initialize implicitly. The test gets run a few times with various configurations to make sure that it's valid under all call-orders.

Testing the whole thing in tandem is going to be difficult simply due to the number of combinations. I think we will probably need a separate tool for creating such tests, like the Ethereum state-test suite.

[Demi-Marie]

@rphmeier hopefully https://github.com/runtimeverification/polkadot-verification will deal with this (by proving the absence of bugs).