Speed up consignment validation

[nicbus]

intro

consignment validation is currently very slow, as can be seen by running rgb-lib's stress test

as detailed in #122, part of the slowness can be attributed to the code re-validating the same nodes multiple times, which will be fixed in that issue

issue

even with #122 fixed, validation can still be very slow for consignments with a long history, as all transitions (nodes) need to be validated again each time rgb-core's Validator::validate is called

once a transition has been validated, it might need re-validating in cases where any concealed or revealed values have changed. not all schemas should have this property, though. as an example, RGB20 transitions should be completely immutable so re-validation should be avoidable. doing so would yield a great speed-up when validating consignments with a long history

in theory, re-validation could be avoided (where possible) by e.g. having rgb-core's Validator::validate function take an optional list of nodes that don't need to be validated again. this way the list of known-good nodes is handled by caller

having the Validator::validate caller store the list of nodes that have already been validated to persistent storage (e.g. to an on-disk DB) should then massively shorten validation times after an RGB node restart, while avoiding additional dependencies on rgb-core's side

I suggest we start by avoiding re-validation where possible and then optimize other areas where speed can be improved (e.g. electrum calls)

electrum

slowness in validation is supposed to be mostly due to electrum calls taking a long time to complete, so replacing electrum with bp-node (currently being developed) as the indexer has been proposed to speed things up

from the stress test logs it's not clear if calls to the electrum server are the main bottleneck

considering that the stress test starts from a clean regtest blockchain and that all services are local, electrum queries are expected to be blazing fast. moreover, electrum query times should remaining pretty consistent as the test progresses, as there are just roughly a hundred blocks to be indexed and the number grows by only one each transfer

[fedsten]

Consignment example consignment_out.txt

[dr-orlovsky]

Solved by removing bulletproofs - 99% of time in RGB core was spent on bulletproof validation as a inside of grin_secp256k1zkp. In v0.10, since we temporarily not using bulletproofs, the issue is solved. There is still seconds spent somewhere in the RGB Tools code, but this is different issue, external to RGB Core.

In the future we will have to do a significant benchmarking of bulletproofs implementation before starting using it.