Closed tomaka closed 1 year ago
While working on https://github.com/paritytech/cargo-contract/issues/988 we ran into the issue of missing child tries in smoldot. More specifically we tried to get https://github.com/AcalaNetwork/chopsticks working with ink! contracts.
Missing child tries is a blocker for anything to do with pallet-contracts
on smoldot and it would be greatly appreciated if it would happen at some point. Currently we unfortunately can't use smoldot for smart contracts.
The next step in adding support would be to update runtime_host.rs
.
This basically contains the logic of the pending changes that have been made to the storage by the runtime but not saved yet. It also contains the logic of "transactions", where the runtime can potentially revert some changes that it has made to the storage.
Adding support for child tries in an efficient way (I'm not looking for the most optimal implementation, but at least the best possible O
-notation complexity) seems extremely complicated, although I haven't precisely looked into it yet.
After runtime_host.rs
is updated, there's also the database code to update (i.e. making the database store the child tries). Other than that, the rest of the changes should be mostly just API changes.
@tomaka Would you be open to writing out a W3F grant for someone to implement this?
For context: We need this not only for using light clients with smart contract Dapps, but also for some more advanced testing tooling and some features that auditors of ink! contracts requested.
Writing a grant would probably take me way more time than doing it myself
I was honestly hoping for this response 😜 .
So do you think there's a chance of implementing it some time in the next weeks?
Honestly the biggest obstacle is that the way child tries work is a bit of an illogical shit show, and I have the biggest trouble in the world finding someone who is able to explain me how they work in Substrate.
My previous attempts at shaking things up a bit (https://github.com/w3f/polkadot-spec/issues?q=is%3Aissue+is%3Aopen+child+tries) haven't been very successful.
This is quite demotivating. Everything would be a degree of magnitude more simple if we had a document explaining clearly how they work.
https://github.com/smol-dot/smoldot/issues/272 should to be tackled first in order to avoid having to maintain a cache of each child trie. Removing the caches would considerably simplify the implementation. An alternative could be to just not have a cache for child tries, but that would obviously not be a long term solution. Having no cache first would just make #272 more difficult to do.
I've noticed a bunch of people follow this issue, so to keep you updated:
runtime_host.rs
.Note that when it comes to using Chopsticks with child tries, Chopsticks will need a non-trivial update for #639.
If someone is willing to help, having test cases would be extremely appreciated. Since I'm unfamiliar with runtime development and Substrate APIs in general, just creating a contracts chain would probably take me ten times the time necessary to implement the feature. And given that the way child tries work is very poorly documented, these test cases would be extremely useful.
Ideally, I would need as test fixture the header and body of a block that uses child tries in some way (e.g. creating, executing, or deleting a contract) plus the storage of its parent. If the parent is the genesis block, then the storage can be in the chain spec.
Remains to do in order to have something potentially working:
runtime_host.rs
.StorageGet
/NextKey
/ClosestDescendantMerkleValue
to indicate from which trie to read from.If someone is willing to help, having test cases would be extremely appreciated. Since I'm unfamiliar with runtime development and Substrate APIs in general, just creating a contracts chain would probably take me ten times the time necessary to implement the feature. And given that the way child tries work is very poorly documented, these test cases would be extremely useful.
Ideally, I would need as test fixture the header and body of a block that uses child tries in some way (e.g. creating, executing, or deleting a contract) plus the storage of its parent. If the parent is the genesis block, then the storage can be in the chain spec.
@pgherveou Could you help here? The overall context for this issue is that it's needed to enable light client support of pallet-contracts
.
@tomaka looking into your request, if you want to give me more info happy to jump on an element (@pg:parity.io) / google meet call
following up here,
We have test cases in pallet-contracts
that does things that are similar to what you described above, for example in this lazy_removal
tests, we compile a basic contract from a .wat file, put some data in it's child_trie storage and check if things works as expected upon termination.
These tests relies on sp_io::TestExternalities
(see https://docs.substrate.io/test/unit-testing/#test-storage-in-a-mock-runtime) to test storage in a mock environment.
With this context, let me know whether or not we could adapt a similar setup in smol-dot to provide the test cases that you mentioned above.
Hi,
I have zero knowledge or motivation to figure out how Substrate's APIs work. What I'm looking for is raw data plus a written (non-code) explanation of what this data is.
One way could be to dump the entire content of the storage of block N, plus the header and body of block N+1. This is only useful if block N+1 modifies child tries in some way, preferably in the most interesting way possible (such as creating or deleting one).
After https://github.com/smol-dot/smoldot/pull/684, child tries support is supposed to work in the light client. In other words, if I was sure that the code didn't contain any bug, I would close this issue as there's nothing left to do. However, given that everything was implemented through guesswork, I'll keep this issue open before there's reasonable confidence that things work.
If you could give me an up-to-date chain spec of a contracts chain so I can try it out, the bootnodes of the one in this repository are all unresponsive, and the @paritytech repo that contains chain specs and that I know exists is for some reason private.
EDIT: found it https://github.com/paritytech/cumulus/blob/master/parachains/chain-specs/contracts-rococo.json
I've opened https://github.com/w3f/polkadot-spec/issues/648 for what it's worth.
Ok got it, I could try to run these tests in a "real" environment and export the raw storage. Is there any specific DB backend or data format export I should use so that it can be consumed in your tests?
For the format, please not a RocksDB/ParityDB database file, as I can't open that without digging through the internals of Substrate. Just JSON or whatever. I can always re-convert myself. If the test creates a new chain from scratch and creates a block 1 (I assume it does? not sure), then exporting the chain specification works, as it contains the storage of block 0.
Another way to create a test, which I could do myself, could be to ask a node for a call proof for Core_execute_block
with a block that uses child tries. The test would be more messy, but in terms of coverage it would be similar.
Alternatively, is it maybe possible to trace a block in Substrate, for debugging purposes or something? It's again not something I'm familiar with.
Basically, there are two problems:
A test that verifies a block solves the first problem (which is good), but not the second one. Furthermore, if the test fails, maybe it's caused by the second problem. Verifying a block is unfortunately very binary: either it succeeds or it fails, and you can't easily figure out why it fails.
In order to solve the second problem, I would need either someone to answer in https://github.com/w3f/polkadot-spec/issues/648, or a trace of a block that actually reaches some of the corner cases.
Ok here what I can do easily Let me know if that would fit your needs, and I will get to it:
pallet-contracts
and also uses the manual-seal
consensus where blocks are authored at every transaction.That could work!
create and run a simple rust script to export rocksdb keys and values to JSON, I can export before and after running the contract call transaction
I specifically need the storage values of the trie, not the key-values that RocksDB contains. Substrate unfortunately has dozens of incomprehensible layers of abstraction between the trie and RocksDB.
This needs to be for the block that touches contracts is authored.
use the node export-blocks command to export the relevant blocks
I'm not sure in which format the blocks are going to be. I'm afraid that it might also export some opaque data structure that can only be deciphered by looking deep inside the internals of Substrate. But I'm not sure of what I'm saying here.
testing the exports block command, instantiating a dummy flipper contract does the following this is the JSON that I get:
{
"block": {
"header": {
"parentHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
"number": "0x0",
"stateRoot": "0xd7a3ce9f2563d88e75baa2efe0a247e07e4e048e65543de49ed4ce36a615fe80",
"extrinsicsRoot": "0x03170a2e7597b7b7e3d84c05391d139a62b157e78786d8c082f29dcf4c111314",
"digest": {
"logs": []
}
},
"extrinsics": []
},
"justifications": null
}
{
"block": {
"header": {
"parentHash": "0x2aa8db7e12a525e68f43484ec634ad9254cf4133ba947c7748f11b69216cc62b",
"number": "0x1",
"stateRoot": "0x2da0f668c29f006d3a69fecc771f5ee88ff466df8d4f314c00bd6634da53c2bc",
"extrinsicsRoot": "0x7a2327f12e4a5a4c8929e098036346570e43ff82e9902782f94545efacdc8700",
"digest": {
"logs": []
}
},
"extrinsics": [
"0x280403000b1011fe998801",
"0x01198400d43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d013c4d5831f171022ea3ed5f8158807b510b81d39b6438eeb13255932d023a132ec79c4e08dd8af41436c531db3757087bc746ecbdb90ae4d906e54bd75209ad820b00000008070072e23c21c6040100010780edff7d78ed150061736d0100000001240760027f7f0060000060047f7f7f7f017f60037f7f7f006000017f60017f017f60017f00027406057365616c310b6765745f73746f726167650002057365616c301176616c75655f7472616e736665727265640000057365616c3005696e7075740000057365616c320b7365745f73746f726167650002057365616c300b7365616c5f72657475726e000303656e76066d656d6f727902010210030b0a000304050001060001010608017f01418080040b0711020463616c6c000d066465706c6f79000e0aaa090a2601017f230041106b22022400200220003a000f20012002410f6a41011006200241106a24000b6f01037f024002402000280208220320026a22042003490d00200420002802044b0d00200420036b2002470d01200028020020036a210541002103037f2002200346047f200505200320056a200120036a2d00003a0000200341016a21030c010b0b1a200020043602080f0b000b000b5502027f027e230041206b22002400200041106a22014200370300200042003703082000411036021c200041086a2000411c6a10012001290300210220002903082103200041206a2400410541042002200384501b0b3f01017f2000280204220145044041020f0b2000200141016b36020420002000280200220041016a3602004101410220002d000022004101461b410020001b0b29002000027f20014504404180800441003a000041010c010b418080044181023b010041020b100c000b12004180800441003b010041004102100c000b990101047f230041206b220124002001428080013702142001418080043602102001410036020c200141106a2001410c6a4104100602402001280214220420012802182202490d0020012802102103200141003602182001200420026b3602142001200220036a3602102000200141106a10052001280218220020012802144b0d00200320022001280210200010031a200141206a24000f0b000b0d0020004180800420011004000ba20301057f230041106b22002400024002400240100741ff01714105470d00200041808001360200418080042000100220002802002201418180014f0d0020014104490d02418080042802002201411876210220014110762104200141087621030240200141ff0171220141e3004704402001412f47200341ff01714186014772200441ff017141db0047720d0441002104200241d901460d010c040b200341ff0171413a47200441ff017141a50147720d0341012104200241d100470d030b2000428080013702042000418080043602002000410036020c20002000410c6a410410062000280204220320002802082201490d00200028020021022000200320016b220336020020022001200120026a22012000100020032000280200220249720d0020002002360204200020013602002000100841ff017122014102460d002004450d01200145100b410041001009000b000b230041106b220024002000418080043602004180800441003a0000200042808081801037020420014100472000100520002802082200418180014f0440000b41002000100c000b410141011009000bf00101057f230041106b2200240002400240100741ff01714105470d0020004180800136020c418080042000410c6a1002200028020c2201418180014f0d00024020014104490d002000418480043602002000200141046b36020441808004280200220141187621022001411076210320014108762104200141ff0171220141ed014704402001419b0147200441ff017141ae014772200341ff0171419d0147200241de004772720d012000100841ff017122004102460d012000100b100a000b200441ff017141cb0047200341ff0171419d0147720d002002411b460d020b410141011009000b000b4100100b100a000b149bae9d5e010901307866303737313362316431326339646335363163366436646363653635363532653538333466323166663932313835386263333566333734623432353837303437"
]
},
"justifications": null
}
I specifically need the storage values of the trie, not the key-values that RocksDB contains. Substrate unfortunately has dozens of incomprehensible layers of abstraction between the trie and RocksDB.
I can probably find out what are the relevant keys, maybe by diffing and filtering the new entries. You would need all the key and values for the path and the actual values stored right?
testing the exports block command, instantiating a dummy flipper contract does the following this is the JSON that I get:
{ "block": { "header": { "parentHash": "0x0000000000000000000000000000000000000000000000000000000000000000", "number": "0x0", "stateRoot": "0xd7a3ce9f2563d88e75baa2efe0a247e07e4e048e65543de49ed4ce36a615fe80", "extrinsicsRoot": "0x03170a2e7597b7b7e3d84c05391d139a62b157e78786d8c082f29dcf4c111314", "digest": { "logs": [] } }, "extrinsics": [] }, "justifications": null } { "block": { "header": { "parentHash": "0x2aa8db7e12a525e68f43484ec634ad9254cf4133ba947c7748f11b69216cc62b", "number": "0x1", "stateRoot": "0x2da0f668c29f006d3a69fecc771f5ee88ff466df8d4f314c00bd6634da53c2bc", "extrinsicsRoot": "0x7a2327f12e4a5a4c8929e098036346570e43ff82e9902782f94545efacdc8700", "digest": { "logs": [] } }, "extrinsics": [ "0x280403000b1011fe998801", "0x01198400d43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d013c4d5831f171022ea3ed5f8158807b510b81d39b6438eeb13255932d023a132ec79c4e08dd8af41436c531db3757087bc746ecbdb90ae4d906e54bd75209ad820b00000008070072e23c21c6040100010780edff7d78ed150061736d0100000001240760027f7f0060000060047f7f7f7f017f60037f7f7f006000017f60017f017f60017f00027406057365616c310b6765745f73746f726167650002057365616c301176616c75655f7472616e736665727265640000057365616c3005696e7075740000057365616c320b7365745f73746f726167650002057365616c300b7365616c5f72657475726e000303656e76066d656d6f727902010210030b0a000304050001060001010608017f01418080040b0711020463616c6c000d066465706c6f79000e0aaa090a2601017f230041106b22022400200220003a000f20012002410f6a41011006200241106a24000b6f01037f024002402000280208220320026a22042003490d00200420002802044b0d00200420036b2002470d01200028020020036a210541002103037f2002200346047f200505200320056a200120036a2d00003a0000200341016a21030c010b0b1a200020043602080f0b000b000b5502027f027e230041206b22002400200041106a22014200370300200042003703082000411036021c200041086a2000411c6a10012001290300210220002903082103200041206a2400410541042002200384501b0b3f01017f2000280204220145044041020f0b2000200141016b36020420002000280200220041016a3602004101410220002d000022004101461b410020001b0b29002000027f20014504404180800441003a000041010c010b418080044181023b010041020b100c000b12004180800441003b010041004102100c000b990101047f230041206b220124002001428080013702142001418080043602102001410036020c200141106a2001410c6a4104100602402001280214220420012802182202490d0020012802102103200141003602182001200420026b3602142001200220036a3602102000200141106a10052001280218220020012802144b0d00200320022001280210200010031a200141206a24000f0b000b0d0020004180800420011004000ba20301057f230041106b22002400024002400240100741ff01714105470d00200041808001360200418080042000100220002802002201418180014f0d0020014104490d02418080042802002201411876210220014110762104200141087621030240200141ff0171220141e3004704402001412f47200341ff01714186014772200441ff017141db0047720d0441002104200241d901460d010c040b200341ff0171413a47200441ff017141a50147720d0341012104200241d100470d030b2000428080013702042000418080043602002000410036020c20002000410c6a410410062000280204220320002802082201490d00200028020021022000200320016b220336020020022001200120026a22012000100020032000280200220249720d0020002002360204200020013602002000100841ff017122014102460d002004450d01200145100b410041001009000b000b230041106b220024002000418080043602004180800441003a0000200042808081801037020420014100472000100520002802082200418180014f0440000b41002000100c000b410141011009000bf00101057f230041106b2200240002400240100741ff01714105470d0020004180800136020c418080042000410c6a1002200028020c2201418180014f0d00024020014104490d002000418480043602002000200141046b36020441808004280200220141187621022001411076210320014108762104200141ff0171220141ed014704402001419b0147200441ff017141ae014772200341ff0171419d0147200241de004772720d012000100841ff017122004102460d012000100b100a000b200441ff017141cb0047200341ff0171419d0147720d002002411b460d020b410141011009000b000b4100100b100a000b149bae9d5e010901307866303737313362316431326339646335363163366436646363653635363532653538333466323166663932313835386263333566333734623432353837303437" ] }, "justifications": null }
It's a bit of an annoying format, but I can do with that with some manual conversions
I can probably find out what are the relevant keys, maybe by diffing and filtering the new entries. You would need all the key and values for the path and the actual values stored right?
I don't need the new entries, I need the old entries. I need the values before the block. Specifically I need the values that are read by the block (including branch nodes), but since it's hard for you to know which values are read and which aren't, we'll do have to do with all values.
It's a bit of an annoying format, but I can do with that with some manual conversions
There is also a binary format, but I assumed that it would not be helpful in this context
I don't need the new entries, I need the old entries. I need the values before the block. Specifically I need the values that are read by the block (including branch nodes), but since it's hard for you to know which values are read and which aren't, we'll do have to do with all values.
That would be in the POV if this was a parachain in Polkadot right, let me see if I can try to figure this out.
I guess the easier would be to only call the contract in the block vs uploading and instantiating it, that should reduce the number of keys that are read. The call would only read and update a storage key that are stored in the contract child trie.
That would be in the POV if this was a parachain in Polkadot right, let me see if I can try to figure this out.
That would give a more complicated and less isolated test because I have to decode the proof and all. As I've mentioned above, I can ask for POVs by sending a request to a node, I don't really need your help for that.
Until I find a more user friendly way to share this:
Here is what I did and the output:
I ran substrate-contracts-node
, and uploaded and instantiated a simple smart contract that set a String in storage.
The zip contains the rocksdb export of the chain at block 0 and 1, and the json block exports
There are a few keys that are added in block 1, they correspond to the new accounts that are created by the contract, and the various storage that are created for managing the contract and it's storage, if that helps I can give you names for all of these keys
Ah thanks, that looks promising!
However I don't understand why the runtime code is not stored at :code
(0x3a636f6465
), like it's always supposed to be, but instead at 0x3a636f646532b090ba46cc0bac21ab1b180c4f1649242c7a7db2b199881880a8a6d4466ad9
Is it a particularity of the contracts chain?
mm good point, I think this last key is the code of the wasm contract not the runtime. First time, I am digging into the guts of substrate, let me double check these values as well and get back to you.
Also Basti told me about these APIs that might also give us similar informations, https://paritytech.github.io/substrate/master/sc_rpc_api/state/trait.StateApiClient.html#method.trace_block
I do this the runtime code under (0x3a636f6465
) when I query it from polkadot.js
there might be something missing from my naive db extract here : https://github.com/pgherveou/rocksdb_export_json/blob/main/src/main.rs
actually this is in the JSON under 3a636f646532b090ba46cc0bac21ab1b180c4f1649242c7a7db2b199881880a8a6d4466ad9
not sure why it's not just 3a636f6465
sorry that's exactly what you said above haha
there might be something missing from my naive db extract here : https://github.com/pgherveou/rocksdb_export_json/blob/main/src/main.rs
It's what I mentioned above: there are tons of layers of abstractions on top of RocksDB. Probably that 32b090ba46cc0bac21ab1b180c4f1649242c7a7db2b199881880a8a6d4466ad9
is the block hash or state root or something.
fyi this is the storage hash
What would be the next steps to make these dataset useful for your test cases?
What would be the next steps to make these dataset useful for your test cases?
Well, do you know why some entries have a storage hash but not all?
Without understanding this I can't really do if key.len() > 32 { key.truncate(key.len() - 32) }
, as it might alter legitimate keys.
I think only the leaves keys have the storage hash. also as it is exported now, the db hold the previous and current keys, (ie old value are not pruned) What would be the ideal export state for you to make this useful?
I would need either all the trie nodes with their value (both branch nodes and non-branch nodes), or just the trie nodes that have a value. The former would be better, because if you give me the latter I calculate the former from the latter. No storage hash or anything like that, just keys and values. I also need only the current keys. I only need block 0, so I assume that this is not a problem because there's no previous value.
I don't know how to phrase that, but basically the simplest thing possible. I don't really care if Substrate likes to make things complicated and mix data in a weird way. I don't need storage hashes or previous values or the age of the mother of the developer who wrote the code. Just keys and values.
I am just trying to help here, for me "simplest" is giving you the raw output that I get from substrate, I understand that smoldot is working without the same abstractions, that's why I am asking what format works best for you.
What I mean by "simplest" is "simple" in terms of purity, not what is simple for you to generate. For example just a value is more simple than a value plus its hash concatenated together. I am generally frustrated by Substrate being over-engineered and not being able to generate simple things.
To give a comparison, imagine if I needed to know the color of a specific pixel in a PNG image, but instead of just the color of the pixel I get the color plus the Huffman table used to decode it concatenated together. I don't care about the Huffman table, I just need the color, and to me "just the color" is more simple than "color and Huffman table".
I understand that generating this can be a frustrating time sink, it's precisely why I asked for help from potentially someone for which it is less a frustrating time sink than it is for me.
Ok looked into it a bit more this pm It turned out that trying to dump the db as JSON was indeed not the best approach. Luckily, Frame exposes some useful rpc state endpoints to get the raw data:
combining these into a small cli I was able to export the db and the storage change set for each block
Thanks! cc https://github.com/smol-dot/smoldot/pull/722
There are a few changes to make:
I need the database of block 0, not of block 1. So https://github.com/pgherveou/contracts-query/blob/630ad7d87dc718846226e92800d61f4e4e618064/src/main.rs#LL31C34-L31C38 and https://github.com/pgherveou/contracts-query/blob/630ad7d87dc718846226e92800d61f4e4e618064/src/main.rs#LL42C54-L42C58 need to be the hash of block 0 rather than None
Don't use getKeysPaged
(https://github.com/pgherveou/contracts-query/blob/630ad7d87dc718846226e92800d61f4e4e618064/src/main.rs#L133) but getKeys
. Right now you're getting only the first 100 keys, and not all of them.
I'm also using the block 1 from your comment a few days ago, is it still the same? I suggest that your script also retrieves block 1 so that we're sure that they correspond.
On my phone now, will update these last few things, later today.
Regarding paging I think there are less than 100 keys so the Api call should pick all keys in one pass.
Regarding previous data, we should not use it, I tweaked the chain to remove transaction fees and deposits in an attempt to get fewer db changes so I could try to understand what's going on at a low level at bit better
I can put all of this data extraction in a shell script if that's useful at all. All is needed is the substrate-contract-node, cargo-contract to compile and execute a contract transaction and this little repo to query the data
:+1: There's absolutely no rush, by the way
Here you go: Archive.zip database is exported at block 0 added block 1 in blocks.json
source if you want to check it https://github.com/pgherveou/contracts-query/blob/404079d270cd7c7e4ae98e4eb42ac49996af1634/src/main.rs#L50-L93
As commented, the design of child tries is a bit confusing. An issue should be opened in the spec repo to ask for some of the behaviors to be specified. See the
TODO
s inhost.rs
for a list of unclear behaviors.