Investigate PoV sizes

[ordian]

gm

I wonder if you're aware of the PoV sizes of the blocks produced by both Basilisk on Kusama and HydraDX on Polkadot. Our scraping services show that the PoV contribution of each of your parachain is around 8GB per day, which is more than 1MB per PoV block on average (this is way above any other parachain).

Are you aware of this or know that might be contributing to the PoV size? (can also probably see the collators logs to confirm)

Given that there not enough extrinsics in the blocks to explain that, is it possible that you for example do some iteration in every block, reading some state? If not, that could indicate a bug in state proof recorder that includes some data it shouldn't.

[mrq1911]

i can confirm this, collator is producing 1MB+ proofs for as long i can see in collator logs (2 weeks). do you have historical data about PoV size?

[ordian]

I can only say it was still the case in July but don't have historical data below that.

[ordian]

Might be related: https://github.com/paritytech/polkadot-sdk/issues/1498

[burdges]

@cheme do you have a radix 2 flavor of https://github.com/paritytech/trie or whatever somewhere?

[cheme]

@cheme do you have a radix 2 flavor of https://github.com/paritytech/trie or whatever somewhere?

Had this branch from 2020 https://github.com/paritytech/trie/pull/84, but today it is a lot behind current trie crate (probably 1 or 2 days of work to update).

[mrq1911]

it looks like we are attaching code wasm in every block proof.

this is example list of tries we are building the proof from:

[
  '466f6465fec44d0052bc537646db8e0528b52ffd00585c1d054e5b860c145110482c271d80d16c5196e4581484cc1e9c3dd313e3c3f9ac6613bb37a83df4080b4d0d1b9842bc0d7ef794e718f12c146bcfd855b46032d5d01c95fa83d8fcb6e6a2c7d8fe6637217b6fb2e5de5ba69452ef159712eb12fe93814af633cfc07d058ae0d1f7855b72b41eb6f3ee44e2c4ae2ae1b4858f59116b7606db23962d616f276265d68f12f6f6ce42cc8fdaa367994b66f1672f4e5bf8e22f2859153d3a872a4252faa817e0941b25dbd9a3ab3ac75e515c8f05579cfbea3505386516773a390dc017bf7b0ea5cce28e59984787e3775ad9f79ed1d52b425a6cea0b073a5e9f674f7d6c7ab1c3a6be7268f3ca2c8ebe2e99b5bfce5ed176af9e9208f274c55d16700a84039bee859815a6e0cc0a5529302b7a7b10b3e4b77f8d031aa0ef4eaabe79e8eb9546661db1aff3308bbd5d79487bd5514759d5b16a3dbab26a3d94ebddbaf290b6d43b34afbe1ebb7a4ad8a333618f6064d576cc627d311b9c32a7ae40e12e3deb72f9f4e5439bdfa557e77848e795efa85d3fdca50f5f5f3ec0f9fdf5f55ad81a5ec17097a2ebc4125f3dce26a857efa99a248cfec3c4f4114442bdbaa6f1421fea9c0fdb595fa7c64ee706f1adef2b07407d3d2447fabec114abd67789c1d862660b18f6507eb7777b77779cefee6ea0cea2258d963237c87166b1b3ab9815c667762079c399f5e70f817c7fd83dbb4b28864f5f3614f1faaa818e0f55cf8ee4f555c3ebd52b72c0177e44fc7d1db39885793b874dfd9efb9c06e0dbcfbac366e6eb1575e6eab1e0fa8c5b8f09e61918628e39c71e93ef31572fb22a3a17bb9efb5c77d9c47c3d6555c8f3eb1ca75e0686aecfbc672831ef9e61fccc8998eb191ee0db39f53430a4e135ef19eaebdbb1192aab68982186282e7c7ba8ac0a9d7c7b88b9b22a5ce1db891a9bfada44ea3973e6c879cc7b86297c746ebb9ed1b1a9af5e0bda773691b4c7c070bf41a49d3367ce9c0f5ddf1ea764d576bf2ef44a822bdba8855321513e7dd540e6658a59fa5a92f3ea1fb3f45583985707c2f3ea0aa43ab4719c324bf739546416bb3ab7dc76aa3a0d48030cabf43b7a3b3b7b450ceaea34200d31acd25fefc9f17aebea45307258a7ccd20fe5ab73ddf1dc676f7d3dc9aa76f55650af889d88f70d4e9fd9971974ed32e1d6773ae07b2db9f9303ebb5c97527b4879157ebf1e7e1f99c5befb426f2fb713f17677f77af732bb7c067f9ab047e765b2ce60c8dee0916f189c0285aeac8fd3007ceaedea15a9afa78486a5a5a51a5e416555289f9de32e328b59abac6227e29e9155ec277cd153ccd2ff98a5ac0ae3abb3f330ab9d3dc5ac7d76f58ada899888e3c432c25f71c95f97bfe04f93f5e8eb114cda6f50bda205c3fd7520068ffc950457ef82c1e92b50f83d3b279915fe40d2c4b71e76cf1eb64756a9733d8d7cebedeaeab5b07ed4a06495baee8cdfc57924edb1ec78a8ed76f2bf1e4be671dda926590fdb23abd425b3c2fd7675aea76495bad0d7aeaede1265957a839255ea44cc75b7b3dfd3245676e99154782cd9afb3779ac4caba792cddafb39f3489955d7a24157e9d9dbd16e2d2d2afafc7e45bfaf58e59faecea31d9a55f27622226622226629dca2a750562f0e87b25c1c5cb6d70cadcba322b545f5059a5aeeedaede1f4b504e655e354c89c6f5d5d4d486c601498c186e9081c488a40b2c391378ee0e0480d8ed8e0c80d8cd071c48c911e8c7431c2c311208e0461c40746e41809736404467620648791101cc9e1480d477438220223688c7461840b233f3072c6c8cb480b8cd8200404a12008dd61e40b23751891c1912d465060e48bd00f8edc70e48b101d467c305283501d423b109a23e403234b477e30c202211e1c09c1912e47ae38a202235c8ce4c0481c478e10aa81501c4680385285111a1c5981113746e6108a239403211c18d1c1081c21398cfc2004c7480e46dc30f286900d8c6c11ba81900d45ee284283226c1471a3889b226d1439a3081a45d4289282225f8a74211287480e88e880881c44cc20c286080c88a821a206113444e0106943e40d913444c020f2022237208203225f108983881b426c40a40d226c085943c80d847c41a40c223608f212e4435008827a08ea12c443d00e413a04710922415009827008da12a425e886201b826a085a0a7222a88920140489090a13c444d0098296085222c80441490481094222e8882023827210e203217282a2088222e88720110409113482a02082be0401115484903884c411a203217208c981101c086143481b42dc088981103884a440881442a01072841012088922c81d429682e420081a41ca040913a48a20540489220898202208b26555c52acd4a8c95192b305659ac56b04263b5c52a8dd59a95991517abd7ca8ad599159ad5182b16acc258b560e5c5ea8a5519ab2f5664acd4acb458a96085c5ca05ab33565dac5ea00a820a8e0a072a355460a85ea0fa428546e70b1e3a74c0f00953020c544ea89ec860e035b103c5e422080be6abc7120094f09e30b1b127f0614136050961f4f4e0f35275c10307e645c78c1e52f420d32f008051a9405545096cf82c71048712da28c14d09681c59a284337a7040e9a08346e6829e1d7a04d123c406c713a3cd8084314850e39960c3c1dc82fbb275092a33b5983910f20aa205a8092031404c00a9a19346fbc25387cf09bc1c3a1d743de878e0e1d001c10b81d783d7c5e3e2ede0f1e0e9d0f9a00b4247876703a7026fa9db815783b7a5fb8177432787bb8233c359c155d1d5d1d9d1cdd1dde169e1b67038705ac0323233d38a2033e01ebc2ac020368d4563cfd8356bc696b164ac9a1d63c5d83036cd82b12fd82f16cdba605bb02cd833ebc576b15c2c13530e064416c68811426ae019010024322c7aba040182bbd851c14e1642da786648686304096f05aa392a1b309c1d2d7656b0c9e1398ef0e0a960c70a6f0b2f8aac4c9684d01b2141188913c405478ce025361fa8e8109213248715104e5c8e80e9c238e2c5880e8e3061648b150d32307a82d8c162e78a8f8c57c64b81978511268cc0b14345f682d30e8ce020bb62a78a9d2d3c29ba34426f1c490225879132de144794f8a2e0a238e1a0e2819119ace01cb1e20816a83819133e26f821c10f112916a4ce089981901dfc08c1ade06362e3418f37d38ed31613d2f008d14306938e12bcfcc081298b9d1787829e1fb61e8ae0a0880cba86c6a17d682fd918227090500600ae30c9d911a4dea8c4084183c81b4170f081e313858f143e647ca0f019e3f3848f133e64f84ce163264818d30c0096f096e01464546429e8f1c1a7862d889e2188c4800b4272215f3d6340327a18d15bf0dc81d281478416840f0a4a4883e7071a103c76642cc8a4c8d0642bc85490b5203341f645f6447682ac89ac8a0c051998ec4c0645903132323c6c645cf098e121c1111218b94116c58e191e2f2b2ab22d8408414406de153c5c322576c864af9d31ab323b53703cc8b408d2829d1e7cc4709c931c9d179c78d0a9d1a161e2814f121d9b530f74c6c8d8e888a16a42270c531d2a147469f828e143c4a90e1d34591a991a3a64645d6459784c14b901096ba066e051e16981e202f502fa01900f4e40409d41b100e505aa05a82e5068502e00da0284039016a01c806e00d201884b096c4c7494a0c6c90e95173e369cb038ade0a483aa072488c13a109a438816ab37de981927480d846ca1bdf1923852020f0ad51a9e13ec8577c4ac83a7cb4f1145a628224511170899a28b41e644e645102e1e98d50d74cae8a1a5c70d292a7ad8d0a306efc509c18dc0481229205039e08058b531f5c08709930f56080f8b531b8fcc690df00745e0405282d4174e4ccf084c2dd8896227053fc4e80941cf971d27563310c2858f1612ccd8e2e811410f367ea4d979629b83db61ceb123c526078f0f9e123d3ca8e09859ec3461421c2a22b8267a8ae0ca6c3fe80102f5e694c34e192d8ecc8a1e2f1e0a76c40819c14e0888b8c9c4eca0c00be301d123839f1f7aae008a235bc2b342080e551bef04425890ad31af4895e9990300137845686f702240c5c185608709ee07141c0f094d0600482233a22708281cec28c15234174c66e7045e0976c2ec2cd173874702cd0d8e072164909046080d26165c0f73059c17ce870c09ae4b7644bb69311a8c4ed36a3a063d46b7e930ba0c0e419381f5d0030a1260400219258031ef006d9873c0306013a0986983550fe60d78d6e06983a78d4a07d519aa1dbc373c39bc372a2b542b505d013a0196019ff029c2870426f8c0043b4cf8c1ce0e3b40ecf0d043839e1bf4d4e0079c1f3df811870967402cc02dc02c5274a47a90aaa3871c3deae891839d3676ded87103002000200400be00a00b00b800005efc00c24f0e3fec20810d12ce009ae3a4c5c98bd316a0112013e011a6324c6b98cce8a1a3a7073d759cda38b97172f3b3c34f087e7820410d12d20072c347071f2f3e3bfc68e3870d7eb85105a1fa41e5c37463d260be7102e30483539a8fcb37824f0712d89080460938a482e0c386d41dde181e0c3c353c6578bae049010f173c5490100312d694c085096b4c68c384337a94d123063dccf88cf84ef01df1638c2048fc50e3fde0a4c5030217478f16f420a3870b7ec849c191922365839516215f908884d139b496b681efe82efd43f7c03c082206d7c1396039980e8e836fc038080203d6825550c40c6601bf780bb682cdf00a8478612c848840880f1c0537c1500469c3638604311141e2f017deb26418877d838108f20308b8c05a168eadc1c6b136d81b6c9b8d62af582b3606bbc66ac1432c0c76069bc5a620080d82ac210408213f14a9c12ab1492c0996081a3880650349531ec08005e2c9e2d0800c60004a0528b080cd61bf2820012c14a0090298aca0e40006683d01b6881de2481124442c20d170fbba5f0e366ea32b522e0065062946314081c26dc350c5080a14534a4c0c4919a86244258c1a1961008a0c49509262c8c014230eac524ce8152a53a42419c100c5c8080326a44ed0012b461b40823265864faa62a4810e4871e2a925d4e8852a3254812205ca9419dee8852a322820a9ca066478e1850f21a584ca80812a53a0bc0045060c546142858a0c06487ac1880a15190c20246502bd623425698a15196248da80141ea924b48a0c5392a458d9000c509264a07205ca94195e8a950d7c4f0acc4c21a1573a607404e50a1423186478c1080628573a60d4a48a0c53ae5c316ac008a92394035366782b4647566480400b3365844231da80d10616f00406a30c6440ca1321a922f4cad10b4631bc5002bd9254c5c88a510c55a6788a0455a660e0274584be30e54a95295692683082d2012a33cc20058a152318a84cd940ebe80519a8b480a486500d40919224050629494609d04909e1812954a44c499a52654a4f6a041b78c18806a39d54106a450603743d2ad0a31764a0720128464952609842450a9693ae11950b4849aab2010c40497ac1c8ca1418aa4c8102c5470aa38166861a6e5e5d9cbe96967e5ded23935fc82f7686d5e209c374b013866118a6a3a3b324685a37a731376bac45ad3955d534e6d658d3545531d65863e6a8ab69dabaacadab9aa6ed2eab165bb5f9d362eca8a176356c57eeae7e5ae49651350de3d1b4c81c63472939b2c69a2a6baa75b7c6cc59e46cebd632a99ab3dc628f941f2a6ad9d7dda68f5923b34932c6cd9a6adaad7dd25155152149eceea1a972b7d4346a92b93b5ad19423d65155d394594ad5e5189ba3b2362b6bd48d9ba32ac7b8296bda1c5935662cca8eacbcfc31f3e8983be64e5559a58a59b6ca5967ca59c6aa42cccbd9aa6a26b499a65455c86452d6dedded9499236f0fe6456236a9b6096c42511137c6135bf2a67533733f3c5ad3e5d555c65052ca4d468db1bb193aaa6251861fcc55d1989963c6b447f308b1a3ce083146c9922577f7019a63c736b5a6a94addde3a55a199a376b7c65a6b37a769918bdcdd52cac8ddcd712a005055a5522da1044de3c8adb2d3183566edb47f6cdaccdbbbaab11663e4b88e9ed6ac2a35ed566deea86deb761d0f137898c07b003ea911cd34d6ccc46a0003fcfc682036017a14d5cc3f3f54589b5b76ec6eedd453dd5dd50d929a16c4cc1ee3b6b1aa9431a25011858a91b7ad654b94dc369494dd51ca8d9148e6a8aa356b1c9b63abaaaac6833955e511398ebbf5a8e3eeba0e00dd0fee1f0008b929dc8f8e8ec91443d7dcac65805bdcaaba6dd4b45b1b0f8f129689626bdd9d9a63568e5131c53432a6ca588c3136777bda6e718b513b462040a2aab62a00002025c7ed4a2965b732ab64de5d8db9ebee661df2a34891c7c0ce975004e3b8eee68e9b94b1b9183b721c0fd991336a8eeb665e5edd886266140cfdadb47ea163f3ea72b3a6a96a5ad7235ce00414747999638c52a208dadd988c586c19b1c8404040403d3ddd246824ec6a9ad6aadaaae974cb1b370a0fe6dd5155ee239a42619fde65d54d43d02ca0aaf546a4ccad2a706f3ccd4c8488169bc8a61a7755b24af94554081da3e4d818b3ecd8c3dd5263d56e6e4ddb5423f36815d8ba187b9929d0a3077333737794719784659e40c7dd8fa8ad75b1952590521f55652d007701949909c0cc3d9839859d66d614244bc9cedc8377763ceec43ef5cea94fdda7e66e70979757ca185beb1377f7e9d43b27b03bc6d3893982cd3b276e6e8d99fbd4dc3b11d839316b2666654dd35ae365e6e6c8cacb9a6a182f6bcc5a7700748688c0271d9d010ca039b6ce082e24691d0188008111507800cb01434322b062f729003cb8219c6acc31aaaeae66d2344d5935e56665ee27cd5b9fa04565d55656d5e6a8abaaaacc51233773d4829855d36254c554314d99394aa99a46ad1bebb89ab658dcd8b1354d6e8c2453660e4163e56656d5348d9ba38ca84fc6ad476659'... 2538366 more characters,
  '5f08dcde934c658227ee1dfafcd6e16903050108dc4d79aad5a9d01a359995838830a80733a0bff7e4eb087bfc621ef1873fec49be4f21c56d926b91f020b5071f14935cb93f001f1127c53d3eac6eed23ffea64',
  '5f0a42f33323cb5ced3b44dd825fda9fcc804545454545454545454545454545454545454545454545454545454545454545',
  '5f0e0621c4869aa60c02be9adcc98a0d1d050108dc4d79aad5a9d01a359995838830a80733a0bff7e4eb087bfc621ef1873fec49be4f21c56d926b91f020b5071f14935cb93f001f1127c53d3eac6eed23ffea64',
  '764704b568d21667356a5a050c118746b4def25cfda6ef3a00000000804545454545454545454545454545454545454545454545454545454545454545',
  '7d0bce545fb382c34570e5dfbf338f5e4e7b9012096b41c4eb3aaf947f6ea429080000',
  '7e0f0c53fa332d4d9712c66fd92efcb64e7b9012096b41c4eb3aaf947f6ea429080000',
  '7e1467a096bcd71a5b6a0c8155e208104e7b9012096b41c4eb3aaf947f6ea429080000',
  '7e3237373ffdfeb1cab4222e3b520d6b4e7b9012096b41c4eb3aaf947f6ea429080200',
  '7e323df7cc47150b3930e2666b0aa3134e7b9012096b41c4eb3aaf947f6ea429080200',
  ...
]

1st one contains our runtime code wasm:

🗜  Compressed:                 Yes, 78.78%
✨ Reserved meta:                OK - [6D, 65, 74, 61]
🎁 Metadata version:            V14
🔥 Core version:                hydradx-178 (hydradx-0.tx1.au1)
🗳️  system.setCode hash:               0x64c439e579c3bfff9f4ebb8be01ca8a33f5c6f565c42531b46011974a9f79c93
🗳️  authorizeUpgrade hash:     0xa059f2c663f68b95f2e72ad34e2ff34569706ebee1c6fe74c519e847eb5dab3a
#️⃣  Blake2-256 hash:           0x32dc435cbda2592facebf36852feb2ec411f7b77cd33a9ec8ba109cb579a7cb9
📦 IPFS:                        https://www.ipfs.io/ipfs/QmU5Lw394PxSziP6vMNH7B2UdFn4XZfXVKQrG2hXG4NELk

why do we attach it in storage proof?

[cheme]

Well you are probably using trie_version 0 (maybe state_version), with version 1 the value is not attach to the node and only include if accessed by the runtime. WARNING: switching requires a migration (or warpsync will be broken).

So if using state_version 0, maybe you query an entry close to key ":code" eg ":codex" that would include the node (and its value) at ":code" into your proof. so any query to key starting with ":code", and some insert or removal of key close to :code (which may result in changing the node prefix at :code and thus touching the node), would include the wasm in the proof.

1st one appears to contain wasm file

yes wasm is in the top trie at key ':code' (utf8 values). would make sense to trace all runtime access during block processing and try to find what key query can touch ':code' trie node (I don't remember how it can be done but maybe with try-runtime and some logging (cannot check right now), if missing traces can be added on sp-io storage function or sp-state-machine trie accesses directly.

[enthusiastmartin]

With state_version set to 1.

2023-09-18 16:40:12 [Parachain] PoV size { header: 0.1787109375kb, extrinsics: 2.865234375kb, storage_proof: 4.306640625kb } 2023-09-18 16:40:12 [Parachain] Compressed PoV size: 6.330078125kb

So I guess, we should consider migrating.

Would have some details how to do that?

[cheme]

There is a link to the md guide and lot of link to different progress on relay chain https://github.com/paritytech/devops/issues/1508#issuecomment-1271565180 . Note that the migration process for parachain may seems more complicated than for a relay (progress by adding extrinsic in each block manually), to avoid going over the block size (with the automatic process used in relay chain there is always a risk that the content of the chain will include a very big value on top of an already big proof). But from my point of view a parachain could still audit cautiously its content and assert such scenario will not happen (even possibly run some value migration ahead and skip them afterward in the automatic migration process: but this is not currently coded in the state-migration pallet). Generally, the migration process is not something complicated, it just requires that every key value in the runtime get written again once (if you look at the state migration automatic process, we just store a progress key and advance a few value at the start of every blocks).

[mrq1911]

There is a link to the md guide and lot of link to different progress on relay chain https://github.com/paritytech/devops/issues/1508#issuecomment-1271565180 .

sry, this link is broken for me, is it in private repo?

[cheme]

:facepalm: yes it is a private one, the link to the guide was https://hackmd.io/JagpUd8tTjuKf9HQtpvHIQ (a post refering to it https://forum.polkadot.network/t/state-trie-migration/852).

[jak-pan]

Solved by #799 and running migration on wasm