AcalaNetwork / chopsticks

Create parallel reality of your Substrate network.
Apache License 2.0
133 stars 80 forks source link

Block production stuck during multi block migration #760

Closed tommycodebox closed 4 months ago

tommycodebox commented 4 months ago

Chopsticks fork of parachain gets stuck after applying migration and emitting parachainSystem.ValidationFunctionApplied event

image

chopsticks produces the next block after migration (#...500) but it doesnt show in polkadotjs and after that it doesnt produce any more blocks with dev_newBlock RPC

image

Is the feature available or planned? now since substrate natively supports multi block migrations, it would be great to be able to test them against production chain data

xlc commented 4 months ago

can you provide the wasm and config file for us to reproduce the issue can you also set LOG_LEVEL=trace and see if it is still loading state if the migration is accessing a lot of state, it can take very long time depends on the latency to the node for example, if the migration is accessing 1000 storages, and there is 100ms latency to the RPC node, that's 100s to download the state

xlc commented 4 months ago

if it is indeed taking too long to download the state, then we will need https://github.com/AcalaNetwork/chopsticks/issues/759 to speed it up

tommycodebox commented 4 months ago

oh, ok, now with full logs enabled, i can see it making a lot of storage calls, seems i will have to wait quite a bit, since we have a lot of storage items on the chain im cloning

...
[10:38:33.983] TRACE (layer): RemoteStorageLayer getKeysPaged
[10:38:33.984] TRACE (layer): RemoteStorageLayer get
[10:38:34.171] TRACE (layer): RemoteStorageLayer getKeysPaged
...

going to see if waiting long enough will succeed and update here

tommycodebox commented 4 months ago

closing issue as its working as expected, its just the amount of data that makes it take a very long time