Can't fork main net using fork-off-substrate tool

[sameh-farouk]

Describe the bug

Can't fork main net network using fork-off-substarte tool.

Logs

Substrate RPC : Response too big, Exceeded max limit of 15728640

Additional context

The tool works for dev, qa and test net. but fail on main net.

[sameh-farouk]

Update: The fork-off-substrate tool uses the unsafe state_getPairs method, which puts a heavy load on the node as it has to iterate through the trie to retrieve all matching keys. This is particularly problematic for the main net state, as it exceeds the default response limits, resulting in failed fork attempts.

I Requested changes to main net private RPC node to increase the max response limit. https://github.com/threefoldtech/tf_operations/issues/2655

[sameh-farouk]

Update: Got a new error

Fetching current state of the live chain. Please wait, it can take a while depending on the size of your chain.
 ██████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 14% | ETA: 48s | 38/256node:internal/deps/undici/undici:12618
    Error.captureStackTrace(err, this);
          ^

TypeError: fetch failed
    at node:internal/deps/undici/undici:12618:11
    at async HttpProvider._send2 (/home/sameh/Projects/tfchain/tools/fork-off-substrate/node_modules/@polkadot/rpc-provider/http/index.cjs:186:20)
    at async fetchChunks (/home/sameh/Projects/tfchain/tools/fork-off-substrate/index.js:170:19)
    at async fetchChunks (/home/sameh/Projects/tfchain/tools/fork-off-substrate/index.js:188:7)
    at async main (/home/sameh/Projects/tfchain/tools/fork-off-substrate/index.js:94:5) {
  cause: HeadersTimeoutError: Headers Timeout Error
      at Timeout.onParserTimeout [as callback] (node:internal/deps/undici/undici:9117:32)
      at Timeout.onTimeout [as _onTimeout] (node:internal/deps/undici/undici:7148:17)
      at listOnTimeout (node:internal/timers:573:17)
      at process.processTimers (node:internal/timers:514:7) {
    code: 'UND_ERR_HEADERS_TIMEOUT'
  }
}

Will try to increase the chunks count using the FORK_CHUNKS_LEVEL environment variable. I tried this before adjusting the RPC node response limit. Hopefully, it would work now.

[sameh-farouk]

Update: The fork process is currently at 90% and still ongoing. Previously, it used to fail early, so there is a high chance to complete this time.

[sameh-farouk]

Update: Issue Fixed. MainNet state is forkable now after adjusting the RPC node max response limit and increases FORK_CHUNKS_LEVEL to 2

I opened a PR to update the fork-off-substrate development documentations.

sameh@sameh-kubuntu:~/Projects/tfchain/tools/fork-off-substrate$ FORK_CHUNKS_LEVEL=2 HTTP_RPC_ENDPOINT=http://10.10.0.56:9933 ALICE="true" npm start

> fork-off-substrate@2.8.1 start
> node index.js

We are intentionally using the HTTP endpoint. If you see any warnings about that, please ignore them.
Custom Schema missing, using default schema.
2024-07-14 14:44:54        API/INIT: Api will be available in a limited mode since the provider does not support subscriptions
2024-07-14 14:44:56        API/INIT: RPC methods not decorated: transaction_unstable_submitAndWatch, transaction_unstable_unwatch
Fetching current state of the live chain. Please wait, it can take a while depending on the size of your chain.
 ████████████████████████████████████████ 100% | ETA: 0s | 65536/65536
2024-07-14 19:02:22 Building chain spec    
2024-07-14 19:02:22 Building chain spec    
Forked genesis generated successfully. Find it at ./data/fork.json

[sameh-farouk]

Verified!