search

threefoldtech / tfchain

Threefold Chain.
Apache License 2.0
15 stars 11 forks source link

forked tfchain with fork-off-substarte not finalizing blocks #839

Closed sameh-farouk closed 10 months ago

sameh-farouk commented 1 year ago

I tried to run fork-off-substrate, but seems some instructions are missing.

I documented here step by step what I did. This is part of a draft PR here to revise the development docs.

Can you advise what is missing or going wrong @DylanVerstraete ?

01:51:25 sameh@sameh-inspiron-3576 ~/projects/tfchain/tools/fork-off-substrate
❯ HTTP_RPC_ENDPOINT=https://tfchain.dev.grid.tf ALICE="true" npm start

> fork-off-substrate@1.0.0 start
> node index.js

We are intentionally using the HTTP endpoint. If you see any warnings about that, please ignore them.
Custom Schema missing, using default schema.
2023-08-02 12:51:51        API/INIT: Api will be available in a limited mode since the provider does not support subscriptions
2023-08-02 12:51:52        API/INIT: RPC methods not decorated: transaction_unstable_submitAndWatch, transaction_unstable_unwatch
Reusing cached storage. Delete ./data/storage.json and rerun the script if you want to fetch latest storage
2023-08-02 13:51:53 Building chain spec    
2023-08-02 13:51:53 Building chain spec    
Forked genesis generated successfully. Find it at ./data/fork.json

01:51:54 sameh@sameh-inspiron-3576 ~/projects/tfchain/tools/fork-off-substrate
❯ ./data/binary --chain ./data/fork.json --alice
2023-08-02 13:52:23 Substrate Node    
2023-08-02 13:52:23 ✌️  version 2.5.0-rc7-78f9b5195ac    
2023-08-02 13:52:23 ❤️  by dylan@threefold.tech, 2017-2023    
2023-08-02 13:52:23 📋 Chain specification: TF Chain Live-fork    
2023-08-02 13:52:23 🏷  Node name: Alice    
2023-08-02 13:52:23 👤 Role: AUTHORITY    
2023-08-02 13:52:23 💾 Database: RocksDb at /home/sameh/.local/share/binary/chains/tfchain_live-fork/db/full    
2023-08-02 13:52:23 ⛓  Native runtime: substrate-threefold-145 (substrate-threefold-1.tx2.au1)    
2023-08-02 13:52:35 Using default protocol ID "sup" because none is configured in the chain specs    
2023-08-02 13:52:35 🏷  Local node identity is: 12D3KooWCPcD2a3MprMKsVnyHnujLCctbnQg7er9pEbd4gEG5FtV    
2023-08-02 13:52:35 💻 Operating system: linux    
2023-08-02 13:52:35 💻 CPU architecture: x86_64    
2023-08-02 13:52:35 💻 Target environment: gnu    
2023-08-02 13:52:35 💻 CPU: Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz    
2023-08-02 13:52:35 💻 CPU cores: 4    
2023-08-02 13:52:35 💻 Memory: 15736MB    
2023-08-02 13:52:35 💻 Kernel: 5.19.5-051905-generic    
2023-08-02 13:52:35 💻 Linux distribution: Ubuntu 22.04.2 LTS    
2023-08-02 13:52:35 💻 Virtual machine: no    
2023-08-02 13:52:35 📦 Highest known block at #0    
2023-08-02 13:52:35 〽️ Prometheus exporter started at 127.0.0.1:9615    
2023-08-02 13:52:35 Running JSON-RPC HTTP server: addr=127.0.0.1:9933, allowed origins=["http://localhost:*", "http://127.0.0.1:*", "https://localhost:*", "https://127.0.0.1:*", "https://polkadot.js.org"]    
2023-08-02 13:52:35 Running JSON-RPC WS server: addr=127.0.0.1:9944, allowed origins=["http://localhost:*", "http://127.0.0.1:*", "https://localhost:*", "https://127.0.0.1:*", "https://polkadot.js.org"]    
2023-08-02 13:52:35 💔 The bootnode you want to connect to at `/ip4/127.0.0.1/tcp/30333/p2p/12D3KooWEfHSfdVy738QyNg5RJE47rGdgNq8pt6qVxHVnQyFsCx1` provided a different peer ID `12D3KooWCPcD2a3MprMKsVnyHnujLCctbnQg7er9pEbd4gEG5FtV` than the one you expect `12D3KooWEfHSfdVy738QyNg5RJE47rGdgNq8pt6qVxHVnQyFsCx1`.    
2023-08-02 13:52:35 💔 The bootnode you want to connect to at `/ip4/127.0.0.1/tcp/30333/p2p/12D3KooWEfHSfdVy738QyNg5RJE47rGdgNq8pt6qVxHVnQyFsCx1` provided a different peer ID `12D3KooWCPcD2a3MprMKsVnyHnujLCctbnQg7er9pEbd4gEG5FtV` than the one you expect `12D3KooWEfHSfdVy738QyNg5RJE47rGdgNq8pt6qVxHVnQyFsCx1`.    
2023-08-02 13:52:40 💤 Idle (0 peers), best: #0 (0xb594…a651), finalized #0 (0xb594…a651), ⬇ 0.2kiB/s ⬆ 0.2kiB/s    
2023-08-02 13:52:45 💤 Idle (0 peers), best: #0 (0xb594…a651), finalized #0 (0xb594…a651), ⬇ 0 ⬆ 0    
2023-08-02 13:52:50 💤 Idle (0 peers), best: #0 (0xb594…a651), finalized #0 (0xb594…a651), ⬇ 0 ⬆ 0    
2023-08-02 13:52:55 💤 Idle (0 peers), best: #0 (0xb594…a651), finalized #0 (0xb594…a651), ⬇ 0 ⬆ 0    
2023-08-02 13:53:00 💤 Idle (0 peers), best: #0 (0xb594…a651), finalized #0 (0xb594…a651), ⬇ 0 ⬆ 0    
2023-08-02 13:53:05 💤 Idle (0 peers), best: #0 (0xb594…a651), finalized #0 (0xb594…a651), ⬇ 0 ⬆ 0    
2023-08-02 13:53:10 💤 Idle (0 peers), best: #0 (0xb594…a651), finalized #0 (0xb594…a651), ⬇ 0 ⬆ 0    
2023-08-02 13:53:15 💤 Idle (0 peers), best: #0 (0xb594…a651), finalized #0 (0xb594…a651), ⬇ 0 ⬆ 0    
2023-08-02 13:53:20 💤 Idle (0 peers), best: #0 (0xb594…a651), finalized #0 (0xb594…a651), ⬇ 0 ⬆ 0    
2023-08-02 13:53:25 💤 Idle (0 peers), best: #0 (0xb594…a651), finalized #0 (0xb594…a651), ⬇ 0 ⬆ 0    
2023-08-02 13:53:30 💤 Idle (0 peers), best: #0 (0xb594…a651), finalized #0 (0xb594…a651), ⬇ 0 ⬆ 0    
2023-08-02 13:53:35 💤 Idle (0 peers), best: #0 (0xb594…a651), finalized #0 (0xb594…a651), ⬇ 0 ⬆ 0    
2023-08-02 13:53:40 💤 Idle (0 peers), best: #0 (0xb594…a651), finalized #0 (0xb594…a651), ⬇ 0 ⬆ 0    
2023-08-02 13:53:45 💤 Idle (0 peers), best: #0 (0xb594…a651), finalized #0 (0xb594…a651), ⬇ 0 ⬆ 0    
2023-08-02 13:53:50 💤 Idle (0 peers), best: #0 (0xb594…a651), finalized #0 (0xb594…a651), ⬇ 0 ⬆ 0    
2023-08-02 13:53:55 💤 Idle (0 peers), best: #0 (0xb594…a651), finalized #0 (0xb594…a651), ⬇ 0 ⬆ 0    
2023-08-02 13:54:00 💤 Idle (0 peers), best: #0 (0xb594…a651), finalized #0 (0xb594…a651), ⬇ 0 ⬆ 0    
2023-08-02 13:54:05 💤 Idle (0 peers), best: #0 (0xb594…a651), finalized #0 (0xb594…a651), ⬇ 0 ⬆ 0    
^C
DylanVerstraete commented 1 year ago

Are you sure you aren't used cached data? See 

Reusing cached storage. Delete ./data/storage.json and rerun the script if you want to fetch latest storage

In the output of the script.

DylanVerstraete commented 1 year ago

Which version of tfchain did you compile and use in the data dir?

sameh-farouk commented 1 year ago

Which version of tfchain did you compile and use in the data dir?

latest development

are you sure you aren't used cached data?

I Cleared it and now see an error complaining about unsafe rpc call, so i guess the only option is to run a node locally with unsafe rpc flag, right?

Now, I started a local node with dev net spec as mentioned here

By default it will use full sync mode, If I left it syncing, and run fork-off-substrate to generate the fork.json and used it to start the forkd chain, still not producing blocks. do I need to let the node fully synced first before trying to run the tool against it.

If yes, is there a shortcut for saving the time? also it could required a lot of storage that not available on my machine atm.

I tried to start the node with --sync fast and --sync warp but both looks like not progressing (I can open another issue for that as well)

sameh-farouk commented 1 year ago

with --sync warp I see no progress at all.

02:43:46 sameh@sameh-inspiron-3576 ~/projects/tfchain/tools/fork-off-substrate
❯ ../../substrate-node/target/debug/tfchain --chain ../../substrate-node/chainspecs/dev/chainSpecRaw.json --ws-external --rpc-methods Unsafe --sync warp
2023-08-02 14:43:54 Substrate Node    
2023-08-02 14:43:54 ✌️  version 2.5.0-rc7-78f9b5195ac    
2023-08-02 14:43:54 ❤️  by dylan@threefold.tech, 2017-2023    
2023-08-02 14:43:54 📋 Chain specification: TF Chain Devnet    
2023-08-02 14:43:54 🏷  Node name: keen-truck-8090    
2023-08-02 14:43:54 👤 Role: FULL    
2023-08-02 14:43:54 💾 Database: RocksDb at /home/sameh/.local/share/tfchain/chains/tfchain_devnet/db/full    
2023-08-02 14:43:54 ⛓  Native runtime: substrate-threefold-145 (substrate-threefold-1.tx2.au1)    
2023-08-02 14:43:58 🔨 Initializing Genesis block/state (state: 0x6a2e…4030, header-hash: 0xbb7f…66be)    
2023-08-02 14:43:58 👴 Loading GRANDPA authority set from genesis on what appears to be first startup.    
2023-08-02 14:44:02 Using default protocol ID "sup" because none is configured in the chain specs    
2023-08-02 14:44:02 🏷  Local node identity is: 12D3KooWAiR6YTUUGpqVWuU86W2jboviFVUQkdVsGiAXAnKeiAUF    
2023-08-02 14:44:02 💻 Operating system: linux    
2023-08-02 14:44:02 💻 CPU architecture: x86_64    
2023-08-02 14:44:02 💻 Target environment: gnu    
2023-08-02 14:44:02 💻 CPU: Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz    
2023-08-02 14:44:02 💻 CPU cores: 4    
2023-08-02 14:44:02 💻 Memory: 15736MB    
2023-08-02 14:44:02 💻 Kernel: 5.19.5-051905-generic    
2023-08-02 14:44:02 💻 Linux distribution: Ubuntu 22.04.2 LTS    
2023-08-02 14:44:02 💻 Virtual machine: no    
2023-08-02 14:44:02 📦 Highest known block at #0    
2023-08-02 14:44:02 〽️ Prometheus exporter started at 127.0.0.1:9615    
2023-08-02 14:44:02 Running JSON-RPC HTTP server: addr=127.0.0.1:9933, allowed origins=["http://localhost:*", "http://127.0.0.1:*", "https://localhost:*", "https://127.0.0.1:*", "https://polkadot.js.org"]    
2023-08-02 14:44:02 Running JSON-RPC WS server: addr=0.0.0.0:9944, allowed origins=["http://localhost:*", "http://127.0.0.1:*", "https://localhost:*", "https://127.0.0.1:*", "https://polkadot.js.org"]    
2023-08-02 14:44:03 🔍 Discovered new external address for our node: /ip4/45.247.28.120/tcp/30333/ws/p2p/12D3KooWAiR6YTUUGpqVWuU86W2jboviFVUQkdVsGiAXAnKeiAUF    
2023-08-02 14:44:07 ⏩ Warping, Downloading finality proofs, 0.00 Mib (6 peers), best: #0 (0xbb7f…66be), finalized #0 (0xbb7f…66be), ⬇ 496.6kiB/s ⬆ 19.2kiB/s    
2023-08-02 14:44:21 ⏩ Warping, Downloading finality proofs, 0.00 Mib (7 peers), best: #0 (0xbb7f…66be), finalized #0 (0xbb7f…66be), ⬇ 269.2kiB/s ⬆ 1.2kiB/s    
2023-08-02 14:44:26 ⏩ Warping, Downloading finality proofs, 0.00 Mib (6 peers), best: #0 (0xbb7f…66be), finalized #0 (0xbb7f…66be), ⬇ 784.7kiB/s ⬆ 11.8kiB/s    
2023-08-02 14:44:38 ⏩ Warping, Downloading finality proofs, 0.00 Mib (6 peers), best: #0 (0xbb7f…66be), finalized #0 (0xbb7f…66be), ⬇ 186.6kiB/s ⬆ 0.1kiB/s    
2023-08-02 14:44:43 ⏩ Warping, Downloading finality proofs, 0.00 Mib (7 peers), best: #0 (0xbb7f…66be), finalized #0 (0xbb7f…66be), ⬇ 976.0kiB/s ⬆ 4.4kiB/s
2023-08-02 14:44:55 ⏩ Warping, Downloading finality proofs, 0.00 Mib (7 peers), best: #0 (0xbb7f…66be), finalized #0 (0xbb7f…66be), ⬇ 102.9kiB/s ⬆ 0.5kiB/s    
2023-08-02 14:45:00 ⏩ Warping, Downloading finality proofs, 0.00 Mib (6 peers), best: #0 (0xbb7f…66be), finalized #0 (0xbb7f…66be), ⬇ 928.4kiB/s ⬆ 4.2kiB/s 
2023-08-02 14:45:12 ⏩ Warping, Downloading finality proofs, 0.00 Mib (6 peers), best: #0 (0xbb7f…66be), finalized #0 (0xbb7f…66be), ⬇ 114.3kiB/s ⬆ 1.2kiB/s    
2023-08-02 14:45:17 ⏩ Warping, Downloading finality proofs, 0.00 Mib (6 peers), best: #0 (0xbb7f…66be), finalized #0 (0xbb7f…66be), ⬇ 940.2kiB/s ⬆ 4.1kiB/s

with --sync fast seems progressing but gives a lot of errors

02:46:33 sameh@sameh-inspiron-3576 ~/projects/tfchain/tools/fork-off-substrate
❯ ../../substrate-node/target/debug/tfchain --chain ../../substrate-node/chainspecs/dev/chainSpecRaw.json --ws-external --rpc-methods Unsafe --sync fast
2023-08-02 14:46:41 Substrate Node    
2023-08-02 14:46:41 ✌️  version 2.5.0-rc7-78f9b5195ac    
2023-08-02 14:46:41 ❤️  by dylan@threefold.tech, 2017-2023    
2023-08-02 14:46:41 📋 Chain specification: TF Chain Devnet    
2023-08-02 14:46:41 🏷  Node name: obedient-door-6028    
2023-08-02 14:46:41 👤 Role: FULL    
2023-08-02 14:46:41 💾 Database: RocksDb at /home/sameh/.local/share/tfchain/chains/tfchain_devnet/db/full    
2023-08-02 14:46:41 ⛓  Native runtime: substrate-threefold-145 (substrate-threefold-1.tx2.au1)    
2023-08-02 14:46:45 🔨 Initializing Genesis block/state (state: 0x6a2e…4030, header-hash: 0xbb7f…66be)    
2023-08-02 14:46:45 👴 Loading GRANDPA authority set from genesis on what appears to be first startup.    
2023-08-02 14:46:49 Using default protocol ID "sup" because none is configured in the chain specs    
2023-08-02 14:46:49 🏷  Local node identity is: 12D3KooWAiR6YTUUGpqVWuU86W2jboviFVUQkdVsGiAXAnKeiAUF    
2023-08-02 14:46:49 💻 Operating system: linux    
2023-08-02 14:46:49 💻 CPU architecture: x86_64    
2023-08-02 14:46:49 💻 Target environment: gnu    
2023-08-02 14:46:49 💻 CPU: Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz    
2023-08-02 14:46:49 💻 CPU cores: 4    
2023-08-02 14:46:49 💻 Memory: 15736MB    
2023-08-02 14:46:49 💻 Kernel: 5.19.5-051905-generic    
2023-08-02 14:46:49 💻 Linux distribution: Ubuntu 22.04.2 LTS    
2023-08-02 14:46:49 💻 Virtual machine: no    
2023-08-02 14:46:49 📦 Highest known block at #0    
2023-08-02 14:46:49 〽️ Prometheus exporter started at 127.0.0.1:9615    
2023-08-02 14:46:49 Running JSON-RPC HTTP server: addr=127.0.0.1:9933, allowed origins=["http://localhost:*", "http://127.0.0.1:*", "https://localhost:*", "https://127.0.0.1:*", "https://polkadot.js.org"]    
2023-08-02 14:46:49 Running JSON-RPC WS server: addr=0.0.0.0:9944, allowed origins=["http://localhost:*", "http://127.0.0.1:*", "https://localhost:*", "https://127.0.0.1:*", "https://polkadot.js.org"]    
2023-08-02 14:46:50 🔍 Discovered new external address for our node: /ip4/45.247.28.120/tcp/30333/ws/p2p/12D3KooWAiR6YTUUGpqVWuU86W2jboviFVUQkdVsGiAXAnKeiAUF    
2023-08-02 14:46:50 Unable to pin block for finality notification. hash: 0xfc37…783a, Error: UnknownBlock: State already discarded for `0xfc37fe868195d09962a802f21419a480e77db99f650ad0471ebf9a7ef420783a`    
2023-08-02 14:46:51 Unable to pin block for finality notification. hash: 0x76bb…2003, Error: UnknownBlock: State already discarded for `0x76bb64dc4ab55a710a4fd193abdca626506ccb6330bf62815045972570c32003`    
2023-08-02 14:46:51 Unable to pin block for finality notification. hash: 0x83ee…ad8f, Error: UnknownBlock: State already discarded for `0x83ee784b30a1c0334813858b3c69092792107c455ffa18a6751f87b598cbad8f`    
2023-08-02 14:46:52 Unable to pin block for finality notification. hash: 0x2084…41c6, Error: UnknownBlock: State already discarded for `0x20849887d1f8506b59a3b2607be8501115ed3f797949791c1402dcf48f8941c6`    
2023-08-02 14:46:52 Unable to pin block for finality notification. hash: 0xb0c5…1cab, Error: UnknownBlock: State already discarded for `0xb0c5efe982f2d0d1aba12a5b7200ab8b2a66c49f932b37090e125da29ae31cab`    
2023-08-02 14:46:53 Unable to pin block for finality notification. hash: 0x72cf…d65f, Error: UnknownBlock: State already discarded for `0x72cf9a1095b1a44735df2d954248ded12f2982c15a0bd978d1cb739d7660d65f`    
2023-08-02 14:46:53 Unable to pin block for finality notification. hash: 0x7ae9…5c86, Error: UnknownBlock: State already discarded for `0x7ae929f145966caf90e828d80cd37ada134c558a250ae464b83f1a20266f5c86`    
2023-08-02 14:46:54 ⚙️  Syncing, target=#6949549 (4 peers), best: #0 (0xbb7f…66be), finalized #3584 (0x7ae9…5c86), ⬇ 215.2kiB/s ⬆ 7.2kiB/s    
2023-08-02 14:46:54 Unable to pin block for finality notification. hash: 0x23ba…8ae8, Error: UnknownBlock: State already discarded for `0x23ba3aec389af70d47f8a9e16888ec240c8c5404964f687e68dfd21931e18ae8`    
2023-08-02 14:46:55 Unable to pin block for finality notification. hash: 0x9d01…9736, Error: UnknownBlock: State already discarded for `0x9d0151ad9758b98c6d940be587636492b21f59f95858c7c08e81b3c5becc9736`    
2023-08-02 14:46:55 Unable to pin block for finality notification. hash: 0x766f…661f, Error: UnknownBlock: State already discarded for `0x766fa0142ed84b9198f1dce2865bb040db8ae90076b08c01d79ff47920fe661f`    
2023-08-02 14:46:56 Unable to pin block for finality notification. hash: 0xc521…f20f, Error: UnknownBlock: State already discarded for `0xc5219669df1ad8021d844cb1f5e119d6da5c8a00725e8a1047abd33b9b6cf20f`    
2023-08-02 14:46:56 Unable to pin block for finality notification. hash: 0x77b4…0c34, Error: UnknownBlock: State already discarded for `0x77b4bba2965822444d8718c04239b1ab04ccb497dde6c19edfc1adc3e9e50c34`    
2023-08-02 14:46:57 Unable to pin block for finality notification. hash: 0xa670…97ed, Error: UnknownBlock: State already discarded for `0xa6703c8dfe786758fa429a5b96f349a34a622ad136d8b9a56f9096a25bd897ed`    
2023-08-02 14:46:57 Unable to pin block for finality notification. hash: 0xa976…1ae3, Error: UnknownBlock: State already discarded for `0xa976eba08334df4a604e2826cd105cd27753ed7ee1b5e7163ba42104c14e1ae3`    
2023-08-02 14:46:58 Unable to pin block for finality notification. hash: 0x667e…52e0, Error: UnknownBlock: State already discarded for `0x667ef0d434c84b957d813457e172f2f2306322541c469622e240bebeafb952e0`    
2023-08-02 14:46:59 Unable to pin block for finality notification. hash: 0x9c83…c5fd, Error: UnknownBlock: State already discarded for `0x9c83d02e5a81be8804b275656b32db1a29a2ce415ab9af3e50f93dfeb206c5fd`    
2023-08-02 14:46:59 ⚙️  Syncing  0.0 bps, target=#6949549 (4 peers), best: #0 (0xbb7f…66be), finalized #8192 (0x9c83…c5fd), ⬇ 225.8kiB/s ⬆ 3.6kiB/s    
2023-08-02 14:46:59 Unable to pin block for finality notification. hash: 0x1676…2515, Error: UnknownBlock: State already discarded for `0x1676985f137a41fe67f7ed13ec7be99272ae461e1e14f96b6e0f48a218942515`    
2023-08-02 14:46:59 Unable to pin block for finality notification. hash: 0xac74…eec9, Error: UnknownBlock: State already discarded for `0xac740e8562118f9dbd86bbe89adf3c41bd24cae93a0254d30403df6faef3eec9`    
2023-08-02 14:47:00 Unable to pin block for finality notification. hash: 0xc71f…ec9b, Error: UnknownBlock: State already discarded for `0xc71f7831813a57b5f108ed00e8160bbe4ef49c937064dec7dc15033af1bcec9b`    
2023-08-02 14:47:00 Unable to pin block for finality notification. hash: 0x310a…6c4e, Error: UnknownBlock: State already discarded for `0x310a2dc5d42763b49afb85dde266e24018f5328d75ebf927a994aaebf1386c4e`    
2023-08-02 14:47:01 Unable to pin block for finality notification. hash: 0x16bb…9e66, Error: UnknownBlock: State already discarded for `0x16bb4ca98212e2c4fe8034a6edef7cf0200cc3ecc3cf5a9124cb51d2a1349e66`    
2023-08-02 14:47:02 Unable to pin block for finality notification. hash: 0x1d7a…484b, Error: UnknownBlock: State already discarded for `0x1d7abb3b4582690e322ea705b7b7dadcefebf40175ba12264cf77900e18f484b`    
2023-08-02 14:47:03 Unable to pin block for finality notification. hash: 0x228b…9830, Error: UnknownBlock: State already discarded for `0x228bc5a7746c6df1056cdce56178fd557f085050f5c3e8392bfe4220a5a39830`    
2023-08-02 14:47:04 ⚙️  Syncing  0.0 bps, target=#6949550 (4 peers), best: #0 (0xbb7f…66be), finalized #11776 (0x228b…9830), ⬇ 391.4kiB/s ⬆ 14.4kiB/s    
2023-08-02 14:47:05 Unable to pin block for finality notification. hash: 0x8a12…d2f9, Error: UnknownBlock: State already discarded for `0x8a12d3c20110ff7ee3b58e5f28a51b710650b273303eb765da5b7df4c4d1d2f9`    
2023-08-02 14:47:06 Unable to pin block for finality notification. hash: 0x4013…d8cc, Error: UnknownBlock: State already discarded for `0x4013f6ac4b9bedef49a11d623e2a9d1213932e075a71f6e34818978e388ad8cc`    
2023-08-02 14:47:06 Unable to pin block for finality notification. hash: 0x7479…2ad1, Error: UnknownBlock: State already discarded for `0x7479be95b142ca5fe2d8c9bd3cce479cf03e59de73c4d12bfdbfd87fff032ad1`    
2023-08-02 14:47:06 Unable to pin block for finality notification. hash: 0x0f77…45a0, Error: UnknownBlock: State already discarded for `0x0f77c274583a31577d948cb10154632d8a16136a93f7aec4fafe9f39532b45a0`    
2023-08-02 14:47:07 Unable to pin block for finality notification. hash: 0xdee2…b1fb, Error: UnknownBlock: State already discarded for `0xdee292a789c06ac2dcdae3299d233046cfeeeee713e62f157258ae764d95b1fb`    
2023-08-02 14:47:07 Unable to pin block for finality notification. hash: 0x4e51…df32, Error: UnknownBlock: State already discarded for `0x4e51716496534478655ec5b5c8b93add8c7f851da134a9386a17199f5adadf32`    
2023-08-02 14:47:08 Unable to pin block for finality notification. hash: 0x21c6…0e6c, Error: UnknownBlock: State already discarded for `0x21c6fee34b9f3c19ce253b1f174ef81d0b39f22b402fa11b384d4331b3800e6c`    
2023-08-02 14:47:09 Unable to pin block for finality notification. hash: 0x3ab5…60b2, Error: UnknownBlock: State already discarded for `0x3ab58d3bdee9c5eed7cc25e1929440eab08222a3dcd6d69e8f2df99daf3a60b2`    
2023-08-02 14:47:09 ⚙️  Syncing  0.0 bps, target=#6949551 (4 peers), best: #0 (0xbb7f…66be), finalized #15872 (0x3ab5…60b2), ⬇ 453.9kiB/s ⬆ 11.3kiB/s    
2023-08-02 14:47:09 Unable to pin block for finality notification. hash: 0xa40c…fc30, Error: UnknownBlock: State already discarded for `0xa40c1144ba27c0185c60f7471186676be1be6e1809b3e9a88c1975734af5fc30`    
2023-08-02 14:47:10 Unable to pin block for finality notification. hash: 0x6a75…74f0, Error: UnknownBlock: State already discarded for `0x6a75f45516ed28466498a1172bd50cb4bd10f73f049435e99d77f5494fec74f0`    
2023-08-02 14:47:10 Unable to pin block for finality notification. hash: 0x4b66…137b, Error: UnknownBlock: State already discarded for `0x4b66bf551d0e4c660c675d5c3f360d5769149a302dd1f48bc701adf2447c137b`    
2023-08-02 14:47:11 Unable to pin block for finality notification. hash: 0x5cd8…43d2, Error: UnknownBlock: State already discarded for `0x5cd814637f9bbc6cdeebca295139264591c818b54e62c46aa232439a339743d2`    
2023-08-02 14:47:12 Unable to pin block for finality notification. hash: 0x5141…f5ec, Error: UnknownBlock: State already discarded for `0x51419592c2516be611d14e9e4e0450cc02c071f53f5f68e9229ba526bdf2f5ec`

Should I wait for the node to finish syncing in fast mode before running the script again?

DylanVerstraete commented 1 year ago

You need to connect to a node that has unsafe rpc, see https://github.com/threefoldtech/tfchain/tree/development/tools/fork-off-substrate#network-list

sameh-farouk commented 1 year ago

So, @DylanVerstraete mentioned that fast and warp not supported for now.

while I'm waiting to get access to a running tfchain node for unsafe rpc operations, I did some cleaning/purge and start over, seems to have the forked chain partially working (creating blocks). but have new errors: Corrupted state and OffchainSignedTxError

logs:

Imported #36 (0x6d7b…ef78)    
2023-08-02 18:44:54 No contracts to bill at block 36, index: 6    
2023-08-02 18:44:54 Requested started id=1157 method=GET uri=https://horizon.stellar.org/paths/strict-receive?source_assets=USDC%3AGA5ZSEJYB37JRC5AVCIA5MOP4RHTM335X2KGX3IHOJAPP5RE34K4KZVN&destination_asset_type=credit_alphanum4&destination_asset_issuer=GBOVQKJYHXRR3DX6NOX2RRYFRCUMSADGDESTDNBDS6CDVLGVESRTAC47&destination_asset_code=TFT&destination_amount=100
2023-08-02 18:44:54 Got price: 8 mUSD    
2023-08-02 18:44:54 Corrupted state at `[38, 170, 57, 78, 234, 86, 48, 224, 124, 72, 174, 12, 149, 88, 206, 247, 185, 157, 136, 14, 198, 129, 121, 156, 12, 243, 14, 136, 134, 55, 29, 169, 222, 30, 134, 169, 168, 199, 57, 134, 76, 243, 204, 94, 194, 190, 165, 159, 212, 53, 147, 199, 21, 253, 211, 28, 97, 20, 26, 189, 4, 169, 159, 214, 130, 44, 133, 88, 133, 76, 205, 227, 154, 86, 132, 231, 165, 109, 162, 125]: Error`    
2023-08-02 18:44:54 Corrupted state at `[38, 170, 57, 78, 234, 86, 48, 224, 124, 72, 174, 12, 149, 88, 206, 247, 185, 157, 136, 14, 198, 129, 121, 156, 12, 243, 14, 136, 134, 55, 29, 169, 222, 30, 134, 169, 168, 199, 57, 134, 76, 243, 204, 94, 194, 190, 165, 159, 212, 53, 147, 199, 21, 253, 211, 28, 97, 20, 26, 189, 4, 169, 159, 214, 130, 44, 133, 88, 133, 76, 205, 227, 154, 86, 132, 231, 165, 109, 162, 125]: Error`    
2023-08-02 18:44:54 Corrupted state at `[38, 170, 57, 78, 234, 86, 48, 224, 124, 72, 174, 12, 149, 88, 206, 247, 185, 157, 136, 14, 198, 129, 121, 156, 12, 243, 14, 136, 134, 55, 29, 169, 222, 30, 134, 169, 168, 199, 57, 134, 76, 243, 204, 94, 194, 190, 165, 159, 212, 53, 147, 199, 21, 253, 211, 28, 97, 20, 26, 189, 4, 169, 159, 214, 130, 44, 133, 88, 133, 76, 205, 227, 154, 86, 132, 231, 165, 109, 162, 125]: Error`    
2023-08-02 18:44:54 Corrupted state at `[38, 170, 57, 78, 234, 86, 48, 224, 124, 72, 174, 12, 149, 88, 206, 247, 185, 157, 136, 14, 198, 129, 121, 156, 12, 243, 14, 136, 134, 55, 29, 169, 222, 30, 134, 169, 168, 199, 57, 134, 76, 243, 204, 94, 194, 190, 165, 159, 212, 53, 147, 199, 21, 253, 211, 28, 97, 20, 26, 189, 4, 169, 159, 214, 130, 44, 133, 88, 133, 76, 205, 227, 154, 86, 132, 231, 165, 109, 162, 125]: Error`    
2023-08-02 18:44:54 (offchain call) Error submitting a transaction to the pool: Transaction pool error: Invalid transaction validity: InvalidTransaction::Payment    
2023-08-02 18:44:54 failure: offchain_signed_tx: tx sent:     
2023-08-02 18:44:54 OffchainSignedTxError

also there was an error when i lunched the script unable to map u16 to a lookup index, is this expected?

❯ npm start

> fork-off-substrate@1.0.0 start
> node index.js

We are intentionally using the HTTP endpoint. If you see any warnings about that, please ignore them.
Custom Schema missing, using default schema.
2023-08-02 17:57:30        API/INIT: Api will be available in a limited mode since the provider does not support subscriptions
2023-08-02 17:57:30        API/INIT: RPC methods not decorated: chainHead_unstable_body, chainHead_unstable_call, chainHead_unstable_follow, chainHead_unstable_genesisHash, chainHead_unstable_header, chainHead_unstable_stopBody, chainHead_unstable_stopCall, chainHead_unstable_stopStorage, chainHead_unstable_storage, chainHead_unstable_unfollow, chainHead_unstable_unpin, transaction_unstable_submitAndWatch, transaction_unstable_unwatch
2023-08-02 17:57:30        METADATA: Unknown types found, no types for BufferIndex, Burn, BurnTransaction, CertificationCodeType, CertificationCodes, CertificationType, Consumption, Contract, ContractBill, ContractBillingInformation, ContractResources, Entity, Farm, FarmingPolicy, Interface, Location, MintTransaction, Node, NruConsumption, Policy, PricingPolicy, PublicConfig, PublicIP, RefundTransaction, Resources, StellarSignature, StorageVersion, TermsAndConditions, Twin, U16F16, Validator, ValueStruct
Unable to map u16 to a lookup index
Unable to map u16 to a lookup index
Unable to map u16 to a lookup index
Unable to map u16 to a lookup index
Unable to map u16 to a lookup index
Unable to map u16 to a lookup index
Unable to map u16 to a lookup index
Unable to map u16 to a lookup index
Unable to map u16 to a lookup index
Unable to map u16 to a lookup index
Unable to map u16 to a lookup index
Unable to map u16 to a lookup index
Unable to map u16 to a lookup index
Unable to map u16 to a lookup index
Unable to map u16 to a lookup index
Unable to map u16 to a lookup index
Unable to map u16 to a lookup index
Unable to map u16 to a lookup index
Unable to map u16 to a lookup index
Unable to map u16 to a lookup index
Fetching current state of the live chain. Please wait, it can take a while depending on the size of your chain.
 ████████████████████████████████████████ 100% | ETA: 0s | 256/256
2023-08-02 18:57:32 Building chain spec    
2023-08-02 18:57:32 Building chain spec    
Forked genesis generated successfully. Find it at ./data/fork.json
DylanVerstraete commented 1 year ago

@sameh-farouk which network and how did you fork it?

sameh-farouk commented 1 year ago

Same as the before, dev network, following steps mentioned in the issue.

DylanVerstraete commented 1 year ago

How did you fork without access to the private node?

DylanVerstraete commented 1 year ago

I can't reproduce your issue

sameh-farouk commented 1 year ago

I know you advised me to use a full synced node as (the preferred) option. However, as I didn't get VPN access to the office yet, I executed the script against a local node (running with unsafe-rpc flag) that has not finished syncing.

Is this a valid option?

DylanVerstraete commented 1 year ago

@sameh-farouk That is not really an option, you need to fork an synced network