WAX ship cannot start from snapshot: deltas is too big

[cc32d9]

WAX node, 2.0.13wax01, with sip enabled, starting from a fresh snapshot, is impossible to start. Giving the error like this, and trying to start from a new block:

info  2022-01-27T11:37:15.125 net-0     net_plugin.cpp:1556           operator()           ] requesting range 163571776 to 163571875, from 37.58.52.123:9876 - 825b55d
warn  2022-01-27T11:37:19.129 nodeos    controller.cpp:379            emit                 ] fc::exception: 3110000 plugin_exception: Plugin exception
deltas is too big
    <- timeout exception in to_detail_string: deadline 2022-01-27T11:37:18.985 exceeded by 143044us 

info  2022-01-27T11:37:19.930 nodeos    state_history_plugin.c:486    store_chain_state    ] Placing initial state in block 163571777
info  2022-01-27T11:37:24.131 net-1     net_plugin.cpp:1619           sync_reassign_fetch  ] reassign_fetch, our last req is 163571875, next expected is 163571777 peer 37.58.52.123:9876 - 825b55d
info  2022-01-27T11:37:24.131 net-1     net_plugin.cpp:1203           cancel_sync          ] sending empty request but not calling sync wait on wax.eu.eosamsterdam.net:9101
info  2022-01-27T11:37:24.131 net-1     net_plugin.cpp:1556           operator()           ] requesting range 163571777 to 163571876, from 168.119.5.143:9880 - f234e88

Full installation procedure here, and nodeos log is attached.

AMD Ryzen 9 PRO 3900 12-Core Processor
128 GB RAM
900GB SSD
Ubuntu 18.04

root@7F0F499:~# uname -a
Linux 7F0F499 5.4.0-96-generic #109~18.04.1-Ubuntu SMP Thu Jan 13 15:06:26 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

fallocate -l 300000000000 /swap
chmod 600 /swap 
mkswap /swap 
swapon /swap

root@7F0F499:~# free -g
              total        used        free      shared  buff/cache   available
Mem:            125           0         125           0           0         124
Swap:           286           0         286

cd /var/local
wget https://apt.eossweden.org/wax/pool/stable/w/wax/wax_2.0.13wax01-1-ubuntu-18.04_amd64.deb
apt install -y ./wax_2.0.13wax01-1-ubuntu-18.04_amd64.deb

mkdir -p /srv/wax/etc
cat > /srv/wax/etc/config.ini <<'EOT'
chain-state-db-size-mb = 96000
reversible-blocks-db-size-mb = 2048
read-mode = head
wasm-runtime = eos-vm-jit
eos-vm-oc-enable = true
p2p-accept-transactions = false
api-accept-transactions = false
http-server-address = 0.0.0.0:8888
p2p-listen-endpoint = 0.0.0.0:9801
access-control-allow-origin = *
plugin = eosio::chain_plugin
plugin = eosio::chain_api_plugin
validation-mode = light
plugin = eosio::state_history_plugin
trace-history = true
chain-state-history = true
state-history-endpoint = 0.0.0.0:8080
p2p-peer-address = wax.eu.eosamsterdam.net:9101
p2p-peer-address = peer.3dkrender.com:9880
EOT

curl https://snapshots-cdn.eossweden.org/wax/2.0/snapshot-163571775.bin.tar.gz | tar xzvf -

root@7F0F499:/var/local# ls -al snapshot-163571775.bin 
-rw-r--r-- 1 1003 1003 20508836146 Jan 27 00:41 snapshot-163571775.bin

nohup /usr/bin/nodeos --data-dir /srv/wax/data --config-dir /srv/wax/etc --disable-replay-opts --snapshot snapshot-163571775.bin &

nohup.out.txt

[praphael]

Try reducing --sync-fetch-span from its default of 100 to something smaller.

https://developers.eos.io/manuals/eos/v2.0/nodeos/plugins/net_plugin/index

[heifner]

The delta is bigger than uint32_t size can hold. I believe this has been fixed in 2.1.x.

[cc32d9]

2.1 started fine from the same snapshot

[heifner]

https://github.com/EOSIO/eos/pull/11051

[cc32d9]

tested successfully, and released 2.0.13wax13 which includes the patch: https://github.com/cc32d9/wax2.0