search

worldwide-asset-exchange / wax-blockchain-legacy

Worldwide Asset eXchange (WAX/wax.io) is an eos compatible blockchain and protocol token designed to make e-commerce transactions faster, easier, and safer.
https://developer.wax.io/
MIT License
125 stars 47 forks source link

env.verify_rsa_sha256_sig unresolveable #28

Closed cc32d9 closed 5 years ago

cc32d9 commented 5 years ago

Some servers under "wax-1.8.1-1.0.0-rc2-19-ge2187f190" cannot get past the block 9135392, with the error as follows:

Aug 18 08:32:43 wax nodeos[18746]: error 2019-08-18T08:32:43.883 nodeos    producer_plugin.cpp:400       on_incoming_block    ] 3070000 wasm_exception: WASM Exception
Aug 18 08:32:43 wax nodeos[18746]: env.verify_rsa_sha256_sig unresolveable
Aug 18 08:32:43 wax nodeos[18746]:     {"module":"env","export":"verify_rsa_sha256_sig"}
Aug 18 08:32:43 wax nodeos[18746]:     nodeos  wasm_interface.hpp:59 resolve
Aug 18 08:32:43 wax nodeos[18746]:     {"mod_name":"env","export_name":"verify_rsa_sha256_sig"}
Aug 18 08:32:43 wax nodeos[18746]:     nodeos  wasm_interface.hpp:61 resolve
Aug 18 08:32:43 wax nodeos[18746]: pending console output:
Aug 18 08:32:43 wax nodeos[18746]:     {"console":""}
Aug 18 08:32:43 wax nodeos[18746]:     nodeos  apply_context.cpp:113 exec_one
Aug 18 08:32:43 wax nodeos[18746]:     {}
Aug 18 08:32:43 wax nodeos[18746]:     nodeos  controller.cpp:1765 apply_block
Aug 18 08:32:43 wax nodeos[18746]: rethrow
Aug 18 08:32:43 wax nodeos[18746]:     {}
Aug 18 08:32:43 wax nodeos[18746]:     nodeos  controller.cpp:1821 push_block

Other servers with the same software version work fine. This failing server has only one peer that is on a server that made it through. Michael at EOS USA has also a machine that fails in the same way, but another machine in the same environment works fine.

cc32d9 commented 5 years ago

I noticed that was from "develop" branch, because Git picked it up as default. Now recompiled from "master", and result is the same, with exactly the same error.

cc32d9 commented 5 years ago

Build procedure: 

mkdir /opt/src/
cd /opt/src
git clone -b master https://github.com/worldwide-asset-exchange/wax-blockchain.git
cd wax-blockchain/
git submodule update --init --recursive
./wax_build.sh -P -y
./wax_install.sh

Now trying without the -P flag

cc32d9 commented 5 years ago

it went through without the -P flag. So it's probably an issue with clang optimizer

jalbiero commented 5 years ago

Hi @cc32d9! Please use our release version (wax-1.8.1-1.0.0 instead of wax-1.8.1-1.0.0-rc2). Be aware that release candidate versions are not ready for production (they are released for testing purposes and may have bugs).

ankh2054 commented 5 years ago

Same issue for me on 1.8.1-1.0.0

igorls commented 5 years ago

Same issue here on 1.8.1-1.0.0 with state-history enabled (build with -P)

jalbiero commented 5 years ago

Hi guys. We are investigating the problem. In the meantime, could you post some useful information for us?

  1. How did you get the code? ($ git clone -b wax-1.8.1-1.0.0 https://github.com/worldwide-asset-exchange/wax-blockchain.git is recommended)
  2. How did you compile the code? (updated instructions are in develop branch: README.md)
  3. Operating system used (Ubuntu 18.04 is recommended, but not necessary)
  4. Result of: $ nodeos --version (expected wax-1.8.1-1.0.0)
  5. Result of: $ cleos get info
  6. Logs from nodeos (especially the first 50 lines)

Thanks

cc32d9 commented 5 years ago

Here's what I used: 

mkdir /opt/src/
cd /opt/src
git clone -b master https://github.com/worldwide-asset-exchange/wax-blockchain.git
cd wax-blockchain/
git submodule update --init --recursive
./wax_build.sh -P -y
./wax_install.sh

later on, I removed -P and it started working. On another machine, this worked with -P too. The machines have different generations of intel Xeon CPU

jalbiero commented 5 years ago

Hi @cc32d9, so you cloned from master, but according to your first post, your version is wax-1.8.1-1.0.0-rc2. That's not possible, rc2 was never merged to master (actually we never merge release candidate versions to master). Could you please check the version of your nodeos binary? (step 4 of my previous post). Last but not least, yesterday we recompiled, from scratch, version wax-1.8.1-1.0.0 and performed a full sync without any issue.

jalbiero commented 5 years ago

Due to our policy we have to close this issue because we waited a week for an answer. Don't hesitate to open a new issue if you still have problems.

Remember, use our updated installation instructions README.md

cc32d9 commented 5 years ago

no, wait, I mentioned above that I cloned later from proper branch and the problem remained. Also other users confirm the problem. Please reopen it, as the issue is still unresolved.

jalbiero commented 5 years ago

But none of you provided the information we asked in order to help us with the resolution (6 items). We cannot reproduce the problem and without those items we cannot be sure that you are running the same version that we have in production. I will reopen the issue, but you will have to provide the information I asked you.

cc32d9 commented 5 years ago

the problem is that it's shaky. The same setup works on one machine and fails on the other. I'll try to reproduce it in a test environment. The case was opened when I was trying to repair the production environment, so some logs were lost in the rush.

jalbiero commented 5 years ago

Please be aware that if no one can provide more information (especially the one we asked) this issue will be closed tomorrow. Thanks.

cc32d9 commented 5 years ago

sorry I was taken away by urgent work. This is a critical issue confirmed by several teams, so it cannot be closed and ignored.

jalbiero commented 5 years ago

Hi @cc32d9 the official way is to use our docker image. If you can, please try with that, it's really easy (more information here: README.md).

I am sorry, I don't want to be rude, but we need evidence and more information to confirm the problem. We are willing to help you, actually, we try, but we couldn't reproduce it, and people here that said that they experienced the same didn't provide any information that lead us to confirm the problem.

cc32d9 commented 5 years ago

I don't have docker in our production infrastructure. The difficulty is that the same installation procedure on two different servers gave different results, and one was failing while the other was OK. I will try to find the time to reproduce it within a couple of days.

matthewdarwin commented 5 years ago

I ran into this problem as well, however I am not runnnig docker. It would get stuck at that block with sync from network, replaying blocks from a pre-existing block log worked fine.

cc32d9 commented 5 years ago

@jalbiero your installation instructions don't have the -P option for the build script, and I have already got a working server without this option. But -P is important because it fixates the compiler version, so that subsequent upgrades are compatible with the data files. This error occurred when I used the -P option.

matthewdarwin commented 5 years ago

I used "-P" as well.

cc32d9 commented 5 years ago

I'm setting up a test instance now, compiling with -P

cc32d9 commented 5 years ago

here we go, I reproduced it on the same physical machine, in a separate LXC container.

root@waxtest:~# cat /etc/os-release 
NAME="Ubuntu"
VERSION="18.04.3 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.3 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

Build procedure:

mkdir /opt/src/
cd /opt/src
git clone -b wax-1.8.1-1.0.0 https://github.com/worldwide-asset-exchange/wax-blockchain.git
cd wax-blockchain/
git submodule update --init --recursive
./wax_build.sh -P -y 
./wax_install.sh

config.ini:

root@waxtest:~# cat /srv/testnode/etc/config.ini 
chain-state-db-size-mb = 65536
reversible-blocks-db-size-mb = 2048
wasm-runtime = wavm
http-server-address = 0.0.0.0:8888
p2p-listen-endpoint = 0.0.0.0:9876
access-control-allow-origin = *
max-clients = 100
plugin = eosio::chain_plugin
plugin = eosio::chain_api_plugin
validation-mode = light
sync-fetch-span = 500
p2p-peer-address = peers.wax.io:9876
p2p-peer-address = br.eosrio.io:35668
p2p-peer-address = p2p.waxsweden.org:35777
p2p-peer-address = wax.blockmatrix.network:13546
p2p-peer-address = waxp2p.eoscafeblock.com:9090
p2p-peer-address = waxpeer.eos42.io:9879
p2p-peer-address = wax.zenblocks.io:13975
p2p-peer-address = peer.wax.alohaeos.com:9876
p2p-peer-address = peer1-wax.eosphere.io:9876
p2p-peer-address = wax-p2p.infinitybloc.io:9876
p2p-peer-address = wax.greymass.com:35777
p2p-peer-address = wax-p2p.eos.barcelona:9876
p2p-peer-address = wax.eosusa.news:9879

get info:

root@waxtest:~# /root/eosio/1.8/bin/cleos get info
{
  "server_version": "2fc665e9",
  "chain_id": "1064487b3cd1a897ce03ae5b6a865651747e2e152090f99c1d19d44e01aea5a4",
  "head_block_num": 9135392,
  "last_irreversible_block_num": 9135065,
  "last_irreversible_block_id": "008b63d999601bf346088db9a2a7a47edcf408ad2865248eeeaa7107149ada42",
  "head_block_id": "008b6520e9aab1d849205c7ae9e43934636dc70eddb9139122208f0cf100a5fc",
  "head_block_time": "2019-08-16T19:43:01.000",
  "head_block_producer": "mining.wax",
  "virtual_block_cpu_limit": 500000000,
  "virtual_block_net_limit": 1048576000,
  "block_cpu_limit": 500000,
  "block_net_limit": 1048576,
  "server_version_string": "wax-1.8.1-1.0.0",
  "fork_db_head_block_num": 9135392,
  "fork_db_head_block_id": "008b6520e9aab1d849205c7ae9e43934636dc70eddb9139122208f0cf100a5fc"
}

The node started from state and blocks log at about block 9083000. The whole log is attached. screenlog.txt

The whole log

cc32d9 commented 5 years ago

get info was taken after the node got stuck.

cc32d9 commented 5 years ago

/proc/cpuinfo shows:

model name  : Intel(R) Xeon(R) CPU E5-2420 0 @ 1.90GHz

On a different physical machine where the sync went smoothly,

model name  : Intel(R) Xeon(R) E-2186G CPU @ 3.80GHz
cc32d9 commented 5 years ago

The whole content of data and config directories at block 9083000 is available at https://snapshots.eosamsterdam.net/wax/wax_issue_28_testnode.tar.gz

jalbiero commented 5 years ago

Excellent @cc32d9, your help is really appreciated. We are analyzing the information that you provided. Thanks!

cc32d9 commented 5 years ago

just doing my job :) The log is best to view in raw mode:

less -r screenlog.txt
jalbiero commented 5 years ago

Hi @cc32d9. We try to reproduce the problem, but we couldn't. Our production node with you config file, a node compiled with -P (pinned compiler) with our config file, and with your config file, all of them synchronized correctly.

FYI: The problem was present in wax-1.8.1-1.0.0-rc1 (because the function _verify_rsa_sha256sig was not exported correctly). rc2 and final release don't have the mentioned issue (at least for us)

cc32d9 commented 5 years ago

thanks, I will try to re-check with 1.8.3 during the week, on the same machine.

cc32d9 commented 5 years ago

I made a fresh build of 1.8.4 and the issue is still reproduceable on this machine.

cc32d9 commented 5 years ago 
root@waxtest:~# /root/eosio/1.8/bin/cleos get info
{
  "server_version": "7328c2db",
  "chain_id": "1064487b3cd1a897ce03ae5b6a865651747e2e152090f99c1d19d44e01aea5a4",
  "head_block_num": 9135392,
  "last_irreversible_block_num": 9135065,
  "last_irreversible_block_id": "008b63d999601bf346088db9a2a7a47edcf408ad2865248eeeaa7107149ada42",
  "head_block_id": "008b6520e9aab1d849205c7ae9e43934636dc70eddb9139122208f0cf100a5fc",
  "head_block_time": "2019-08-16T19:43:01.000",
  "head_block_producer": "mining.wax",
  "virtual_block_cpu_limit": 500000000,
  "virtual_block_net_limit": 1048576000,
  "block_cpu_limit": 500000,
  "block_net_limit": 1048576,
  "server_version_string": "wax-1.8.4-1.0.0",
  "fork_db_head_block_num": 9135392,
  "fork_db_head_block_id": "008b6520e9aab1d849205c7ae9e43934636dc70eddb9139122208f0cf100a5fc"
}
cc32d9 commented 5 years ago

I copied the nodeos binary from a server that was working well onto this test machine, and it produces the same error. So it's not a compiler problem, but a runtime problem. Probably some cryptographic primitives are not implemented in this CPU model?

The one failing:

vendor_id       : GenuineIntel
cpu family      : 6
model           : 45
model name      : Intel(R) Xeon(R) CPU E5-2420 0 @ 1.90GHz
stepping        : 7
microcode       : 0x718
cpu MHz         : 1292.526
cache size      : 15360 KB
physical id     : 0
siblings        : 12
core id         : 0
cpu cores       : 6
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm epb pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid xsaveopt dtherm arat pln pts md_clear flush_l1d
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds
bogomips        : 3790.87
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

The one working fine:

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 158
model name      : Intel(R) Xeon(R) E-2186G CPU @ 3.80GHz
stepping        : 10
microcode       : 0xb4
cpu MHz         : 901.072
cache size      : 12288 KB
physical id     : 0
siblings        : 12
core id         : 0
cpu cores       : 6
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 22
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_act_window hwp_epp md_clear flush_l1d
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs
bogomips        : 7584.00
clflush size    : 64
cache_alignment : 64
address sizes   : 39 bits physical, 48 bits virtual
power management:
matthewdarwin commented 5 years ago

I tried my own binary on 3 different servers, at least 1 of which didn't work on earlier version. All work now.

processor       : 31
vendor_id       : GenuineIntel
cpu family      : 6
model           : 45
model name      : Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz
stepping        : 7
microcode       : 0x714
cpu MHz         : 3300.134
cache size      : 20480 KB
physical id     : 1
siblings        : 16
core id         : 7
cpu cores       : 8
apicid          : 47
initial apicid  : 47
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid xsaveopt dtherm ida arat pln pts flush_l1d
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs
bogomips        : 5801.74
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:
processor       : 3
vendor_id       : GenuineIntel
cpu family      : 6
model           : 158
model name      : Intel(R) Xeon(R) CPU E3-1230 v6 @ 3.50GHz
stepping        : 9
microcode       : 0x84
cpu MHz         : 3699.999
cache size      : 8192 KB
physical id     : 0
siblings        : 4
core id         : 3
cpu cores       : 4
apicid          : 6
initial apicid  : 6
fpu             : yes
fpu_exception   : yes
cpuid level     : 22
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs
bogomips        : 7008.00
clflush size    : 64
cache_alignment : 64
address sizes   : 39 bits physical, 48 bits virtual
power management:
processor       : 7
vendor_id       : GenuineIntel
cpu family      : 6
model           : 158
model name      : Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz
stepping        : 12
microcode       : 0xa2
cpu MHz         : 4653.017
cache size      : 16384 KB
physical id     : 0
siblings        : 8
core id         : 7
cpu cores       : 8
apicid          : 14
initial apicid  : 14
fpu             : yes
fpu_exception   : yes
cpuid level     : 22
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp flush_l1d arch_capabilities
bugs            : spectre_v1 spectre_v2 spec_store_bypass mds swapgs
bogomips        : 7200.00
clflush size    : 64
cache_alignment : 64
address sizes   : 39 bits physical, 48 bits virtual
power management:
cc32d9 commented 5 years ago

Reproduced with i9: 

processor   : 15
vendor_id   : GenuineIntel
cpu family  : 6
model       : 158
model name  : Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz
stepping    : 13
microcode   : 0xb8
cpu MHz     : 4850.207
cache size  : 16384 KB
physical id : 0
siblings    : 16
core id     : 7
cpu cores   : 8
apicid      : 15
initial apicid  : 15
fpu     : yes
fpu_exception   : yes
cpuid level : 22
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d arch_capabilities
bugs        : spectre_v1 spectre_v2 spec_store_bypass swapgs
bogomips    : 7200.00
clflush size    : 64
cache_alignment : 64
address sizes   : 39 bits physical, 48 bits virtual
power management:
cc32d9 commented 5 years ago

and now, the most interesting part. Same error without -P flag. The server is with i9 CPU as in previous message.

Building steps:

mkdir /srv/wax/src/
cd /srv/wax/src/
git clone -b master https://github.com/worldwide-asset-exchange/wax-blockchain.git
cd wax-blockchain/
git submodule update --init --recursive
./wax_build.sh -i /srv/wax/opt -y
./wax_install.sh

Downloaded Eric's snapshot from right before the bad block: https://snapshots.waxsweden.org/snapshots/57abfa/1.8.eosio/snapshot-9123753.bin.tar.gz

/srv/wax/opt/bin/nodeos --data-dir /srv/wax/data --config-dir /srv/wax/etc --snapshot tmp/snapshot-9123753.bin

it results in the same error, "WASM Exception env.verify_rsa_sha256_sig unresolveable"

cc32d9 commented 5 years ago

ok that snapshpot was made with vanilla eosio. If starting from a snapshot which was created by wax software, it goes through the bad block

cc32d9 commented 5 years ago

if started from a snapshot made with Wax software that has run from the genesis, both with and without -P nodes go over this block without issues. I will retest on the machine that was originally used for reporting.

cc32d9 commented 5 years ago

tested on the other machine, with the same results. So, the outcome is, that the state produced by vanilla nodeos produces this error after switching to WAX software. If WAX software has replayed from genesis, the error does not pop up with or without -P option.