bug(dashmate): Dashmate v0.25.15 & v0.25.16-rc.1+rc.2 on Mainnet -> error + interfering with dashmate status and stop commands

[qwizzie]

Evonode 1 (Ubuntu 20.04.6 LTS / Node v21.2.0 / NPM v10.2.4) Edit : later downgraded to Node v20.10.0, same problem

Updated Dashmate from v0.25.13 to v0.25.15 and through manual command (dashmate config set core.docker.image dashpay/dashd:20.0.0) started Core v20. Dashmate had no problem starting Core v20, but gives error with Dashmate status and dashmate stop commands

CompileError: WebAssembly.Module(): Compiling function #331:"memchr::arch::wasm32::simd128:: packedpair::Find..." failed: Wasm SIMD unsupported @+1956688

Evonode 2 (Ubuntu 20.04.6 LTS / Node v21.2.0 / NPM v10.2.4) Edit : later downgraded to Node v20.10.0, same problem

Updated Dashmate from v0.25.13 to v0.25.15 and gave dashmate a start command for v19.3 (this was just a dashmate update, i was already running Core v19.3 on this node, but with Dashmate v0.25.13 Dashmate had no problem starting Core v19.3, but gives error with Dashmate status and dashmate stop commands

CompileError: WebAssembly.Module(): Compiling function #331:"memchr::arch::wasm32::simd128:: packedpair::Find..." failed: Wasm SIMD unsupported @+1956688

See : https://www.dash.org/forum/index.php?threads/dashmate-discussion.53951/page-3#post-236933

[qwizzie]

Reverted back to Dashmate v0.25.13 by issueing 'npm install -g dashmate@0.25.13' & 'dashmate update'. Dashmate status and stop commands work again and both nodes are running Core v20.

Both of my Evonodes got PoSe scored (2200+), i strongly suspect because of faulty Dashmate version 0.25.15

[qwizzie]

Issue seems related to Node.JS version. Dashmate v0.25.13 had no problem with Node.JS v20.8.0 / Core v19.3

Dashmate does seem to run into problems, when coming across Node.JS v21.2.0 / Core v20.0 (at least with Dashmate v0.25.15)

Solution : have users limit Node.JS to v18 for now (fully supported in Dashmate) and extend this to Node.JS v20 later on.

Related : https://github.com/dashpay/platform/pull/1562

Pshenmic mentioned following code for reverting Node.JS to v18 :

nvm install 18 nvm use 18

[qwizzie]

See : https://www.dash.org/forum/index.php?threads/dashmate-discussion.53951/page-3#post-236976

But I rolled out a release candidate build for your, so you can proceed with Core v20 masternode updates. I tested this release locally and on the PE testnet fullnode, my node was able to sync up and shows correct image versions.

This release contains a breaking change and upgrade to Node.JS v20 is necessary

Assuming you meant for us to test Dashmate v0.25.16-rc.1 and not wait for a v0.25.16 final release, i have tested this as follows :

$ nvm install 20 Downloading and installing node v20.9.0... Downloading https://nodejs.org/dist/v20.9.0/node-v20.9.0-linux-x64.tar.xz... ############################################################################################################################################################################################################ 100.0% Computing checksum with sha256sum Checksums matched! Now using node v20.9.0 (npm v10.1.0) $ nvm alias default 20 default -> 20 (-> v20.9.0) $ nvm use 20 Now using node v20.9.0 (npm v10.1.0) $npm install -g dashmate@0.25.16-rc.1

added 664 packages in 37s

113 packages are looking for funding run npm fund for details $ dashmate update

CompileError: WebAssembly.Module(): Compiling function #1302:"bytecount::simd::wasm::chunk_num_chars::he46d9f..." failed: Wasm SIMD unsupported @+2532414

Rather similar looking error i had with Dashmate v0.25.15

I did not try to do a 'dashmate start' after seeing this error, as i am afraid i will again be left with no control over the 'dashmate status' and 'dashmate stop' commands.

I reverted back to dashmate v0.25.13 and Dashmate is now running Core v20.0.0 again, through previously set manual configuration (dashmate config set core.docker.image dashpay/dashd:20.0.0)

All i can say is that Node.JS v20.9.0 works without problem on Dashmate v0.25.13 but has a problem with Dashmate v0.25.16-rc.1 (at least for me).

I suspect Node.JS v20.9.0 also has a problem with Dashmate v0.25.15 but have not actually tested this, as i was either on Dashmate v0.25.13 (Node.JS v21 & v20 --> ok) or Dashmate v0.25.15 (Node.JS v21 --> error) or Dashmate v0.25.16-rc.1 (Node.JS v20 --> error).

Or maybe the problem is actually unrelated to Node.JS and more to do with changes (however minor in scope) in Dashmate v0.25.15 / v0.25.16-rc1

[pshenmic]

https://github.com/dashpay/platform/pull/1591

[pshenmic]

According @qwizzie report on the Dash Forum, the issue happens on the Ubuntu 22.04 Ubuntu 20.04 x86_64 with the given CPU:

$lscpu

Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 44 bits physical, 48 bits virtual
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 1
Core(s) per socket: 8
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6136 CPU @ 3.00GHz
Stepping: 4
CPU MHz: 2992.972
BogoMIPS: 5985.94
Hypervisor vendor: Microsoft
Virtualization type: full
L1d cache: 256 KiB
L1i cache: 256 KiB
L2 cache: 8 MiB
L3 cache: 198 MiB
NUMA node0 CPU(s): 0-7

[qwizzie]

Actually my OS is Ubuntu 20.04.6 LTS / GNU/Linux 5.4.0-166-generic x86_64 Not 22.04 as pshenmic mentioned above.

With regards to the fix to disable WASM-DPP initialization :

Dashmate load up and init WASM DPP on every command run, but in fact, it is only used in one place, where latest appVersion being extracted and set in local setup.

This causes cross-platform issues on some specific CPUs, more details could be read here #1583

This PR removes initialization of WASM DPP in the dashmate

I just hope this does not mean i will have problems again with my specific CPU, once Platform gets released on Mainnet and Evonodes need to update to get access to both L1 & L2. Or can it be coded in such a way that WASM DPP initialization will not give cross-platform issues in the future ?

Also why no cross-platform issue with Dashmate v0.25.13 for my CPU ? What changed between v0.25.13 and v0.25.15 to affect cross-platform this way ? Are we sure that removing certain obsolete dependencies of Rust crates, did not cause the cross-platform issue in the first place ? See : https://github.com/dashpay/platform/pull/1578

[qwizzie]

I made a sample application to reproduce your case, could you please clone this repo https://github.com/pshenmic/wasm-dpp-nodejs-load-sample anywhere on your system, and try to run it: node index.js

On my side, it loads successfully:

% node index.js
Wasm DPP successfully loaded

Cloned directly into my home directory.

~$ git clone https://github.com/pshenmic/wasm-dpp-nodejs-load-sample Cloning into 'wasm-dpp-nodejs-load-sample'... remote: Enumerating objects: 14, done. remote: Counting objects: 100% (14/14), done. remote: Compressing objects: 100% (8/8), done. remote: Total 14 (delta 5), reused 14 (delta 5), pack-reused 0 Unpacking objects: 100% (14/14), 4.37 KiB | 1.09 MiB/s, done. $ cd wasm-dpp-nodejs-load-sample (i also tried to issue the node index.js command outside this directory, same error) ~/wasm-dpp-nodejs-load-sample $ node index.js node:internal/modules/esm/resolve:844 throw new ERR_MODULE_NOT_FOUND(packageName, fileURLToPath(base), null); ^

Error [ERR_MODULE_NOT_FOUND]: Cannot find package '@dashevo/wasm-dpp' imported from /home/USER/wasm-dpp-nodejs-load-sample/index.js at packageResolve (node:internal/modules/esm/resolve:844:9) at moduleResolve (node:internal/modules/esm/resolve:901:20) at defaultResolve (node:internal/modules/esm/resolve:1121:11) at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:396:12) at ModuleLoader.resolve (node:internal/modules/esm/loader:365:25) at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:240:38) at ModuleWrap. (node:internal/modules/esm/module_job:85:39) at link (node:internal/modules/esm/module_job:84:36) { code: 'ERR_MODULE_NOT_FOUND' }

Node.js v20.10.0

Maybe the problem is with @dashevo/wasm-dpp (incorrect workspace ?) Actual workspace : https://github.com/dashpay/platform/tree/v1.0-dev/packages/wasm-dpp ?

[pshenmic]

@qwizzie sorry, i forgot about npm install, could you try after that?

[qwizzie]

@pshenmic

cd wasm-dpp-nodejs-load-sample ~/wasm-dpp-nodejs-load-sample$ npm install ~/wasm-dpp-nodejs-load-sample$ node index.js

CompileError: WebAssembly.Module(): Compiling function #331:"memchr::arch::wasm3 2::simd128::packedpair::Find..." failed: Wasm SIMD unsupported @+1955693

Node.js v20.10.0

I think this is because your sample application is looking for the @dashevo workspace. Same with v0.25.16-rc.4, that is also referencing @dashevo/wasm-dpp in its package json.

But the actual workspace has changed to https://github.com/dashpay/platform/tree/v1.0-dev/packages/wasm-dpp which is giving problems to Evonode owners and Masternode owners using Dashmate v0.25.15 (+v0.25.16-rc.x) on Mainnet.

[pshenmic]

Ok, that reproduced your issue and proved that the my pr will fix your issue.

The workspaces you are mentioning is not related to your issue

[qwizzie]

Dashmate v0.25.16-rc.6 fixed my problem. Issue solved.