search

ChainSafe / lodestar

🌟 TypeScript Implementation of Ethereum Consensus
https://lodestar.chainsafe.io
Apache License 2.0
1.17k stars 287 forks source link

Not able to build lodestar on contabo vpss #4945

Closed twoeths closed 1 year ago

twoeths commented 1 year ago

Describe the bug

Got this error when building lodestar from source

yarn install v1.22.19
[1/5] Validating package.json...
[2/5] Resolving packages...
[3/5] Fetching packages...
[4/5] Linking dependencies...
warning " > @lodestar/api@1.2.1" has unmet peer dependency "fastify@3.29.4".
warning " > @lodestar/beacon-node@1.2.1" has unmet peer dependency "c-kzg@^1.0.7".
warning " > @lodestar/reqresp@1.2.1" has unmet peer dependency "libp2p@0.39.2".
warning "workspace-aggregator-a358551c-f04b-4ee5-a2f7-b59dd384aef0 > @lodestar/beacon-node > @chainsafe/bls@7.1.1" has unmet peer dependency "@chainsafe/blst@^0.2.4".
warning "workspace-aggregator-a358551c-f04b-4ee5-a2f7-b59dd384aef0 > @lodestar/beacon-node > prometheus-gc-stats@0.6.3" has incorrect peer dependency "prom-client@>= 10 <= 12".
warning "workspace-aggregator-a358551c-f04b-4ee5-a2f7-b59dd384aef0 > @lodestar/beacon-node > @multiformats/multiaddr > dns-over-http-resolver > native-fetch@4.0.2" has unmet peer dependency "undici@*".
[5/5] Building fresh packages...
[11/18] ⠄ @chainsafe/blst
[2/18] ⠄ electron
[7/18] ⠄ bcrypto
[9/18] ⠄ snappy
error /usr/src/lodestar/node_modules/@chainsafe/blst: Command failed.
Exit code: 1
Command: node dist/scripts/install.js
Arguments:
Directory: /usr/src/lodestar/node_modules/@chainsafe/blst
Output:
node:internal/modules/cjs/loader:1243
  return process.dlopen(module, path.toNamespacedPath(filename));
                 ^

Error: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /usr/src/lodestar/node_modules/@chainsafe/blst/prebuild/linux-x64-108-binding.node)
    at Module._extensions..node (node:internal/modules/cjs/loader:1243:18)
    at Module.load (node:internal/modules/cjs/loader:1037:32)
    at Module._load (node:internal/modules/cjs/loader:878:12)
    at Module.require (node:internal/modules/cjs/loader:1061:19)
    at require (node:internal/modules/cjs/helpers:103:18)
    at [eval]:1:1
    at Script.runInThisContext (node:vm:129:12)
    at Object.runInThisContext (node:vm:313:38)
    at node:internal/process/execution:79:19
    at [eval]-wrapper:6:22 {
  code: 'ERR_DLOPEN_FAILED'
}

Node.js v18.12.1
Cached BLST native bindings not OK
Retrieving BLST native bindings /usr/src/lodestar/node_modules/@chainsafe/blst/prebuild/linux-x64-108-binding.node ...
node:internal/modules/cjs/loader:1243
  return process.dlopen(module, path.toNamespacedPath(filename));
                 ^

Error: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /usr/src/lodestar/node_modules/@chainsafe/blst/prebuild/linux-x64-108-binding.node)
    at Module._extensions..node (node:internal/modules/cjs/loader:1243:18)
    at Module.load (node:internal/modules/cjs/loader:1037:32)
    at Module._load (node:internal/modules/cjs/loader:878:12)
    at Module.require (node:internal/modules/cjs/loader:1061:19)
    at require (node:internal/modules/cjs/helpers:103:18)
    at [eval]:1:1
    at Script.runInThisContext (node:vm:129:12)
    at Object.runInThisContext (node:vm:313:38)
    at node:internal/process/execution:79:19
    at [eval]-wrapper:6:22 {
  code: 'ERR_DLOPEN_FAILED'
}

Node.js v18.12.1
Error: Error importing BLST native bindings: Command failed: node -e 'require("/usr/src/lodestar/node_modules/@chainsafe/blst/prebuild/linux-x64-108-binding.node")'
node:internal/modules/cjs/loader:1243
  return process.dlopen(module, path.toNamespacedPath(filename));
                 ^

Error: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /usr/src/lodestar/node_modules/@chainsafe/blst/prebuild/linux-x64-108-binding.node)
    at Module._extensions..node (node:internal/modules/cjs/loader:1243:18)
    at Module.load (node:internal/modules/cjs/loader:1037:32)
    at Module._load (node:internal/modules/cjs/loader:878:12)
    at Module.require (node:internal/modules/cjs/loader:1061:19)
    at require (node:internal/modules/cjs/helpers:103:18)
    at [eval]:1:1
    at Script.runInThisContext (node:vm:129:12)
    at Object.runInThisContext (node:vm:313:38)
    at node:internal/process/execution:79:19
    at [eval]-wrapper:6:22 {
  code: 'ERR_DLOPEN_FAILED'
}

Node.js v18.12.1

    at checkExecSyncError (node:child_process:871:11)
    at Object.execSync (node:child_process:943:15)
    at Object.testBindings (/usr/src/lodestar/node_modules/@chainsafe/blst/dist/scripts/testBindings.js:8:21)
    at Object.downloadBindings (/usr/src/lodestar/node_modules/@chainsafe/blst/dist/scripts/downloadBindings.js:11:26)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async install (/usr/src/lodestar/node_modules/@chainsafe/blst/dist/scripts/install.js:34:9) {
  status: 1,
  signal: null,
  output: [
    null,
    <Buffer >,
    <Buffer 6e 6f 64 65 3a 69 6e 74 65 72 6e 61 6c 2f 6d 6f 64 75 6c 65 73 2f 63 6a 73 2f 6c 6f 61 64 65 72 3a 31 32 34 33 0a 20 20 72 65 74 75 72 6e 20 70 72 6f ... 800 more bytes>
  ],
  pid: 2164487,
  stdout: <Buffer >,
  stderr: <Buffer 6e 6f 64 65 3a 69 6e 74 65 72 6e 61 6c 2f 6d 6f 64 75 6c 65 73 2f 63 6a 73 2f 6c 6f 61 64 65 72 3a 31 32 34 33 0a 20 20 72 65 74 75 72 6e 20 70 72 6f ... 800 more bytes>
}
Building BLST native bindings /usr/src/lodestar/node_modules/@chainsafe/blst/prebuild/linux-x64-108-binding.node from source...
BLST_WRAP_CPP_PREBUILD /usr/src/lodestar/node_modules/@chainsafe/blst/prebuild/blst_wrap.cpp exists, SWIG will be skipped
Launching node-gyp {
  nodeJsExec: '/home/devops/.nvm/versions/node/v18.12.1/bin/node',
  nodeGypExec: '/usr/src/lodestar/node_modules/@chainsafe/blst/node_modules/node-gyp/bin/node-gyp.js',
  cwd: '/usr/src/lodestar/node_modules/@chainsafe/blst/blst/bindings/node.js',
  BLST_WRAP_CPP_PREBUILD: '/usr/src/lodestar/node_modules/@chainsafe/blst/prebuild/blst_wrap.cpp'
}
gyp info it worked if it ends with ok
gyp info using node-gyp@8.4.1
gyp info using node@18.12.1 | linux | x64
gyp info find Python using Python version 3.8.10 found at "/usr/bin/python3"
gyp info spawn /usr/bin/python3
gyp info spawn args [
gyp info spawn args   '/usr/src/lodestar/node_modules/@chainsafe/blst/node_modules/node-gyp/gyp/gyp_main.py',
gyp info spawn args   'binding.gyp',
gyp info spawn args   '-f',
gyp info spawn args   'make',
gyp info spawn args   '-I',
gyp info spawn args   '/usr/src/lodestar/node_modules/@chainsafe/blst/blst/bindings/node.js/build/config.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/usr/src/lodestar/node_modules/@chainsafe/blst/node_modules/node-gyp/addon.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/home/devops/.cache/node-gyp/18.12.1/include/node/common.gypi',
gyp info spawn args   '-Dlibrary=shared_library',
gyp info spawn args   '-Dvisibility=default',
gyp info spawn args   '-Dnode_root_dir=/home/devops/.cache/node-gyp/18.12.1',
gyp info spawn args   '-Dnode_gyp_dir=/usr/src/lodestar/node_modules/@chainsafe/blst/node_modules/node-gyp',
gyp info spawn args   '-Dnode_lib_file=/home/devops/.cache/node-gyp/18.12.1/<(target_arch)/node.lib',
gyp info spawn args   '-Dmodule_root_dir=/usr/src/lodestar/node_modules/@chainsafe/blst/blst/bindings/node.js',
gyp info spawn args   '-Dnode_engine=v8',
gyp info spawn args   '--depth=.',
gyp info spawn args   '--no-parallel',
gyp info spawn args   '--generator-output',
gyp info spawn args   'build',
gyp info spawn args   '-Goutput_dir=.'
gyp info spawn args ]
gyp info spawn make
gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build' ]
make: Entering directory '/usr/src/lodestar/node_modules/@chainsafe/blst/blst/bindings/node.js/build'
  ACTION binding_gyp_blst_target_blst_wrap Release/obj.target/blst/geni/blst_wrap.cpp
/bin/sh: 1: python: not found
make: *** [blst.target.mk:13: Release/obj.target/blst/geni/blst_wrap.cpp] Error 127
gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/src/lodestar/node_modules/@chainsafe/blst/node_modules/node-gyp/lib/build.js:194:23)
gyp ERR! stack     at ChildProcess.emit (node:events:513:28)
gyp ERR! stack     at ChildProcess._handle.onexit (node:internal/child_process:291:12)
gyp ERR! System Linux 5.4.0-62-generic
gyp ERR! command "/home/devops/.nvm/versions/node/v18.12.1/bin/node" "/usr/src/lodestar/node_modules/@chainsafe/blst/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /usr/src/lodestar/node_modules/@chainsafe/blst/blst/bindings/node.js
gyp ERR! node -v v18.12.1
gyp ERR! node-gyp -v v8.4.1
gyp ERR! not ok
make: Leaving directory '/usr/src/lodestar/node_modules/@chainsafe/blst/blst/bindings/node.js/build'
Error: Error building BLST native bindings: Command failed: /home/devops/.nvm/versions/node/v18.12.1/bin/node /usr/src/lodestar/node_modules/@chainsafe/blst/node_modules/node-gyp/bin/node-gyp.js rebuild
gyp info it worked if it ends with ok
gyp info using node-gyp@8.4.1
gyp info using node@18.12.1 | linux | x64
gyp info find Python using Python version 3.8.10 found at "/usr/bin/python3"
gyp info spawn /usr/bin/python3
gyp info spawn args [
gyp info spawn args   '/usr/src/lodestar/node_modules/@chainsafe/blst/node_modules/node-gyp/gyp/gyp_main.py',
gyp info spawn args   'binding.gyp',
gyp info spawn args   '-f',
gyp info spawn args   'make',
gyp info spawn args   '-I',
gyp info spawn args   '/usr/src/lodestar/node_modules/@chainsafe/blst/blst/bindings/node.js/build/config.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/usr/src/lodestar/node_modules/@chainsafe/blst/node_modules/node-gyp/addon.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/home/devops/.cache/node-gyp/18.12.1/include/node/common.gypi',
gyp info spawn args   '-Dlibrary=shared_library',
gyp info spawn args   '-Dvisibility=default',
gyp info spawn args   '-Dnode_root_dir=/home/devops/.cache/node-gyp/18.12.1',
gyp info spawn args   '-Dnode_gyp_dir=/usr/src/lodestar/node_modules/@chainsafe/blst/node_modules/node-gyp',
gyp info spawn args   '-Dnode_lib_file=/home/devops/.cache/node-gyp/18.12.1/<(target_arch)/node.lib',
gyp info spawn args   '-Dmodule_root_dir=/usr/src/lodestar/node_modules/@chainsafe/blst/blst/bindings/node.js',
gyp info spawn args   '-Dnode_engine=v8',
gyp info spawn args   '--depth=.',
gyp info spawn args   '--no-parallel',
gyp info spawn args   '--generator-output',
gyp info spawn args   'build',
gyp info spawn args   '-Goutput_dir=.'
gyp info spawn args ]
gyp info spawn make
gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build' ]
/bin/sh: 1: python: not found
make: *** [blst.target.mk:13: Release/obj.target/blst/geni/blst_wrap.cpp] Error 127
gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/src/lodestar/node_modules/@chainsafe/blst/node_modules/node-gyp/lib/build.js:194:23)
gyp ERR! stack     at ChildProcess.emit (node:events:513:28)
gyp ERR! stack     at ChildProcess._handle.onexit (node:internal/child_process:291:12)
gyp ERR! System Linux 5.4.0-62-generic
gyp ERR! command "/home/devops/.nvm/versions/node/v18.12.1/bin/node" "/usr/src/lodestar/node_modules/@chainsafe/blst/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /usr/src/lodestar/node_modules/@chainsafe/blst/blst/bindings/node.js
gyp ERR! node -v v18.12.1
gyp ERR! node-gyp -v v8.4.1
gyp ERR! not ok

    at ChildProcess.exithandler (node:child_process:412:12)
    at ChildProcess.emit (node:events:513:28)
    at maybeClose (node:internal/child_process:1091:16)
    at ChildProcess._handle.onexit (node:internal/child_process:302:5) {
  code: 1,
  killed: false,
  signal: null,
  cmd: '/home/devops/.nvm/versions/node/v18.12.1/bin/node /usr/src/lodestar/node_modules/@chainsafe/blst/node_modules/node-gyp/bin/node-gyp.js rebuild'
}
Error: Error downloading and building BLST native bindings /usr/src/lodestar/node_modules/@chainsafe/blst/prebuild/lin

Expected behavior

No error

twoeths commented 1 year ago

This issue does not always happen on contabo vpss machines.

The error is versionGLIBC_2.32' not found`:

ldd --version
ldd (Ubuntu GLIBC 2.31-0ubuntu9.2) 2.31
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

but other success nodes have same GLIBC (2.31) version

dapplion commented 1 year ago

Maybe those contabo VPSS are running an old version of linux?

twoeths commented 1 year ago

Maybe those contabo VPSS are running an old version of linux?

@dapplion no, it's ubuntu 20.04.3, some other nodes with same OS worked

Update: confirm that v0.2.7 was built with ubuntu v22 while the node has Ubuntu v20

twoeths commented 1 year ago

there's an issue with @chainsafe/blst 0.2.7

twoeths commented 1 year ago

it works well if I use v1.3.x with @chainsafe/blst 0.2.6, it seems that when releasing blst we always go with latest ubuntu and it make it incompatible for not-upgraded machines

see https://github.com/ChainSafe/blst-ts/actions/runs/3706833809/jobs/6285801787

Ubuntu
  22.0[4](https://github.com/ChainSafe/blst-ts/actions/runs/3706833809/jobs/6285801787#step:1:4).1
  LTS

not able to get github action log of v0.2.6 https://github.com/ChainSafe/blst-ts/actions/runs/3001398317/jobs/4817458661 since it's expired

dapplion commented 1 year ago

Then the pre-built files should be tagged better to ensure are built for the correct environment

twoeths commented 1 year ago

found the difference between v20 ubuntu nodes

also not sure if it's related to https://github.com/ChainSafe/blst-ts/issues/77

dapplion commented 1 year ago

blst-ts v0.2.7 has prebuilds up to modules version 108 = linux-x64-108-binding.node, while v0.2.4 has up to modules version 102 linux-x64-102-binding.node. That means that using v0.2.4 with an older NodeJS version will always trigger a new build.

You can check which modules version is used by each node version with

$ node -e "console.log(process.version, process.versions.modules)"
v18.12.1 108

The modules version is the C++ ABI of NodeJS. I found the mapping of ABI to NodeJS version here https://github.com/electron/node-abi/blob/main/abi_registry.json

twoeths commented 1 year ago

@dapplion v0.2.6 has prebuilds up to modules version 108 too and it works fine

twoeths commented 1 year ago

on the error node, it failed to testBinding (with child process execSync, the difference between v0.2.6 and v0.2.7), that's why it did an installation

Node.js v18.12.1
Cached BLST native bindings not OK
Retrieving BLST native bindings /usr/src/lodestar/node_modules/@chainsafe/blst/prebuild/linux-x64-108-binding.node ...
node:internal/modules/cjs/loader:1243
  return process.dlopen(module, path.toNamespacedPath(filename));
                 ^

on the successfully node, if I run the install script, it's a success so no need to do the installation

node scripts/install.js
Using existing BLST native bindings from /usr/src/lodestar/node_modules/@chainsafe/blst/prebuild/linux-x64-108-binding.node

this really links to the child process invocation we added to v0.2.7 https://github.com/ChainSafe/blst-ts/pull/76

twoeths commented 1 year ago

could be fixed by https://github.com/ChainSafe/blst-ts/pull/79

philknows commented 1 year ago

@dapplion v0.2.6 has prebuilds up to modules version 108 too and it works fine

Does this include arm64 devices? I remember trying to build v0.2.6 on an arm64 device and was still getting the SIGSEGV exit signal, which is why https://github.com/ChainSafe/blst-ts/pull/76 was made.

twoeths commented 1 year ago

@philknows v0.2.8 is same to v0.2.7 but we use ubuntu 20.04 instead of ubuntu 22.04 to generate binding files