prestonvanloon
commented
4 years ago You need to have cmake installed on your system to build the c++ sources. We should update the docs as well to reflect this requirement.
Closed handelaar2 closed 4 years ago
prestonvanloon
commented
4 years ago You need to have cmake installed on your system to build the c++ sources. We should update the docs as well to reflect this requirement.
handelaar2
commented
4 years ago I already figured that cmake was missing but I installed it from the snap store. Maybe that was the issue. $sudo snap install cmake --classic $which make => /snap/bin/cmake $cmake -version => cmake version 3.16.1 CMake suite maintained and supported by Kitware..... This morning I installed cmake from the tar. Now it says: $which cmake" => /usr/local/bin/cmake (I guess this is the right cmake)
I started again with a "bazel clean" to start 'new build'. And this is the result of 'bazel build //beacon-chain:beacon-chain' . Now I get a different error. Something else missing:
$ bazel build //beacon-chain:beacon-chain INFO: Analyzed target //beacon-chain:beacon-chain (0 packages loaded, 0 targets configured). INFO: Found 1 target... ERROR: /home/ubuntu/prysm/beacon-chain/BUILD.bazel:80:1: GoLink beacon-chain/linux_amd64_stripped/beacon-chain failed (Exit 1) builder failed: error executing command bazel-out/host/bin/external/go_sdk/builder '-param=bazel-out/k8-fastbuild/bin/beacon-chain/linux_amd64_stripped/beacon-chain-0.params' -- -extld /usr/bin/gcc '-buildid=redacted' -w -extldflags ... (remaining 1 argument(s) skipped)
Use --sandbox_debug to see verbose messages from the sandbox link: warning: package "github.com/golang/protobuf/protoc-gen-go/descriptor" is provided by more than one rule: @com_github_golang_protobuf//protoc-gen-go/descriptor:go_default_library @io_bazel_rules_go//proto/wkt:descriptor_go_proto Set "importmap" to different paths in each library. This will be an error in the future. link: warning: package "github.com/golang/protobuf/ptypes/empty" is provided by more than one rule: @com_github_golang_protobuf//ptypes/empty:go_default_library @io_bazel_rules_go//proto/wkt:empty_go_proto Set "importmap" to different paths in each library. This will be an error in the future. link: warning: package "github.com/golang/protobuf/descriptor" is provided by more than one rule: @com_github_golang_protobuf//descriptor:go_default_library_gen @com_github_golang_protobuf//descriptor:go_default_library Set "importmap" to different paths in each library. This will be an error in the future. fatal error: runtime: out of memory
runtime stack: runtime.throw(0x698de9, 0x16) /usr/local/go/src/runtime/panic.go:774 +0x72 runtime.sysMap(0xc014000000, 0x4000000, 0x877c38) /usr/local/go/src/runtime/mem_linux.go:169 +0xc5 runtime.(mheap).sysAlloc(0x860120, 0x6000, 0x2000, 0x41421f) /usr/local/go/src/runtime/malloc.go:701 +0x1cd runtime.(mheap).grow(0x860120, 0x3, 0xffffffff) /usr/local/go/src/runtime/mheap.go:1255 +0xa3 runtime.(mheap).allocSpanLocked(0x860120, 0x3, 0x877c48, 0x7f25efd06900) /usr/local/go/src/runtime/mheap.go:1170 +0x266 runtime.(mheap).alloc_m(0x860120, 0x3, 0x45005c, 0x7f25efd06900) /usr/local/go/src/runtime/mheap.go:1022 +0xc2 runtime.(*mheap).alloc.func1() /usr/local/go/src/runtime/mheap.go:1093 +0x4c runtime.systemstack(0x0) /usr/local/go/src/runtime/asm_amd64.s:370 +0x66 runtime.mstart() /usr/local/go/src/runtime/proc.go:1146
goroutine 1 [running]: runtime.systemstack_switch() /usr/local/go/src/runtime/asm_amd64.s:330 fp=0xc0007e2bb0 sp=0xc0007e2ba8 pc=0x4585c0 runtime.(mheap).alloc(0x860120, 0x3, 0x1005c, 0x6a4dd0) /usr/local/go/src/runtime/mheap.go:1092 +0x8a fp=0xc0007e2c00 sp=0xc0007e2bb0 pc=0x42546a runtime.(mcentral).grow(0x861bd8, 0x0) /usr/local/go/src/runtime/mcentral.go:255 +0x7b fp=0xc0007e2c40 sp=0xc0007e2c00 pc=0x4173bb runtime.(mcentral).cacheSpan(0x861bd8, 0x0) /usr/local/go/src/runtime/mcentral.go:106 +0x2fe fp=0xc0007e2ca0 sp=0xc0007e2c40 pc=0x416ede runtime.(mcache).refill(0x7f25fb7b7008, 0x5c) /usr/local/go/src/runtime/mcache.go:138 +0x85 fp=0xc0007e2cc0 sp=0xc0007e2ca0 pc=0x416985 runtime.(mcache).nextFree(0x7f25fb7b7008, 0x4c945c, 0x4, 0x20, 0xc013ff0620) /usr/local/go/src/runtime/malloc.go:854 +0x87 fp=0xc0007e2cf8 sp=0xc0007e2cc0 pc=0x40b7e7 runtime.mallocgc(0x1800, 0x68b800, 0x1, 0x0) /usr/local/go/src/runtime/malloc.go:1022 +0x793 fp=0xc0007e2d98 sp=0xc0007e2cf8 pc=0x40c123 runtime.growslice(0x68b800, 0xc00643f9c8, 0x157, 0x157, 0x158, 0xc013f57f20, 0xc013fe3500, 0x90) /usr/local/go/src/runtime/slice.go:181 +0x1e2 fp=0xc0007e2e00 sp=0xc0007e2d98 pc=0x443a72 cmd/link/internal/ld.(deadcodepass).mark(0xc0007e3278, 0xc00a10dfc0, 0xc00a10d990) /usr/local/go/src/cmd/link/internal/ld/deadcode.go:196 +0x1bd fp=0xc0007e2e88 sp=0xc0007e2e00 pc=0x56875d cmd/link/internal/ld.(*deadcodepass).flood(0xc0007e3278) /usr/local/go/src/cmd/link/internal/ld/deadcode.go:353 +0x4fc fp=0xc0007e30a0 sp=0xc0007e2e88 pc=0x56961c cmd/link/internal/ld.deadcode(0xc000698000) /usr/local/go/src/cmd/link/internal/ld/deadcode.go:61 +0xe3 fp=0xc0007e32d0 sp=0xc0007e30a0 pc=0x567873 cmd/link/internal/ld.Main(0x84bdc0, 0x10, 0x20, 0x1, 0x7, 0x10, 0x69ac6c, 0x1b, 0x697753, 0x14, ...) /usr/local/go/src/cmd/link/internal/ld/main.go:211 +0xb81 fp=0xc0007e3428 sp=0xc0007e32d0 pc=0x5ab3d1 main.main() /usr/local/go/src/cmd/link/main.go:65 +0x1d6 fp=0xc0007e3f60 sp=0xc0007e3428 pc=0x614616 runtime.main() /usr/local/go/src/runtime/proc.go:203 +0x21e fp=0xc0007e3fe0 sp=0xc0007e3f60 pc=0x42f5de runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0007e3fe8 sp=0xc0007e3fe0 pc=0x45a511 link: error running subcommand: exit status 2 Target //beacon-chain:beacon-chain failed to build Use --verbose_failures to see the command lines of failed build steps. INFO: Elapsed time: 29.655s, Critical Path: 27.56s INFO: 0 processes. FAILED: Build did NOT complete successfully
prestonvanloon
commented
4 years ago Looks like the process ran out of memory when building .
What are your system specs? How much RAM do you have?
handelaar2
commented
4 years ago Oops, I overlooked the "out of memory" message in the log. Sorry. Running the t2.micro in below overview, so 1GiB. (in fact 'free -m' says total=983)
Any suggestions on the minimal size in above list to build (and run) the beacon+validator.
(README.md does not mention minimal requirements)
prestonvanloon
commented
4 years ago Maybe 2gb is a safe minimum. Here’s an article that may help too https://docs.bazel.build/versions/master/memory-saving-mode.html
handelaar2
commented
4 years ago Those 'memory saving' bazel parameters did not bring relief for the 1gb image. The out-of-memory error continued. So I fired up a new instance, now with the 2gb option. This time both beacon and validator compiled!
The beacon is currently syncing at ~25 blocks per sec. (commit fd93751b (Dec15 after Faucet fix) ) INFO initial-sync: Processing block 28869/249258 - estimated time remaining 2h26m20s blocksPerSecond=25.1 peers=5/9
I had to install the following on top of the Amazon-EC2 Ubuntu image to make it work:
Install bazel (including curl) sudo apt install curl curl https://bazel.build/bazel-release.pub.gpg | sudo apt-key add - echo "deb [arch=amd64] https://storage.googleapis.com/bazel-apt stable jdk1.8" | sudo tee /etc/apt/sources.list.d/bazel.list sudo apt update && sudo apt install bazel sudo apt update && sudo apt full-upgrade
Install make (needed for cmake) sudo apt-get install build-essential
Install openssl (needed for cmake) sudo apt install libssl-dev
Install cmake wget https://github.com/Kitware/CMake/releases/download/v3.16.1/cmake-3.16.1.tar.gz tar -zxvf cmake-3.16.1.tar.gz cd cmake-3.16.1 ./bootstrap make make install
Install prysm cd $HOME git clone https://github.com/prysmaticlabs/prysm cd prysm bazel build //beacon-chain:beacon-chain bazel build //validator:validator
I guess I also have to install golang, as I see some info messages that it cannot find a go-source.
Edit: the standard disk space of these EC2 instances (8G) is not enough to hold the block db (out-of-space during sync). After increasing the volume on the amazon dashboard, one also have to run: lsblk (to check) sudo growpart /dev/xvda 1 (increase partition) sudo resize2fs /dev/xvda1 (increase filesystem) df -h (to check)
Running via Bazel gives quite some overhead. After the build, I run beacon and validator like this: run beacon: cd $HOME ./prysm/bazel-bin/beacon-chain/linux_amd64_stripped/beacon-chain --datadir=$HOME/prysm-data
create validator(s) - only when needed cd $HOME ./prysm/bazel-bin/validator/linux_amd64_stripped/validator accounts create --password="changeme" --keystore-path=$HOME/prysm-data
run validator cd $HOME ./prysm/bazel-bin/validator/linux_amd64_stripped/validator --keystore-path=$HOME/prysm-data --password="changeme"
rauljordan
commented
4 years ago This is very useful @handelaar2, instead of closing the issue, maybe we can add this to the README on suggestions for how to run on a standard Ubuntu EC2 instance for other users with your situation. @har00ga maybe we can add to our top-level README? We can also add to Gitbook
har00ga
commented
4 years ago For sure, definitely worth adding. Updating the README to match changes in the Gitbook was already on my agenda for the afternoon, so this suggestion has good timing. :slightly_smiling_face:
prestonvanloon
commented
4 years ago Thanks for such a detailed resolution @handelaar2. You don't need go installed on your system, it is provided via bazel. I'd also like to see cmake be provided from bazel as well, but we'll have to address that at a later time.
Closing this. We need to add cmake as an extra system dependency for now.
Edit: the cmake native toolchain is provided here: https://github.com/prysmaticlabs/prysm/blob/d3375d98a8849cb37032af0c36b1046cc70da01e/BUILD.bazel#L141-L145
prestonvanloon
commented
4 years ago If anyone else runs into this issue and the above tips do not work, we've added a build flag in #4309 to exclude Kafka from the build.
Try using --define kafka_enabled=false in your bazel build for beacon chain targets. This flag is not needed for validator client.
yashbhavsar007
commented
4 years ago I tried by using using --define kafka_enabled=false but i am getting this error. ERROR: /home/ubuntu/.cache/bazel/_bazel_ubuntu/0541ece9377351509275eb8affcf50ed/external/com_github_herumi_bls_eth_go_binary/bls/BUILD.bazel:38:1: GoCompilePkg external/com_github_herumi_bls_eth_go_binary/bls/linux_amd64_stripped/go_default_library%/github.com/herumi/bls-eth-go-binary/bls.a failed (Exit 1) builder failed: error executing command bazel-out/host/bin/external/go_sdk/builder compilepkg -sdk external/go_sdk -installsuffix linux_amd64 -src external/com_github_herumi_bls_eth_go_binary/bls/bls.go -src ... (remaining 43 argument(s) skipped)
Use --sandbox_debug to see verbose messages from the sandbox compilepkg: error running subcommand: exit status 2 bls.go:676:105: cannot use _cgo2 (type *[40]_Ctype_uchar) as type unsafe.Pointer in argument to _Cfunc_blsVerifyAggregatedHashWithDomain Target //beacon-chain:beacon-chain failed to build Use --verbose_failures to see the command lines of failed build steps. INFO: Elapsed time: 14.990s, Critical Path: 2.80s INFO: 110 processes: 110 linux-sandbox. FAILED: Build did NOT complete successfully
I am on AWS ec2 t2.large instance with 8 Gib of memory. Bazel version : bazel 2.0.0 go version go1.12.9 linux/amd64
Fails: bazel build //beacon-chain:beacon-chain bazel version 1.2.1
DEBUG: Rule 'io_bazel_rules_docker' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "14ac30773fdb393ddec90e158c9ec7ebb3f8a4fd533ec2abbfd8789ad81a284b" DEBUG: Call stack for the definition of repository 'io_bazel_rules_docker' which is a http_archive (rule definition at /home/ubuntu/.cache/bazel/_bazel_ubuntu/c9cb51ba1d0dce1f4433e7d178375968/external/bazel_tools/tools/build_defs/repo/http.bzl:292:16):
Use --sandbox_debug to see verbose messages from the sandbox
rules_foreign_cc: Build failed! rules_foreign_cc: Keeping temp build directory and dependencies directory for debug. rules_foreign_cc: Please note that the directories inside a sandbox are still cleaned unless you specify '--sandbox_debug' Bazel command line flag.
rules_foreign_cc: Printing build logs:
BEGIN BUILD LOGS Bazel external C/C++ Rules #0.0.8. Building library 'librdkafka' Environment:__EXT_BUILD_ROOT=/home/ubuntu/.cache/bazel/_bazel_ubuntu/c9cb51ba1d0dce1f4433e7d178375968/sandbox/linux-sandbox/725/execroot/prysm INSTALLDIR=/home/ubuntu/.cache/bazel/_bazel_ubuntu/c9cb51ba1d0dce1f4433e7d178375968/sandbox/linux-sandbox/725/execroot/prysm/bazel-out/k8-fastbuild/bin/external/prysm/third_party/kafka/librdkafka PWD=/home/ubuntu/.cache/bazel/_bazel_ubuntu/c9cb51ba1d0dce1f4433e7d178375968/sandbox/linux-sandbox/725/execroot/prysm BUILD_TMPDIR=/tmp/tmp.yLK2T2Aw7o TMPDIR=/tmp EXT_BUILD_DEPS=/tmp/tmp.aTv5crjCVE SHLVL=2 BUILD_LOG=bazel-out/k8-fastbuild/bin/external/prysm/third_party/kafka/librdkafka/logs/CMake.log BUILD_SCRIPT=bazel-out/k8-fastbuild/bin/external/prysm/third_party/kafka/librdkafka/logs/CMake_script.sh PATH=/home/ubuntu/.cache/bazel/_bazelubuntu/c9cb51ba1d0dce1f4433e7d178375968/sandbox/linux-sandbox/725/execroot/prysm:/bin:/usr/bin:/usr/local/bin =/usr/bin/env __bazel-out/k8-fastbuild/bin/external/prysm/third_party/kafka/librdkafka/logs/CMake_script.sh: line 51: /tmp/tmp.aTv5crjCVE/bin/cmake/bin/cmake: No such file or directory
END BUILD LOGS rules_foreign_cc: Build script location: bazel-out/k8-fastbuild/bin/external/prysm/third_party/kafka/librdkafka/logs/CMake_script.sh rules_foreign_cc: Build log location: bazel-out/k8-fastbuild/bin/external/prysm/third_party/kafka/librdkafka/logs/CMake.log
Target //beacon-chain:beacon-chain failed to build Use --verbose_failures to see the command lines of failed build steps. ERROR: /home/ubuntu/prysm/beacon-chain/BUILD.bazel:80:1 error executing shell command: '/bin/bash -c function cleanup_function() { local ecode=$? if [ $ecode -eq 0 ]; then cleanup_on_success else cleanup_on_failure fi } set -e function cleanup_on_success() { printf "rules_foreign_cc: ...' failed (Exit 127) bash failed: error executing command /bin/bash -c ... (remaining 1 argument(s) skipped)
Use --sandbox_debug to see verbose messages from the sandbox INFO: Elapsed time: 0.780s, Critical Path: 0.04s INFO: 1 process: 1 linux-sandbox. FAILED: Build did NOT complete successfully