Rebar3/erlang issues - Githubissues

lhendre commented 2 years ago

Trying to compile/build the project but i'm getting the error below. It saying I need to recompile with Erlang 25, but that is the current version I am on. Looking into if it is another dependency issue currently.

 ./rebar3 as prod tar
===> Verifying dependencies...
+ mkdir -p _build/cmake
+ cd _build/cmake
+ type cmake3
+ CMAKE=cmake
+ cmake ../../c_src
-- cmake args -DCMAKE_POSITION_INDEPENDENT_CODE=ON;-DCMAKE_CXX_COMPILER=/Library/Developer/CommandLineTools/usr/bin/c++;-DCMAKE_AR=/Library/Developer/CommandLineTools/usr/bin/ar;-DCMAKE_BUILD_TYPE=Release;-DFAIL_ON_WARNINGS=OFF;-DPORTABLE=ON;-DUSE_RTTI=1;-DWITH_TOOLS=OFF;-DWITH_TESTS=OFF;-DWITH_GFLAGS=OFF;-DWITH_JEMALLOC=OFF;-DWITH_TBB=;-DWITH_SNAPPY=;-DWITH_LZ4=;-DWITH_ZLIB=;-DWITH_BZ2=;-DWITH_ZSTD=
-- Found RocksDB library: /Users/lucashendren/workspace/arweave/ar2/arweave/_build/default/lib/rocksdb/_build/cmake/rocksdb-prefix/src/rocksdb-build/librocksdb.a
-- Found RocksDB includes: /Users/lucashendren/workspace/arweave/ar2/arweave/_build/default/lib/rocksdb/c_src/../deps/rocksdb/include
-- Using OTP lib: /usr/local/Cellar/erlang/25.0/lib/erlang/lib - found
-- Using erl_interface version: erl_interface-5.3
-- Using erts version: erts-13.0
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/lucashendren/workspace/arweave/ar2/arweave/_build/default/lib/rocksdb/_build/cmake
+ echo done.
done.
make: `/Users/lucashendren/workspace/arweave/ar2/arweave/_build/default/lib/prometheus_process_collector/c_src/../priv/prometheus_process_collector.so' is up to date.
=ERROR REPORT==== 22-May-2022::22:23:40.929654 ===
beam/beam_load.c(148): Error loading module enc:
  please re-compile this module with an Erlang/OTP 25 compiler

escript: exception error: undefined function enc:main/1
  in function  escript:run/2 (escript.erl, line 750)
  in call from escript:start/1 (escript.erl, line 277)
  in call from init:start_em/1 
  in call from init:do_boot/3 
===> Hook for compile failed!

ldmberman commented 2 years ago

Hi @lhendre, try to remove _build and recompile.

lhendre commented 2 years ago

@ldmberman removed the _build directory and still having issues, will keep exploring

ThomasBlock commented 2 years ago

Hi @ldmberman . Thank you for the other reply. Following that i have to compile arweave for the first time. I have linux experience, but no erlang experience. i see the same error as described here. what could be wrong? here my commandline with a fresh Ubuntu20.04 VM:

echo "deb https://packages.erlang-solutions.com/ubuntu $(lsb_release -cs) contrib" | sudo tee /etc/apt/sources.list.d/erlang.list
curl -fsSL https://packages.erlang-solutions.com/ubuntu/erlang_solutions.asc | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/erlang.gpg
apt update && apt upgrade -y
apt install build-essential erlang git libsqlite3-dev libgmp-dev curl cmake
erl
Erlang/OTP 25 [erts-13.0.3] [source] [64-bit] [smp:24:10] [ds:24:10:10] [async-threads:1] [jit:ns]

git clone --recursive https://github.com/ArweaveTeam/arweave.git
cd arweave
./rebar3 as prod tar
===> Verifying dependencies...
===> Fetching b64fast (from {git,"https://github.com/ArweaveTeam/b64fast.git",
                   {ref,"a0ef55ec66ecf705848716c195bf45665f78818a"}})
===> Fetching pc (from {git,"https://github.com/blt/port_compiler.git",{tag,"v1.12.0"}})
===> Analyzing applications...
===> Compiling pc
===> Fetching cowboy v2.9.0
===> Fetching graphql (from {git,"https://github.com/shopgun/graphql-erlang.git",
                   {ref,"fc76a2df7d1ddcf8bcc3bb46363e7ca8fca80939"}})
===> Fetching geas_rebar3 (from {git,"https://github.com/crownedgrouse/geas_rebar3.git",
                       {branch,"master"}})
===> Fetching geas v2.7.9
===> Fetching samovar v1.0.2
===> Analyzing applications...
===> Compiling samovar
===> Compiling geas
===> Compiling geas_rebar3
===> Fetching rebar3_hex v7.0.2
===> Fetching hex_core v0.8.4
===> Fetching verl v1.1.1
===> Analyzing applications...
===> Compiling verl
===> Compiling hex_core
===> Compiling rebar3_hex
===> Fetching gun v1.3.2
===> Fetching jiffy v1.0.8
===> Fetching prometheus v4.6.0
===> Fetching rebar3_archive_plugin v0.0.1
===> Fetching rebar3_elvis_plugin (from {git,"https://github.com/deadtrickster/rebar3_elvis_plugin.git",
                               "master"})
===> WARNING: It is recommended to use {branch, Name}, {tag, Tag} or {ref, Ref}, otherwise updating the dep may not work as expected.
===> Fetching katana_code v0.1.0
===> Fetching zipper v1.0.1
===> Fetching aleppo v1.0.0
===> Analyzing applications...
===> Compiling zipper
===> Compiling aleppo
_build/default/plugins/aleppo/src/aleppo.erl:6:2: Warning: record ale_context has field(s) without type information
_build/default/plugins/aleppo/src/aleppo.erl:12:1: Warning: missing specification for function process_file/1
_build/default/plugins/aleppo/src/aleppo.erl:15:1: Warning: missing specification for function process_file/2
_build/default/plugins/aleppo/src/aleppo.erl:25:1: Warning: missing specification for function process_tokens/1
_build/default/plugins/aleppo/src/aleppo.erl:32:1: Warning: missing specification for function process_tokens/2
_build/default/plugins/aleppo/src/aleppo.erl:307:1: Warning: missing specification for function scan_file/1

===> Compiling katana_code
===> Compiling rebar3_elvis_plugin
===> Analyzing applications...
===> Compiling rebar3_archive_plugin
===> Fetching prometheus_cowboy v0.1.8
===> Fetching prometheus_process_collector v1.6.0
===> Fetching rocksdb (from {git,"https://gitlab.com/arweave1/erlang-rocksdb.git",
                   {ref,"ed4d05d58d174485f883b5cd3e057c64d9e7ff3a"}})
===> Fetching cowlib v2.8.0
===> Fetching prometheus_httpd v2.1.11
===> Fetching ranch v1.8.0
===> Fetching accept v0.3.5
+ mkdir -p _build/cmake
+ cd _build/cmake
+ type cmake3
+ CMAKE=cmake
+ cmake ../../c_src
-- The C compiler identification is GNU 9.4.0
-- The CXX compiler identification is GNU 9.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- cmake args -DCMAKE_POSITION_INDEPENDENT_CODE=ON;-DCMAKE_CXX_COMPILER=/usr/bin/c++;-DCMAKE_AR=/usr/bin/ar;-DCMAKE_BUILD_TYPE=Release;-DFAIL_ON_WARNINGS=OFF;-DPORTABLE=ON;-DUSE_RTTI=1;-DWITH_TOOLS=OFF;-DWITH_TESTS=OFF;-DWITH_GFLAGS=OFF;-DWITH_JEMALLOC=OFF;-DWITH_TBB=;-DWITH_SNAPPY=;-DWITH_LZ4=;-DWITH_ZLIB=;-DWITH_BZ2=;-DWITH_ZSTD=
-- Found RocksDB library: /root/arweave/_build/default/lib/rocksdb/_build/cmake/rocksdb-prefix/src/rocksdb-build/librocksdb.a
-- Found RocksDB includes: /root/arweave/_build/default/lib/rocksdb/c_src/../deps/rocksdb/include
-- Using OTP lib: /usr/lib/erlang/lib - found
-- Using erl_interface version: erl_interface-5.3
-- Using erts version: erts-13.0.3
-- Performing Test HAVE_OMIT_LEAF_FRAME_POINTER
-- Performing Test HAVE_OMIT_LEAF_FRAME_POINTER - Success
-- Configuring done
-- Generating done
-- Build files have been written to: /root/arweave/_build/default/lib/rocksdb/_build/cmake
+ echo done.
done.
make: Entering directory '/root/arweave/_build/default/lib/prometheus_process_collector/c_src'
g++ -O3 -finline-functions -fPIC -I /usr/lib/erlang/erts-13.0.3/include/ -I /usr/lib/erlang/lib/erl_interface-5.3/include -std=c++11 -Wall  -c -o prometheus_process_collector_nif.o prometheus_process_collector_nif.cc
g++ -O3 -finline-functions -fPIC -I /usr/lib/erlang/erts-13.0.3/include/ -I /usr/lib/erlang/lib/erl_interface-5.3/include -std=c++11 -Wall  -c -o prometheus_process_info_linux.o prometheus_process_info_linux.cc
cc prometheus_process_collector_nif.o prometheus_process_info_linux.o -shared -L /usr/lib/erlang/lib/erl_interface-5.3/lib -lei -lstdc++ -o /root/arweave/_build/default/lib/prometheus_process_collector/c_src/../priv/prometheus_process_collector.so
make: Leaving directory '/root/arweave/_build/default/lib/prometheus_process_collector/c_src'
=ERROR REPORT==== 17-Aug-2022::16:38:29.833856 ===
beam/beam_load.c(148): Error loading module enc:
  please re-compile this module with an Erlang/OTP 25 compiler

escript: exception error: undefined function enc:main/1
  in function  escript:run/2 (escript.erl, line 750)
  in call from escript:start/1 (escript.erl, line 277)
  in call from init:start_em/1 
  in call from init:do_boot/3 
===> Hook for compile failed!

ThomasBlock commented 2 years ago

So here are additional findings:

the erlang Version in official Ubuntu is too old ( OTP 22 )
the erlang Version in deb https://packages.erlang-solutions.com is too new i guess ( OTP 25 )
the binaries of OTP 23 ( https://packages.erlang-solutions.com/erlang/debian/pool/esl-erlang_23.0-1~ubuntu~focal_amd64.deb ) don't include openssl i guess? see below..

So it seems i also have to compile OTP 23 myself? ( Ubuntu 20.04 on all of my systems )

===> Compiling c_src/b64fast.c
===> Linking /root/arweave/_build/default/lib/b64fast/priv/b64fast.so
===> "/root/arweave/_build/default/lib/prometheus_cowboy/ebin/prometheus_cowboy.app" is missing description entry
make: Entering directory '/root/arweave/apps/arweave/lib/RandomX/build'
make[1]: Entering directory '/root/arweave/apps/arweave/lib/RandomX/build'
make[2]: Entering directory '/root/arweave/apps/arweave/lib/RandomX/build'
Scanning dependencies of target randomx
make[2]: Leaving directory '/root/arweave/apps/arweave/lib/RandomX/build'
make[2]: Entering directory '/root/arweave/apps/arweave/lib/RandomX/build'
[  2%] Building CXX object CMakeFiles/randomx.dir/src/aes_hash.cpp.o
[  5%] Building C object CMakeFiles/randomx.dir/src/argon2_ref.c.o
[  8%] Building C object CMakeFiles/randomx.dir/src/argon2_ssse3.c.o
[ 11%] Building C object CMakeFiles/randomx.dir/src/argon2_avx2.c.o
[ 14%] Building CXX object CMakeFiles/randomx.dir/src/bytecode_machine.cpp.o
[ 17%] Building CXX object CMakeFiles/randomx.dir/src/cpu.cpp.o
[ 20%] Building CXX object CMakeFiles/randomx.dir/src/dataset.cpp.o
[ 23%] Building CXX object CMakeFiles/randomx.dir/src/soft_aes.cpp.o
[ 26%] Building CXX object CMakeFiles/randomx.dir/src/virtual_memory.cpp.o
[ 29%] Building CXX object CMakeFiles/randomx.dir/src/vm_interpreted.cpp.o
[ 32%] Building CXX object CMakeFiles/randomx.dir/src/allocator.cpp.o
[ 35%] Building CXX object CMakeFiles/randomx.dir/src/assembly_generator_x86.cpp.o
[ 38%] Building CXX object CMakeFiles/randomx.dir/src/instruction.cpp.o
[ 41%] Building CXX object CMakeFiles/randomx.dir/src/randomx.cpp.o
[ 44%] Building CXX object CMakeFiles/randomx.dir/src/superscalar.cpp.o
[ 47%] Building CXX object CMakeFiles/randomx.dir/src/vm_compiled.cpp.o
[ 50%] Building CXX object CMakeFiles/randomx.dir/src/vm_interpreted_light.cpp.o
[ 52%] Building C object CMakeFiles/randomx.dir/src/argon2_core.c.o
[ 55%] Building CXX object CMakeFiles/randomx.dir/src/blake2_generator.cpp.o
[ 58%] Building CXX object CMakeFiles/randomx.dir/src/instructions_portable.cpp.o
[ 61%] Building C object CMakeFiles/randomx.dir/src/reciprocal.c.o
[ 64%] Building CXX object CMakeFiles/randomx.dir/src/virtual_machine.cpp.o
[ 67%] Building CXX object CMakeFiles/randomx.dir/src/vm_compiled_light.cpp.o
[ 70%] Building C object CMakeFiles/randomx.dir/src/blake2/blake2b.c.o
[ 73%] Building CXX object CMakeFiles/randomx.dir/src/jit_compiler_x86.cpp.o
[ 76%] Building C object CMakeFiles/randomx.dir/src/jit_compiler_x86_static.S.o
[ 79%] Linking CXX static library librandomx.a
make[2]: Leaving directory '/root/arweave/apps/arweave/lib/RandomX/build'
[ 79%] Built target randomx
make[2]: Entering directory '/root/arweave/apps/arweave/lib/RandomX/build'
Scanning dependencies of target randomx-benchmark
make[2]: Leaving directory '/root/arweave/apps/arweave/lib/RandomX/build'
make[2]: Entering directory '/root/arweave/apps/arweave/lib/RandomX/build'
[ 82%] Building CXX object CMakeFiles/randomx-benchmark.dir/src/tests/benchmark.cpp.o
[ 85%] Building CXX object CMakeFiles/randomx-benchmark.dir/src/tests/affinity.cpp.o
[ 88%] Linking CXX executable randomx-benchmark
make[2]: Leaving directory '/root/arweave/apps/arweave/lib/RandomX/build'
[ 88%] Built target randomx-benchmark
make[2]: Entering directory '/root/arweave/apps/arweave/lib/RandomX/build'
Scanning dependencies of target randomx-codegen
make[2]: Leaving directory '/root/arweave/apps/arweave/lib/RandomX/build'
make[2]: Entering directory '/root/arweave/apps/arweave/lib/RandomX/build'
[ 91%] Building CXX object CMakeFiles/randomx-codegen.dir/src/tests/code-generator.cpp.o
[ 94%] Linking CXX executable randomx-codegen
make[2]: Leaving directory '/root/arweave/apps/arweave/lib/RandomX/build'
[ 94%] Built target randomx-codegen
make[2]: Entering directory '/root/arweave/apps/arweave/lib/RandomX/build'
Scanning dependencies of target randomx-tests
make[2]: Leaving directory '/root/arweave/apps/arweave/lib/RandomX/build'
make[2]: Entering directory '/root/arweave/apps/arweave/lib/RandomX/build'
[ 97%] Building CXX object CMakeFiles/randomx-tests.dir/src/tests/tests.cpp.o
[100%] Linking CXX executable randomx-tests
make[2]: Leaving directory '/root/arweave/apps/arweave/lib/RandomX/build'
[100%] Built target randomx-tests
make[1]: Leaving directory '/root/arweave/apps/arweave/lib/RandomX/build'
make: Leaving directory '/root/arweave/apps/arweave/lib/RandomX/build'
make: Entering directory '/root/arweave/apps/arweave/c_src'
g++ -O3 -finline-functions -Wall -fPIC -I /usr/lib/erlang/erts-11.0/include/ -I /usr/lib/erlang/lib/erl_interface-4.0/include -I ../lib/RandomX/src -std=c++11  -c -o /root/arweave/apps/arweave/c_src/feistel_msgsize_key_cipher.o /root/arweave/apps/arweave/c_src/feistel_msgsize_key_cipher.cpp
/root/arweave/apps/arweave/c_src/feistel_msgsize_key_cipher.cpp:1:10: fatal error: openssl/sha.h: No such file or directory
    1 | #include <openssl/sha.h>
      |          ^~~~~~~~~~~~~~~
compilation terminated.
make: *** [Makefile:77: /root/arweave/apps/arweave/c_src/feistel_msgsize_key_cipher.o] Error 1
make: Leaving directory '/root/arweave/apps/arweave/c_src'
===> Hook for compile failed!

ThomasBlock commented 2 years ago

Ah that were just missing header files (libssl-dev). I found now a solution. Maybe you want to clarify README.md - i guess this is not intuitive for everyone.. Here my commandline to compile Arweave 2.5.3 on Ubuntu 20.04 :

sudo -s
apt install curl build-essential git libsqlite3-dev libgmp-dev cmake libssl-dev -y
echo "deb https://packages.erlang-solutions.com/ubuntu $(lsb_release -cs) contrib" | sudo tee /etc/apt/sources.list.d/erlang.list
curl -fsSL https://packages.erlang-solutions.com/ubuntu/erlang_solutions.asc | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/erlang.gpg
apt update && apt upgrade -y
apt-get install esl-erlang=1:23.0-1 -y
apt-mark hold esl-erlang
git clone --recursive https://github.com/ArweaveTeam/arweave.git
cd arweave
./rebar3 as prod tar

ldmberman commented 2 years ago

Good point, we need to add the OpenSSL dev headers requirement to the list.

ArweaveTeam / arweave

Rebar3/erlang issues #370