search

helium / blockchain-etl

Blockchain follower that follows and stores the Helium blockchain
Apache License 2.0
64 stars 37 forks source link

"docker build" not working #158

Closed jluxenberg closed 3 years ago

jluxenberg commented 3 years ago

I'm getting a cryptic error trying to use the Dockerfile to build via docker build -t helium/etl .:

#12 141.2 make[6]: *** [CMakeFiles/rocksdb.dir/build.make:401: CMakeFiles/rocksdb.dir/db/db_impl/db_impl.cc.o] Error 1

I've tried this on the 1.1.85 release tag, and also on the commit which added Docker support (0f627bc); both give the same error.

Here's my abridged build log:

blockchain-etl $ docker --version
Docker version 20.10.5, build 55c4c88
blockchain-etl $ docker build -t helium/etl .
[+] Building 151.4s (12/16)
 => [internal] load build definition from Dockerfile                                                                                                                          0.0s
 => => transferring dockerfile: 1.07kB                                                                                                                                        0.0s
 => [internal] load .dockerignore                                                                                                                                             0.0s
 => => transferring context: 2B                                                                                                                                               0.0s
 => [internal] load metadata for docker.io/library/erlang:22.3.2-alpine                                                                                                       1.1s
 => [builder 1/8] FROM docker.io/library/erlang:22.3.2-alpine@sha256:c6102a9670269d064669fc0f0e7c4eb60c8c3d6b5acba34b7a1728b692e07fea                                         0.0s
 => [internal] load build context                                                                                                                                             0.1s
 => => transferring context: 3.04MB                                                                                                                                           0.1s
 => CACHED [runner 2/5] RUN apk add --no-cache --update ncurses dbus gmp libsodium gcc                                                                                        0.0s
 => CACHED [runner 3/5] RUN ulimit -n 64000                                                                                                                                   0.0s
 => CACHED [runner 4/5] WORKDIR /opt/etl                                                                                                                                      0.0s
 => CACHED [builder 2/8] RUN apk add --no-cache --update     git tar build-base linux-headers autoconf automake libtool pkgconfig     dbus-dev bzip2 bison flex gmp-dev cmak  0.0s
 => CACHED [builder 3/8] WORKDIR /usr/src/etl                                                                                                                                 0.0s
 => CACHED [builder 4/8] ADD . /usr/src/etl/                                                                                                                                  0.0s
 => ERROR [builder 5/8] RUN rebar3 as docker_etl tar                                                                                                                        150.0s
------
 > [builder 5/8] RUN rebar3 as docker_etl tar:
#12 1.975 ===> Fetching erlfmt v0.12.0
#12 2.043 ===> Downloaded package, caching at /root/.cache/rebar3/hex/hexpm/packages/erlfmt-0.12.0.tar
#12 2.119 ===> Compiling erlfmt
#12 11.77 ===> Linking _build/default/plugins/erlfmt to _build/docker_etl/plugins/erlfmt

...

#12 92.71 [100%] Built target snappy_unittest
#12 92.72 [ 34%] Performing install step for 'snappy'
#12 92.75 [ 62%] Built target snappy
#12 92.76 [100%] Built target snappy_unittest
#12 92.77 Install the project...
#12 92.78 -- Install configuration: ""
#12 92.78 -- Installing: /usr/src/etl/_build/default/lib/rocksdb/_build/cmake/snappy/lib64/libsnappy.a
#12 92.78 -- Installing: /usr/src/etl/_build/default/lib/rocksdb/_build/cmake/snappy/include/snappy-c.h
#12 92.78 -- Installing: /usr/src/etl/_build/default/lib/rocksdb/_build/cmake/snappy/include/snappy-sinksource.h
#12 92.78 -- Installing: /usr/src/etl/_build/default/lib/rocksdb/_build/cmake/snappy/include/snappy.h
#12 92.78 -- Installing: /usr/src/etl/_build/default/lib/rocksdb/_build/cmake/snappy/include/snappy-stubs-public.h
#12 92.78 -- Installing: /usr/src/etl/_build/default/lib/rocksdb/_build/cmake/snappy/lib64/cmake/Snappy/SnappyTargets.cmake
#12 92.78 -- Installing: /usr/src/etl/_build/default/lib/rocksdb/_build/cmake/snappy/lib64/cmake/Snappy/SnappyTargets-noconfig.cmake
#12 92.78 -- Installing: /usr/src/etl/_build/default/lib/rocksdb/_build/cmake/snappy/lib64/cmake/Snappy/SnappyConfig.cmake
#12 92.78 -- Installing: /usr/src/etl/_build/default/lib/rocksdb/_build/cmake/snappy/lib64/cmake/Snappy/SnappyConfigVersion.cmake
#12 92.79 [ 36%] Completed 'snappy'
#12 92.81 [ 36%] Built target snappy
#12 92.81 Scanning dependencies of target rocksdb
#12 92.82 [ 38%] Creating directories for 'rocksdb'
#12 92.85 [ 40%] No download step for 'rocksdb'
#12 92.87 [ 43%] No patch step for 'rocksdb'
#12 92.87 [ 45%] No update step for 'rocksdb'
#12 92.88 [ 47%] Performing configure step for 'rocksdb'
#12 92.95 -- The CXX compiler identification is GNU 9.3.0
#12 93.01 -- The C compiler identification is GNU 9.3.0
#12 93.02 -- The ASM compiler identification is GNU
#12 93.02 -- Found assembler: /usr/bin/gcc
#12 93.03 -- Check for working CXX compiler: /usr/bin/g++
#12 93.10 -- Check for working CXX compiler: /usr/bin/g++ -- works
#12 93.10 -- Detecting CXX compiler ABI info
#12 93.18 -- Detecting CXX compiler ABI info - done
#12 93.22 -- Detecting CXX compile features
#12 93.22 -- Detecting CXX compile features - done
#12 93.23 -- Check for working C compiler: /usr/bin/gcc
#12 93.30 -- Check for working C compiler: /usr/bin/gcc -- works
#12 93.30 -- Detecting C compiler ABI info
#12 93.36 -- Detecting C compiler ABI info - done
#12 93.39 -- Detecting C compile features
#12 93.39 -- Detecting C compile features - done
#12 93.39 -- Could NOT find Snappy (missing: Snappy_DIR)
#12 93.40 -- Found Snappy: /usr/src/etl/_build/default/lib/rocksdb/_build/cmake/snappy/lib64/libsnappy.a
#12 93.40 -- Found lz4: /usr/src/etl/_build/default/lib/rocksdb/_build/cmake/lz4/lib64/liblz4.a
#12 93.40 -- Found Git: /usr/bin/git (found version "2.24.4")
#12 93.41 -- Performing Test HAVE_OMIT_LEAF_FRAME_POINTER
#12 93.48 -- Performing Test HAVE_OMIT_LEAF_FRAME_POINTER - Success
#12 93.48 -- Performing Test HAS_ARMV8_CRC
#12 93.51 -- Performing Test HAS_ARMV8_CRC - Failed
#12 93.51 -- Performing Test HAVE_SSE42
#12 93.63 -- Performing Test HAVE_SSE42 - Success
#12 93.63 -- Performing Test HAVE_THREAD_LOCAL
#12 93.70 -- Performing Test HAVE_THREAD_LOCAL - Success
#12 93.70 -- Enabling RTTI
#12 93.70 -- Performing Test HAVE_FALLOCATE
#12 93.77 -- Performing Test HAVE_FALLOCATE - Success
#12 93.77 -- Performing Test HAVE_SYNC_FILE_RANGE_WRITE
#12 93.84 -- Performing Test HAVE_SYNC_FILE_RANGE_WRITE - Success
#12 93.84 -- Performing Test HAVE_PTHREAD_MUTEX_ADAPTIVE_NP
#12 93.87 -- Performing Test HAVE_PTHREAD_MUTEX_ADAPTIVE_NP - Failed
#12 93.88 -- Looking for malloc_usable_size
#12 93.94 -- Looking for malloc_usable_size - found
#12 93.94 -- Looking for sched_getcpu
#12 94.01 -- Looking for sched_getcpu - found
#12 94.01 -- Looking for getauxval
#12 94.04 -- Looking for getauxval - not found
#12 94.04 -- Looking for pthread.h
#12 94.11 -- Looking for pthread.h - found
#12 94.11 -- Performing Test CMAKE_HAVE_LIBC_PTHREAD
#12 94.17 -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
#12 94.17 -- Found Threads: TRUE
#12 94.18 -- JNI library is disabled
#12 94.19 -- Configuring done
#12 94.42 -- Generating done
#12 94.43 -- Build files have been written to: /usr/src/etl/_build/default/lib/rocksdb/_build/cmake/rocksdb-prefix/src/rocksdb-build
#12 94.44 [ 50%] Performing build step for 'rocksdb'
#12 94.49 Scanning dependencies of target build_version
#12 94.50 [  0%] Building CXX object CMakeFiles/build_version.dir/build_version.cc.o
#12 94.52 [  0%] Built target build_version
#12 94.53 Scanning dependencies of target rocksdb
#12 95.80 [  0%] Building CXX object CMakeFiles/rocksdb.dir/cache/cache.cc.o
#12 95.80 [  0%] Building CXX object CMakeFiles/rocksdb.dir/cache/clock_cache.cc.o
#12 95.80 [  0%] Building CXX object CMakeFiles/rocksdb.dir/cache/lru_cache.cc.o
#12 95.81 [  0%] Building CXX object CMakeFiles/rocksdb.dir/cache/sharded_cache.cc.o
#12 95.81 [  2%] Building CXX object CMakeFiles/rocksdb.dir/db/arena_wrapped_db_iter.cc.o
#12 95.81 [  2%] Building CXX object CMakeFiles/rocksdb.dir/db/blob/blob_file_addition.cc.o
#12 96.43 [  2%] Building CXX object CMakeFiles/rocksdb.dir/db/blob/blob_file_builder.cc.o
#12 97.40 [  2%] Building CXX object CMakeFiles/rocksdb.dir/db/blob/blob_file_garbage.cc.o
#12 98.05 [  2%] Building CXX object CMakeFiles/rocksdb.dir/db/blob/blob_file_meta.cc.o
#12 98.24 [  2%] Building CXX object CMakeFiles/rocksdb.dir/db/blob/blob_log_format.cc.o
#12 98.97 [  4%] Building CXX object CMakeFiles/rocksdb.dir/db/blob/blob_log_reader.cc.o
#12 99.10 [  4%] Building CXX object CMakeFiles/rocksdb.dir/db/blob/blob_log_writer.cc.o
#12 99.50 [  4%] Building CXX object CMakeFiles/rocksdb.dir/db/builder.cc.o
#12 99.90 [  4%] Building CXX object CMakeFiles/rocksdb.dir/db/c.cc.o
#12 101.7 [  4%] Building CXX object CMakeFiles/rocksdb.dir/db/column_family.cc.o
#12 103.6 [  6%] Building CXX object CMakeFiles/rocksdb.dir/db/compacted_db_impl.cc.o
#12 103.8 [  6%] Building CXX object CMakeFiles/rocksdb.dir/db/compaction/compaction.cc.o
#12 104.1 [  6%] Building CXX object CMakeFiles/rocksdb.dir/db/compaction/compaction_iterator.cc.o
#12 109.5 [  6%] Building CXX object CMakeFiles/rocksdb.dir/db/compaction/compaction_picker.cc.o
#12 111.2 [  6%] Building CXX object CMakeFiles/rocksdb.dir/db/compaction/compaction_job.cc.o
#12 111.3 [  6%] Building CXX object CMakeFiles/rocksdb.dir/db/compaction/compaction_picker_fifo.cc.o
#12 112.5 [  8%] Building CXX object CMakeFiles/rocksdb.dir/db/compaction/compaction_picker_level.cc.o
#12 114.1 [  8%] Building CXX object CMakeFiles/rocksdb.dir/db/compaction/compaction_picker_universal.cc.o
#12 115.2 [  8%] Building CXX object CMakeFiles/rocksdb.dir/db/compaction/sst_partitioner.cc.o
#12 115.8 [  8%] Building CXX object CMakeFiles/rocksdb.dir/db/convenience.cc.o
#12 116.8 [  8%] Building CXX object CMakeFiles/rocksdb.dir/db/db_filesnapshot.cc.o
#12 117.1 [ 10%] Building CXX object CMakeFiles/rocksdb.dir/db/db_impl/db_impl.cc.o
#12 117.9 [ 10%] Building CXX object CMakeFiles/rocksdb.dir/db/db_impl/db_impl_write.cc.o
#12 120.6 [ 10%] Building CXX object CMakeFiles/rocksdb.dir/db/db_impl/db_impl_compaction_flush.cc.o
#12 122.2 [ 10%] Building CXX object CMakeFiles/rocksdb.dir/db/db_impl/db_impl_files.cc.o
#12 122.8 [ 10%] Building CXX object CMakeFiles/rocksdb.dir/db/db_impl/db_impl_open.cc.o
#12 124.9 [ 10%] Building CXX object CMakeFiles/rocksdb.dir/db/db_impl/db_impl_debug.cc.o
#12 125.0 [ 12%] Building CXX object CMakeFiles/rocksdb.dir/db/db_impl/db_impl_experimental.cc.o
#12 141.2 g++: fatal error: Killed signal terminated program cc1plus
#12 141.2 compilation terminated.
#12 141.2 make[6]: *** [CMakeFiles/rocksdb.dir/build.make:401: CMakeFiles/rocksdb.dir/db/db_impl/db_impl.cc.o] Error 1
#12 141.2 make[6]: *** Waiting for unfinished jobs....
#12 149.7 make[5]: *** [CMakeFiles/Makefile2:374: CMakeFiles/rocksdb.dir/all] Error 2
#12 149.7 make[4]: *** [CMakeFiles/Makefile2:386: CMakeFiles/rocksdb.dir/rule] Error 2
#12 149.7 make[3]: *** [Makefile:281: rocksdb] Error 2
#12 149.8 make[2]: *** [CMakeFiles/rocksdb.dir/build.make:112: rocksdb-prefix/src/rocksdb-stamp/rocksdb-build] Error 2
#12 149.8 make[1]: *** [CMakeFiles/Makefile2:80: CMakeFiles/rocksdb.dir/all] Error 2
#12 149.8 make: *** [Makefile:84: all] Error 2
#12 149.8 + exit 1
#12 149.8 ===> Hook for compile failed!
#12 149.8
------
executor failed running [/bin/sh -c rebar3 as docker_etl tar]: exit code: 1
jluxenberg commented 3 years ago

Ah; my Docker VM was only being given 2GB of RAM. I bumped it up to 4GB and the build worked.

(got a clue that it might be a memory issue from this thread https://github.com/facebook/rocksdb/issues/2858#issuecomment-328307318)