Automated docker build

[holiman]

There should be an automated docker build at hub.docker.com/r/ethereum .

• [ ] Define a docker file
• [ ] Set up an automated build

Preferrably using a multistage build to get the size down as much as possible. Examples:

• Multistage cpp-ethereum (though, seems to be failing): https://hub.docker.com/r/ethereum/eth/~/dockerfile/
• Old cpp-ethereum: https://hub.docker.com/r/ethereum/client-go/~/dockerfile/

[holiman]

When trying to build with a multistage docker file:

RUN apk add --no-cache \
        git \
        cmake \
        g++ \
        make \
        linux-headers \
        leveldb-dev --repository http://dl-cdn.alpinelinux.org/alpine/edge/testing/
#RUN git clone --recursive https://github.com/ethereum/cpp-ethereum --branch develop --single-branch --depth 1
RUN git clone --recursive https://github.com/ethereum/testeth --branch develop --single-branch --depth 1
RUN mkdir /build
#todo -DSTATIC_LINKING=1  ??
RUN cd /build && cmake /testeth -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=RelWithDebInfo  -DEVM_OPTIMIZE=0
RUN cd /build && make -j8 && make install
# Install stage 
FROM alpine:latest 
RUN apk add --no-cache \
        libstdc++ \
        leveldb --repository http://dl-cdn.alpinelinux.org/alpine/edge/testing/
COPY --from=builder /usr/bin/testeth /usr/bin/testeth
ENTRYPOINT ["/usr/bin/testeth"]

I ran into the following error:

common.mkdir bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/numa/linux
gcc.compile.c++ bin.v2/libs/filesystem/build/gcc-6.3.0/release/link-static/threading-multi/portability.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/numa/linux/pin_thread.o
In file included from /usr/include/pthread.h:30:0,
                 from libs/fiber/src/numa/linux/pin_thread.cpp:10:
libs/fiber/src/numa/linux/pin_thread.cpp: In function 'void boost::fibers::numa::pin_thread(uint32_t)':
libs/fiber/src/numa/linux/pin_thread.cpp:27:5: error: 'memset' was not declared in this scope
     CPU_ZERO( & set);
     ^

    "/usr/bin/c++" "-std=c++11"   -O3 -finline-functions -Wno-inline -Wall -pthread -m64  -DBOOST_ALL_NO_LIB=1 -DBOOST_DISABLE_ASSERTS -DBOOST_FIBERS_SOURCE -DBOOST_FILESYSTEM_STATIC_LINK=1 -DBOOST_SYSTEM_STATIC_LINK=1 -DNDEBUG  -I"." -c -o "bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/numa/linux/pin_thread.o" "libs/fiber/src/numa/linux/pin_thread.cpp"

...failed gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/numa/linux/pin_thread.o...
common.mkdir bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/algo
gcc.compile.c++ bin.v2/libs/filesystem/build/gcc-6.3.0/release/link-static/threading-multi/utf8_codecvt_facet.o
gcc.compile.c++ bin.v2/libs/filesystem/build/gcc-6.3.0/release/link-static/threading-multi/operations.o
gcc.archive bin.v2/libs/filesystem/build/gcc-6.3.0/release/link-static/threading-multi/libboost_filesystem-mt.a
common.copy stage/lib/libboost_filesystem-mt.a
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/algo/algorithm.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/algo/round_robin.o
common.mkdir bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/algo/numa
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/algo/shared_work.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/algo/work_stealing.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/barrier.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/algo/numa/work_stealing.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/condition_variable.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/future.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/fiber.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/context.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/mutex.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/properties.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/recursive_mutex.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/recursive_timed_mutex.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/timed_mutex.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/scheduler.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/numa/linux/topology.o
...skipped <pbin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi>libboost_fiber-mt.a(clean) for lack of <pbin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi>numa/linux/pin_thread.o...
...skipped <pbin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi>libboost_fiber-mt.a for lack of <pbin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi>numa/linux/pin_thread.o...
...skipped <pstage/lib>libboost_fiber-mt.a for lack of <pbin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi>libboost_fiber-mt.a...
...failed updating 2 targets...
...skipped 6 targets...
...updated 124 targets...
make[2]: *** [CMakeFiles/Boost-fiber.dir/build.make:114: Boost-fiber-prefix/src/Boost-fiber-stamp/Boost-fiber-build] Error 1
make[1]: *** [CMakeFiles/Makefile2:68: CMakeFiles/Boost-fiber.dir/all] Error 2
make: *** [Makefile:84: all] Error 2

[hunter ** FATAL ERROR **] Build step failed (dir: /root/.hunter/_Base/a412c45/e35271b/2450d4d/Build/Boost/__fiber
[hunter ** FATAL ERROR **] [Directory:/root/.hunter/_Base/Download/Hunter/0.19.232/a412c45/Unpacked/cmake/projects/Boost/fiber]

------------------------------ WIKI -------------------------------
    https://github.com/ruslo/hunter/wiki/error.external.build.failed
-------------------------------------------------------------------

CMake Error at /root/.hunter/_Base/Download/Hunter/0.19.232/a412c45/Unpacked/cmake/modules/hunter_wiki.cmake:12 (message):
Call Stack (most recent call first):
  /root/.hunter/_Base/Download/Hunter/0.19.232/a412c45/Unpacked/cmake/modules/hunter_fatal_error.cmake:20 (hunter_wiki)
  /root/.hunter/_Base/Download/Hunter/0.19.232/a412c45/Unpacked/cmake/modules/hunter_download.cmake:648 (hunter_fatal_error)
  /root/.hunter/_Base/Download/Hunter/0.19.232/a412c45/Unpacked/cmake/projects/Boost/fiber/hunter.cmake:16 (hunter_download)
  /root/.hunter/_Base/Download/Hunter/0.19.232/a412c45/Unpacked/cmake/modules/hunter_add_package.cmake:53 (include)
  CMakeLists.txt:106 (hunter_add_package)

-- Configuring incomplete, errors occurred!
See also "/build/CMakeFiles/CMakeOutput.log".
See also "/build/CMakeFiles/CMakeError.log".
The command '/bin/sh -c cd /build && cmake /testeth -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=RelWithDebInfo  -DEVM_OPTIMIZE=0' returned a non-zero code: 1

[holiman]

This works:

cmake /testeth -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=RelWithDebInfo -DTOOLS=OFF -DTESTS=OFF

This does not :

cmake /testeth -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=RelWithDebInfo  -DEVM_OPTIMIZE=0

I'm guessing either TOOLS or TESTS cause the failure above. cc @chfast

[holiman]

Seems to fail when I leave out -DTOOLS=OFF

[holiman]

When I use TESTS (but TOOLS disabled), the cmake works, but make fails with the following error :

[ 97%] Building CXX object test/CMakeFiles/testeth.dir/unittests/libtesteth/blockchainTest.cpp.o
[ 98%] Building CXX object test/CMakeFiles/testeth.dir/unittests/libtesteth/testHelperTest.cpp.o
[ 98%] Building CXX object test/CMakeFiles/testeth.dir/unittests/libweb3core/memorydb.cpp.o
[ 99%] Building CXX object test/CMakeFiles/testeth.dir/unittests/libweb3core/overlaydb.cpp.o
[ 99%] Building CXX object test/CMakeFiles/testeth.dir/unittests/libweb3jsonrpc/AccountHolder.cpp.o
[ 99%] Building CXX object test/CMakeFiles/testeth.dir/unittests/libweb3jsonrpc/Client.cpp.o
make[2]: *** No rule to make target 'deps/lib/libbinaryen.a', needed by 'test/testeth'.  Stop.
make[2]: *** Waiting for unfinished jobs....
[100%] Building CXX object test/CMakeFiles/testeth.dir/unittests/libweb3jsonrpc/jsonrpc.cpp.o
make[1]: *** [CMakeFiles/Makefile2:963: test/CMakeFiles/testeth.dir/all] Error 2
make: *** [Makefile:141: all] Error 2
The command '/bin/sh -c cd /build && make -j2 && make install' returned a non-zero code: 2

(note^ I turned down paralellism to 2, in case it was a race issue)

[holiman]

(tried with 1 aswell, same problem)

[holiman]

It builds it correctly, but seems to install it at another place than we search for it

[100%] Built target s2wasm
[ 10%] Performing install step for 'binaryen'
[  7%] Built target wasm
[ 11%] Built target ir
[ 14%] Built target asmjs
[ 16%] Built target cfg
[ 59%] Built target passes
[ 67%] Built target support
[ 70%] Built target wasm-reduce
[ 74%] Built target emscripten-optimizer
[ 77%] Built target wasm-opt
[ 80%] Built target wasm-shell
[ 82%] Built target wasm-as
[ 84%] Built target binaryen
[ 86%] Built target asm2wasm
[ 88%] Built target wasm-merge
[ 90%] Built target wasm-metadce
[ 92%] Built target wasm-dis
[ 94%] Built target wasm-ctor-eval
[ 96%] Built target wasm2asm
[100%] Built target s2wasm
Install the project...
-- Install configuration: "Release"
-- Installing: /build/deps/lib64/libbinaryen.a
-- Installing: /build/deps/include/binaryen-c.h
-- Installing: /build/deps/share/binaryen/wasm.js
-- Installing: /build/deps/share/binaryen/binaryen.js
-- Installing: /build/deps/bin/wasm-shell
-- Installing: /build/deps/bin/wasm-opt
-- Installing: /build/deps/bin/wasm-merge
-- Installing: /build/deps/bin/wasm-metadce
-- Installing: /build/deps/bin/asm2wasm
-- Installing: /build/deps/bin/wasm2asm
-- Installing: /build/deps/bin/s2wasm
-- Installing: /build/deps/bin/wasm-as
-- Installing: /build/deps/bin/wasm-dis
-- Installing: /build/deps/bin/wasm-ctor-eval
-- Installing: /build/deps/bin/wasm-reduce
[ 10%] Completed 'binaryen'
[ 10%] Built target binaryen

Notably: Installing: /build/deps/lib64/libbinaryen.a, err msg: No rule to make target 'deps/lib/libbinaryen.a'. lib64 vs lib.

[pirapira]

try git submodule init and git submodule update.

[holiman]

Ok, finally got it working. Here's my docker image

# Build stage 
FROM alpine:latest as builder
RUN apk add --no-cache \
        git \
        cmake \
        g++ \
        make \
        linux-headers \
        leveldb-dev --repository http://dl-cdn.alpinelinux.org/alpine/edge/testing/
RUN git clone --recursive https://github.com/ethereum/testeth --branch develop --single-branch --depth 1
RUN mkdir /build
# See https://github.com/ethereum/cpp-ethereum/issues/4834
RUN cd /build && cmake /testeth -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=RelWithDebInfo -DTOOLS=OFF
RUN cmake --version
RUN cd /build && make -j1 && make install; exit 0
RUN cd /build && cp -r deps/lib64/libbinaryen.a deps/lib/; cp -r deps/lib64/libmpir.a deps/lib/
RUN cd /build && make -j1 && make install

# Install stage 
FROM alpine:latest
RUN apk add --no-cache \
        libstdc++ \
        leveldb --repository http://dl-cdn.alpinelinux.org/alpine/edge/testing/
COPY --from=builder /build/test/testeth /usr/bin/testeth

#ENTRYPOINT ["/usr/bin/testeth"]

I disabled the entrypoint, but you should probably enable it if you use this as a 'regular' docker image. And you can probably set -j8 back aswell, I lowered it to ensure that the build failure wasn't due to paralellism.

[pirapira]

Thank you. I'll try the old cpp image to see what should happen.

(I've never used docker for testeth because my machine somewhere can compile testeth reasonably fast.)

[pirapira]

@holiman the links

https://hub.docker.com/r/ethereum/eth/~/dockerfile/ https://hub.docker.com/r/ethereum/client-go/~/dockerfile/

go 404.

[olk]

for error 'CPU_ZERO() ... missing memset' see https://github.com/boostorg/fiber/pull/142 -> was a bug in musl

[axic]

Need to update the Hera submodule (point it to a recent commit) to fix this. See https://github.com/ethereum/cpp-ethereum/pull/4906 as a reference.

[chfast]

Maintaining both testeth in cpp-ethereum and the one here looks like a waste of time.

[pirapira]

@axic @gumb0 please talk with @chfast and decide if you guys still want to oust testeth from cpp-ethereum.

[axic]

Well it was supposed to be changed to use RPC only and then moved out. It seems that instead of testeth being moved to RPC, retesteth is being written in such a way.

[pirapira]

@axic that's a new piece of information. That sounds like I can keep using testeth in cpp-ethereum repo for filling tests. Do you still mind if I keep maintaining testeth in cpp-ethereum repository with PRs like https://github.com/ethereum/testeth/pull/13 and https://github.com/ethereum/testeth/pull/14?

[pirapira]

I thought @axic and @gumb0 didn't want to see test-maintaining PRs like #13 and #14 in cpp-ethereum repository.

[pirapira]

There are worse PRs like https://github.com/ethereum/testeth/pull/22 for testing new tests. (The current process uses the Travis on testeth repository for PRs in tests repository.) I don't think I should bring these back to cpp-ethereum repo. Issue: https://github.com/ethereum/tests/issues/436

[pirapira]

The duplicate of testeth cpp-ethereum repositories will be discussed in #23.