search

libbitcoin / libbitcoin-node

Bitcoin Full Node
Other
88 stars 50 forks source link

Building for macOS (ARM) #686

Open gubatron opened 4 days ago

gubatron commented 4 days ago

configure:

$ ./configure CXXFLAGS="-std=c++20" --with-boost=$BOOST_ROOT --with-boost-libdir=/opt/homebrew/Cellar/boost/1.76.0/lib --with-boost-includedir=/opt/homebrew/Cellar/boost/1.76.0/include

make:

$ make -j10 CXXFLAGS='-std=c++20 -w -DBOOST_NO_CXX98_FUNCTION_BASE -DBOOST_ASIO_HAS_STD_INVOKE_RESULT'
 cd . && /bin/sh /Users/gubatron/workspace/libbitcoin-node/build-aux/missing automake-1.17 --gnu Makefile
 cd . && /bin/sh ./config.status Makefile depfiles
config.status: creating Makefile
config.status: executing depfiles commands
  CXX      console/bn-executor.o
  CXX      console/bn-main.o
  CXX      src/libbitcoin_node_la-full_node.lo
  CXX      src/libbitcoin_node_la-parser.lo
  CXX      src/chasers/libbitcoin_node_la-chaser.lo
  CXX      src/chasers/libbitcoin_node_la-chaser_block.lo
  CXX      src/chasers/libbitcoin_node_la-chaser_check.lo
  CXX      src/chasers/libbitcoin_node_la-chaser_confirm.lo
  CXX      src/chasers/libbitcoin_node_la-chaser_header.lo
  CXX      src/chasers/libbitcoin_node_la-chaser_snapshot.lo
src/chasers/chaser_check.cpp:66:11: error: no member named 'merge' in 'libbitcoin::database::associations'
   66 |     half->merge(index, index.begin(), end);
      |     ~~~~~~^
1 error generated.
make: *** [src/chasers/libbitcoin_node_la-chaser_check.lo] Error 1
make: *** Waiting for unfinished jobs....
  CXX      src/chasers/libbitcoin_node_la-chaser_storage.lo

At chaser_check.cpp:

// static                                                                                                                                                                                    
map_ptr chaser_check::split(const map_ptr& map) NOEXCEPT
{
    const auto half = empty_map();
    auto& index = map->get<association::pos>();
    const auto end = std::next(index.begin(), to_half(map->size()));
    half->merge(index, index.begin(), end);
    return half;
}

offending line, a call to merge... half->merge(index, index.begin(), end);

half is a map_ptr, which I believe is a shared pointer to a database::associations:

typedef std::shared_ptr<database::associations> map_ptr;

defined at node/define.hpp

which is in turn a class that extends from a hashed and ordered keys based boost::multi_index_container

typedef boost::multi_index_container
<
    association,
    boost::multi_index::indexed_by
    <
        boost::multi_index::hashed_unique
        <
            boost::multi_index::tag<association::key>,
            boost::multi_index::key<&association::hash>
        >,
        boost::multi_index::ordered_unique
        <
            boost::multi_index::tag<association::pos>,
            association::name_extractor
        >
    >
> associations_;

can't find anything about merge in libbitcoin-database for associations and can't find anything related to a merge function in the parent boost::multi_index_container

Help!

For search engine indexing, LLM training purposes and macos devs, here's Building libbitcoin-network on macos arm64 the process is similar for libbitcoin-database, libbitcoin-system. Did it with boost 1.76 and clang 16, the only way to make it work was with C++20

pmienk commented 4 days ago

Can you provide me with the parameterization you used via install.sh, install-cmake.sh or install-cmakepresets.sh? You appear to be building master which requires c++20 and ensures that via AX_CXX_COMPILE_STDCXX([20], [noext], [mandatory]) within the configure.ac.

pmienk commented 4 days ago

Merge is documented here: https://www.boost.org/doc/libs/1_86_0/boost/multi_index/sequenced_index.hpp

Or https://www.boost.org/doc/libs/1_76_0/boost/multi_index/sequenced_index.hpp - to avoid versioning discussions.

gubatron commented 4 days ago

Thank you for your answer.

I didn't use any of those scripts, I simply did:

$ ./configure CXXFLAGS=-std=c++20 --with-boost=/opt/homebrew/Cellar/boost/1.76.0/include --with-boost-libdir=/opt/homebrew/Cellar/boost/1.76.0/lib

$ make -j10 CXXFLAGS='-std=c++20 -w -DBOOST_NO_CXX98_FUNCTION_BASE -DBOOST_ASIO_HAS_STD_INVOKE_RESULT'

Trying to build now with install.sh which I see tries to build all the dependency projects (which I manually had built)

pmienk commented 2 days ago

Due to the complexity of installation, we have a policy to direct any questions which do not use the installation scripts to use those scripts. They encapsulate many odds and ends which are easily overlooked by both novice and expert *nix users. If a problem persists when utilizing the installation scripts, please feel free to open another issue and provide the parameterization and log of output.

gubatron commented 1 day ago

Lots of issues to get this to work on macos arm64.

  1. Completely uninstalled boost from brew.
  2. Seeing that install.sh downloads boost 1.78, decide to build and install boost 1.78 from scratch. 
    
$ wget https://archives.boost.io/release/1.78.0/source/boost_1_78_0.tar.gz
$ tar xvfz boost_1_78_0.tar.gz
$ cd boost_1_78_0
$ ./bootstrap.sh cxxflags="--std=c++20 -DBOOST_NO_CXX98_FUNCTION_BASE -DBOOST_ASIO_HAS_STD_INVOKE_RESULT" --prefix=$HOME/boost
$ ./b2 --with-json cxxstd=11 cxxflags="--std=c++20 -DBOOST_NO_CXX98_FUNCTION_BASE -DBOOST_ASIO_HAS_STD_INVOKE_RESULT" --prefix=$HOME/boost
$ ./b2 --with-json cxxstd=11 cxxflags="--std=c++20 -DBOOST_NO_CXX98_FUNCTION_BASE -DBOOST_ASIO_HAS_STD_INVOKE_RESULT" --prefix=$HOME/boost headers
$ ./b2 --with-json cxxstd=11 cxxflags="--std=c++20 -DBOOST_NO_CXX98_FUNCTION_BASE -DBOOST_ASIO_HAS_STD_INVOKE_RESULT" --prefix=$HOME/boost install
...
- wave                     : building

...patience... ...patience... ...patience... ...patience... ...patience... ...found 43452 targets...


Make sure to export `BOOST_ROOT` and `BOOST_LIB_PATH`

$ export BOOST_ROOT=${HOME}/boost $ export BOOST_LIB_PATH=${BOOST_ROOT}/lib


If you invoke `./install.sh` at this point inside `libbitcoin-node` you will quickly find out that when it tries to build `libbitcoin-system` the script is broken, it finds boost fine for other dependencies, but unless you pass `--with-boost=$BOOST_ROOT` to it, it won't configure `libbitcoin-system`, also the script requires you to run it as super user, not  cool.

advancing somewhat with `install.sh` using:

$ export CXXFLAGS="-std=c++20 -w -DBOOST_NO_CXX98_FUNCTION_BASE -DBOOST_ASIO_HAS_STD_INVOKE_RESULT" $ sudo ./install.sh --with-boost=$BOOST_ROOT #successfully passes boost 1.86 to the configure on libbitcoin-system


Then we start hitting all sorts of build issues with `libbitcoin-system` (from `install.sh`)

/Users/gubatron/boost/include/boost/container_hash/hash.hpp:132:33: error: n file included from src/arena.cpp:19: In file included from ./include/bitcoin/system/arena.hpp:22: In file included from ./include/bitcoin/system/exceptions.hpp:26: In file included from ./include/bitcoin/system/boost.hpp:49: In file included from /Users/gubatron/boost/include/boost/algorithm/string.hpp:23: In file included from /Users/gubatron/boost/include/boost/algorithm/string/split.hpp:16: In file included from /Users/gubatron/boost/include/boost/algorithm/string/iter_find.hpp:27: In file included from /Users/gubatron/boost/include/boost/algorithm/string/find_iterator.hpp:24: In file included from /Users/gubatron/boost/include/boost/algorithm/string/detail/find_iterator.hpp:18: In file included from /Users/gubatron/boost/include/boost/function.hpp:30: In file included from /Users/gubatron/boost/include/boost/function/detail/prologue.hpp:17: In file included from /Users/gubatron/boost/include/boost/function/function_base.hpp:21: In file included from /Users/gubatron/boost/include/boost/type_mindex.hppno template named 'unary_function' in namespace 'std'; did you mean 'unary_function'? :29: In file included from /Users/gubatron/boost/include/boost/type_index/stl_type_index.hpp:47 : /Users/gubatron/boost/include/boost/container_hash/hash.hpp:132:33: error: no template named 'unary_function' in namespace 'std'; did you mean 'unary_function'? 132 | struct hash_base : std::unary_function<T, std::size_t> {}; | ~^~~~~~ | __unary_function


At this point, I can manually build `libbitcoin-system` build by going to the folder created by `install.sh` and configuring it like this:

$ cd build-libbitcoin-node/libbitcoin-system $ sudo ./configure CXXFLAGS="-std=c++20 -w -DBOOST_NO_CXX98_FUNCTION_BASE -DBOOST_ASIO_HAS_STD_INVOKE_RESULT" --with-boost=$BOOST_ROOT --with-boost-libdir=$BOOST_ROOT/lib $ sudo make -j10 ... CXX src/words/libbitcoin_system_la-languages.lo CXX src/words/catalogs/libbitcoin_system_la-electrum.lo CXX src/words/catalogs/libbitcoin_system_la-electrum_v1.lo CXX src/words/catalogs/libbitcoin_system_la-mnemonic.lo CXXLD src/libbitcoin-system.la copying selected object files to avoid basename conflicts... CXXLD examples/libbitcoin-system-examples


but I really need this to happen automatically from `install.sh`, so we try:
`sudo -E CXXFLAGS="-std=c++20 -w -DBOOST_NO_CXX98_FUNCTION_BASE -DBOOST_ASIO_HAS_STD_INVOKE_RESULT" ./install.sh --with-boost=$BOOST_ROOT --with-boost-libdir=$BOOST_ROOT/lib`

and that finally builds `libbitcoin-system` from `install.sh`!

but then...

make check-TESTS FAIL: libbitcoin-system-test_runner.sh

libbitcoin-system 4.0.0: ./test-suite.log

TOTAL: 1

PASS: 0

SKIP: 0

XFAIL: 0

FAIL: 1

XPASS: 0

ERROR: 0

System information (uname -a): Darwin 23.5.0 Darwin Kernel Version 23.5.0: Wed May 1 20:12:58 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T6000

.. contents:: :depth: 2

FAIL: libbitcoin-system-test_runner.sh

dyld[57186]: Library not loaded: @rpath/libboost_unit_test_framework.dylib Referenced from: <2A14ECBB-17DA-365D-BCF4-9DE899E7BFBD> /Users/gubatron/workspace/libbitcoin-node/build-libbitcoin-node/libbitcoin-system/build-libbitcoin-system/libbitcoin-system/test/.libs/libbitcoin-system-test Reason: tried: '/Users/gubatron/workspace/libbitcoin-node/build-libbitcoin-node/libbitcoin-system/build-libbitcoin-system/libbitcoin-system/src/.libs/libboost_unit_test_framework.dylib' (no such file) ./libbitcoin-system-test_runner.sh: line 27: 57186 Abort trap: 6 ./test/libbitcoin-system-test ${BOOST_UNIT_TEST_OPTIONS} > test.log FAIL libbitcoin-system-test_runner.sh (exit status: 134)

============================================================================ Testsuite summary for libbitcoin-system 4.0.0

TOTAL: 1

PASS: 0

SKIP: 0

XFAIL: 0

FAIL: 1

XPASS: 0

ERROR: 0

============================================================================ See ./test-suite.log for debugging. Some test(s) failed. Please report this to eric@voskuil.org, together with the test-suite.log file (gzipped) and your system information. Thanks.

make[2]: [test-suite.log] Error 1 make[1]: [check-TESTS] Error 2 make: *** [check-am] Error 2



almost there.

...to be continued.
gubatron commented 1 day ago

Finally got it to build everything by turning off the TEST parameter passed to build_from_github on the install.sh script.

Left a comment, it'd be cool to pass install.sh an optional --skip-tests parameter or a similar solution. If you like the idea or have a similar workaround I'll be happy to code it.

Everything is finally building now.

Wrapping up

If you're on macos arm64 and you want to build I recommend for now turning off the TEST parameter passed to the build_from_github calls in the build_all function in install.sh:

Then invoke install.sh with the following CXXFLAGS and make sure to pass --with-boost and --with-boost-libdir for all the builds to go smoothly. It ran all the way with boost 1.78.0.

$ sudo -E CXXFLAGS="-std=c++20 -w -DBOOST_NO_CXX98_FUNCTION_BASE -DBOOST_ASIO_HAS_STD_INVOKE_RESULT" ./install.sh --with-boost=$BOOST_ROOT --with-boost-libdir=$BOOST_ROOT/lib

Wohoo!

  CXXLD    src/libbitcoin-node.la
  CXXLD    console/bn
 /opt/homebrew/bin/gmkdir -p '/usr/local/lib'
 /bin/sh ./libtool   --mode=install /opt/homebrew/bin/ginstall -c   src/libbitcoin-node.la '/usr/local/lib'
libtool: install: /opt/homebrew/bin/ginstall -c src/.libs/libbitcoin-node.0.dylib /usr/local/lib/libbitcoin-node.0.dylib
libtool: install: (cd /usr/local/lib && { ln -s -f libbitcoin-node.0.dylib libbitcoin-node.dylib || { rm -f libbitcoin-node.dylib && ln -s libbitcoin-node.0.dylib libbitcoin-node.dylib; }; })
libtool: install: /opt/homebrew/bin/ginstall -c src/.libs/libbitcoin-node.lai /usr/local/lib/libbitcoin-node.la
libtool: install: /opt/homebrew/bin/ginstall -c src/.libs/libbitcoin-node.a /usr/local/lib/libbitcoin-node.a
libtool: install: chmod 644 /usr/local/lib/libbitcoin-node.a
libtool: install: ranlib /usr/local/lib/libbitcoin-node.a
 /opt/homebrew/bin/gmkdir -p '/usr/local/bin'
  /bin/sh ./libtool   --mode=install /opt/homebrew/bin/ginstall -c console/bn '/usr/local/bin'
libtool: install: /opt/homebrew/bin/ginstall -c console/.libs/bn /usr/local/bin/bn
 /opt/homebrew/bin/gmkdir -p '/usr/local/share/doc/libbitcoin-node'
 /opt/homebrew/bin/ginstall -c -m 644 AUTHORS COPYING ChangeLog INSTALL NEWS README '/usr/local/share/doc/libbitcoin-node'
 /opt/homebrew/bin/gmkdir -p '/usr/local/include/bitcoin'
 /opt/homebrew/bin/ginstall -c -m 644 include/bitcoin/node.hpp '/usr/local/include/bitcoin'
 /opt/homebrew/bin/gmkdir -p '/usr/local/include/bitcoin/node'
 /opt/homebrew/bin/ginstall -c -m 644 include/bitcoin/node/block_arena.hpp include/bitcoin/node/block_memory.hpp include/bitcoin/node/chase.hpp include/bitcoin/node/configuration.hpp include/bitcoin/node/define.hpp include/bitcoin/node/error.hpp include/bitcoin/node/events.hpp include/bitcoin/node/full_node.hpp include/bitcoin/node/parser.hpp include/bitcoin/node/settings.hpp include/bitcoin/node/version.hpp '/usr/local/include/bitcoin/node'
 /opt/homebrew/bin/gmkdir -p '/usr/local/include/bitcoin/node/chasers'
 /opt/homebrew/bin/ginstall -c -m 644 include/bitcoin/node/chasers/chaser.hpp include/bitcoin/node/chasers/chaser_block.hpp include/bitcoin/node/chasers/chaser_check.hpp include/bitcoin/node/chasers/chaser_confirm.hpp include/bitcoin/node/chasers/chaser_header.hpp include/bitcoin/node/chasers/chaser_organize.hpp include/bitcoin/node/chasers/chaser_snapshot.hpp include/bitcoin/node/chasers/chaser_storage.hpp include/bitcoin/node/chasers/chaser_template.hpp include/bitcoin/node/chasers/chaser_transaction.hpp include/bitcoin/node/chasers/chaser_validate.hpp include/bitcoin/node/chasers/chasers.hpp '/usr/local/include/bitcoin/node/chasers'
 /opt/homebrew/bin/gmkdir -p '/usr/local/include/bitcoin/node/impl/chasers'
 /opt/homebrew/bin/ginstall -c -m 644 include/bitcoin/node/impl/chasers/chaser_organize.ipp '/usr/local/include/bitcoin/node/impl/chasers'
 /opt/homebrew/bin/gmkdir -p '/usr/local/include/bitcoin/node/protocols'
 /opt/homebrew/bin/ginstall -c -m 644 include/bitcoin/node/protocols/protocol.hpp include/bitcoin/node/protocols/protocol_block_in.hpp include/bitcoin/node/protocols/protocol_block_in_31800.hpp include/bitcoin/node/protocols/protocol_block_out.hpp include/bitcoin/node/protocols/protocol_header_in_31800.hpp include/bitcoin/node/protocols/protocol_header_in_70012.hpp include/bitcoin/node/protocols/protocol_header_out_31800.hpp include/bitcoin/node/protocols/protocol_header_out_70012.hpp include/bitcoin/node/protocols/protocol_observer.hpp include/bitcoin/node/protocols/protocol_performer.hpp include/bitcoin/node/protocols/protocol_transaction_in.hpp include/bitcoin/node/protocols/protocol_transaction_out.hpp include/bitcoin/node/protocols/protocols.hpp '/usr/local/include/bitcoin/node/protocols'
 /opt/homebrew/bin/gmkdir -p '/usr/local/include/bitcoin/node/sessions'
 /opt/homebrew/bin/ginstall -c -m 644 include/bitcoin/node/sessions/attach.hpp include/bitcoin/node/sessions/session.hpp include/bitcoin/node/sessions/session_inbound.hpp include/bitcoin/node/sessions/session_manual.hpp include/bitcoin/node/sessions/session_outbound.hpp include/bitcoin/node/sessions/sessions.hpp '/usr/local/include/bitcoin/node/sessions'
 /opt/homebrew/bin/gmkdir -p '/usr/local/lib/pkgconfig'
 /opt/homebrew/bin/ginstall -c -m 644 libbitcoin-node.pc '/usr/local/lib/pkgconfig'
 /opt/homebrew/bin/gmkdir -p '/usr/local/etc/libbitcoin'
 /opt/homebrew/bin/ginstall -c -m 644 data/bn.cfg '/usr/local/etc/libbitcoin'

real    4m3.017s
user    20m28.493s
sys 1m55.896s
evoskuil commented 1 day ago

also the script requires you to run it as super user

We never run install.sh with sudo and recommend against it.

evoskuil commented 1 day ago

std::unary_function is a deprecated and subsequently non-existent std lib feature used by some versions of boost. We see this as a warning in some build configurations.

evoskuil commented 1 day ago

Left a comment, it'd be cool to pass install.sh an optional --skip-tests parameter or a similar solution. If you like the idea or have a similar workaround I'll be happy to code it.

We don't want the tests to be disabled in the build script. We just need to resolve whatever issue is there. Did you attempt a statically-linked build?

dyld[57186]: Library not loaded: @rpath/libboost_unit_test_framework.dylib
evoskuil commented 1 day ago

It looks like the only issues building for arm64 are getting the boost dependency right (which is always the most difficult build issue), and resolving the test linkage. Thanks a lot for the info!

gubatron commented 1 day ago

We never run install.sh with sudo and recommend against it.

Agreed, it forced me to as it needed to write in /usr/local for some reason.

We don't want the tests to be disabled in the build script. We just need to resolve whatever issue is there. Did you attempt a statically-linked build?

Nope, but my guess is that maybe it had something to do with building optional testing libraries in boost.

evoskuil commented 1 day ago

Why did you build boost independently of the install.sh?

gubatron commented 1 day ago

because it would stop building when trying to configure libbitcoin-system, it'd say it couldn't find boost. This I solved passing --with-boost= and --with-boost-libdir=

evoskuil commented 19 hours ago

I just installed Sequoia, Xcode and CL tools, Brew, and then successfully built libbitcoin-system as follows:

% brew install autoconf automake libtool pkgconfig wget

% wget https://raw.githubusercontent.com/libbitcoin/libbitcoin-node/master/install.sh
% chmod +x install.sh
% CXXFLAGS="-O3 -DBOOST_NO_CXX98_FUNCTION_BASE" ./install-node.sh --prefix=/Users/bigmac/Prefix --build-boost --disable-shared --enable-isystem --enable-avx512 --enable-avx2 --enable-sse4

This was on x86_64, not arm64 (maybe why tests pass), but this difference probably doesn’t affect the build process based on the description above. Everything builds clean up to libbitcoin-node (two unused var warnings in network), where I hit a bunch of boost errors. Working on that now.

evoskuil commented 19 hours ago

Removing --enable-isystem fixed the break (can cause conflicts) but does then allow a lot of boost warnings. Tests passing (your test fails are probs arm related).