install.sh fails on FreeBSD 12.1

[ghost]

I'm having issues when attempting to install joinmarket on FreeBSD 12.1. The docs say that install.sh should only be used for installing on linux and macOS, but I was also told (in a separate issue) that I should still use it.

Steps to reproduce:

1. create new 12.1 jail (not relevant to this)
2. install and configure bitcoin-core (not relevant to this)
3. run the following commands:

   pkg install -y autotools bash gcc gmake libsodium pkgconf python py37-sqlite3 py37-virtualenv
   git clone https://github.com/Joinmarket-Org/joinmarket-clientserver /usr/local/bin/joinmarket
   cd /usr/local/bin/joinmarket
   ./install.sh

Output: (after answering "n" to installing qt dependencies)

        INFO: Joinmarket-Qt for GUI Taker and Tumbler modes is available.
        Install Qt dependencies (~160mb) ? [y|n] : n
OS can not be determined. Trying to build.
Running virtualenv with interpreter /usr/local/bin/python3
Already using interpreter /usr/local/bin/python3
Using base prefix '/usr/local'
New python executable in /usr/local/bin/joinmarket/jmvenv/bin/python3
Also creating executable in /usr/local/bin/joinmarket/jmvenv/bin/python
Installing setuptools, pip, wheel...
done.
Requirement already up-to-date: pip in ./jmvenv/lib/python3.7/site-packages (20.2.3)
Requirement already up-to-date: setuptools in ./jmvenv/lib/python3.7/site-packages (50.3.0)
/usr/local/bin/joinmarket/deps /usr/local/bin/joinmarket
/usr/local/bin/joinmarket/deps/cache /usr/local/bin/joinmarket/deps /usr/local/bin/joinmarket
sha256: 0d9540b13ffcd7cd44cc361b8744b93d88aa76ba.tar.gz: No such file or directory
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   164  100   164    0     0    352      0 --:--:-- --:--:-- --:--:--   352
100  168k    0  168k    0     0   172k      0 --:--:-- --:--:-- --:--:-- 1619k
SHA256 (0d9540b13ffcd7cd44cc361b8744b93d88aa76ba.tar.gz) = 0803d2dddbf6dd702c379118f066f638bcef6b07eea959f12d31ad2f4721fbe1
/usr/local/bin/joinmarket/deps /usr/local/bin/joinmarket
/usr/local/bin/joinmarket/deps/secp256k1-0d9540b13ffcd7cd44cc361b8744b93d88aa76ba /usr/local/bin/joinmarket/deps /usr/local/bin/joinmarket
make: option requires an argument -- j
usage: make [-BeikNnqrstWwX]
            [-C directory] [-D variable] [-d flags] [-f makefile]
            [-I directory] [-J private] [-j max_jobs] [-m directory] [-T file]
            [-V variable] [-v variable] [variable=value] [target ...]
libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, 'build-aux'.
libtoolize: copying file 'build-aux/ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'build-aux/m4'.
libtoolize: copying file 'build-aux/m4/libtool.m4'
libtoolize: copying file 'build-aux/m4/ltoptions.m4'
libtoolize: copying file 'build-aux/m4/ltsugar.m4'
libtoolize: copying file 'build-aux/m4/ltversion.m4'
libtoolize: copying file 'build-aux/m4/lt~obsolete.m4'
configure.ac:10: installing 'build-aux/compile'
configure.ac:5: installing 'build-aux/config.guess'
configure.ac:5: installing 'build-aux/config.sub'
configure.ac:9: installing 'build-aux/install-sh'
configure.ac:9: installing 'build-aux/missing'
Makefile.am: installing 'build-aux/depcomp'
parallel-tests: installing 'build-aux/test-driver'
checking build system type... amd64-unknown-freebsd12.1
checking host system type... amd64-unknown-freebsd12.1
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... build-aux/install-sh -c -d
checking for gawk... no
checking for mawk... no
checking for nawk... nawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking how to print strings... printf
checking whether make supports the include directive... no
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking whether gcc understands -c and -o together... yes
checking dependency style of gcc... none
checking for a sed that does not truncate output... /usr/bin/sed
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for fgrep... /usr/bin/grep -F
checking for ld used by gcc... /usr/local/bin/ld
checking if the linker (/usr/local/bin/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... /usr/local/bin/nm -B
checking the name lister (/usr/local/bin/nm -B) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 196608
checking how to convert amd64-unknown-freebsd12.1 file names to amd64-unknown-freebsd12.1 format... func_convert_file_noop
checking how to convert amd64-unknown-freebsd12.1 file names to toolchain format... func_convert_file_noop
checking for /usr/local/bin/ld option to reload object files... -r
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for dlltool... no
checking how to associate runtime and link libraries... printf %s\n
checking for ar... ar
checking for archiver @FILE support... no
checking for strip... strip
checking for ranlib... ranlib
checking command to parse /usr/local/bin/nm -B output from gcc object... ok
checking for sysroot... no
checking for a working dd... /bin/dd
checking how to truncate binary pipes... /bin/dd bs=4096 count=1
checking for mt... mt
checking if mt is a manifest tool... no
checking how to run the C preprocessor... gcc -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for dlfcn.h... yes
checking for objdir... .libs
checking if gcc supports -fno-rtti -fno-exceptions... no
checking for gcc option to produce PIC... -fPIC -DPIC
checking if gcc PIC flag -fPIC -DPIC works... yes
checking if gcc static flag -static works... yes
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (/usr/local/bin/ld) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... freebsd12.1 ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... no
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking whether make supports nested variables... (cached) yes
checking for pkg-config... /usr/local/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for ar... /usr/bin/ar
checking for ranlib... /usr/bin/ranlib
checking for strip... /usr/bin/strip
checking for gcc... gcc
checking whether we are using the GNU C compiler... (cached) yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... (cached) none needed
checking whether gcc understands -c and -o together... (cached) yes
checking dependency style of gcc... (cached) none
checking how to run the C preprocessor... gcc -E
checking for gcc option to accept ISO C89... (cached) none needed
checking dependency style of gcc... none
checking if gcc supports -std=c89 -pedantic -Wall -Wextra -Wcast-align -Wnested-externs -Wshadow -Wstrict-prototypes -Wno-unused-function -Wno-long-long -Wno-overlength-strings... yes
checking if gcc supports -fvisibility=hidden... yes
checking for __int128... yes
checking if native gcc supports -Wall -Wextra -Wno-unused-function... yes
checking for working native compiler: gcc... yes
checking for x86_64 assembly availability... yes
checking gmp.h usability... yes
checking gmp.h presence... yes
checking for gmp.h... yes
checking for __gmpz_init in -lgmp... yes
checking for CRYPTO... no
checking openssl/crypto.h usability... yes
checking openssl/crypto.h presence... yes
checking for openssl/crypto.h... yes
checking for main in -lcrypto... yes
checking for EC functions in libcrypto... yes
checking whether byte ordering is bigendian... no
configure: ******
configure: WARNING: experimental build
configure: Experimental features do not have stable APIs or properties, and may not be safe for production use.
configure: Building ECDH module: yes
configure: ******
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating libsecp256k1.pc
config.status: creating src/libsecp256k1-config.h
config.status: executing depfiles commands
config.status: error: in `/usr/local/bin/joinmarket/deps/secp256k1-0d9540b13ffcd7cd44cc361b8744b93d88aa76ba':
config.status: error: Something went wrong bootstrapping makefile fragments
    for automatic dependency tracking.  If GNU make was not used, consider
    re-running the configure script with MAKE="gmake" (or whatever is
    necessary).  You can also try re-running configure with the
    '--disable-dependency-tracking' option to at least be able to build
    the package (albeit without support for automatic dependency tracking).
See `config.log' for more details
make: option requires an argument -- j
usage: make [-BeikNnqrstWwX]
            [-C directory] [-D variable] [-d flags] [-f makefile]
            [-I directory] [-J private] [-j max_jobs] [-m directory] [-T file]
            [-V variable] [-v variable] [variable=value] [target ...]
make: option requires an argument -- j
usage: make [-BeikNnqrstWwX]
            [-C directory] [-D variable] [-d flags] [-f makefile]
            [-I directory] [-J private] [-j max_jobs] [-m directory] [-T file]
            [-V variable] [-v variable] [variable=value] [target ...]
libsecp256k1 was not built. Exiting.

To make the

make: option requires an argument -- j

errors go away something similar to the following should probably be used (I believe this is different from how it is done on linux, which would probably explain it)

make -j$(sysctl -n hw.ncpu)

However, I'm not quite sure where to put it. Also, I'm unclear whether this would solve it, or if it is an unrelated error.

Just for your information, checking out a39991a before installing allows the install to finish without errors.

[kristapsk]

Just for your information, checking out a39991a before installing allows the install to finish without errors.

Looks like you are trying with some older JM release (pre-0.7.0)? What that commit does, it uses GNU make instead of FreeBSD make in case of FreeBSD.

[ghost]

The errors I get is when not checking out any particular commit (in other words the most recent commit) or if I check out v0.7.0, no difference. I only included the reference to the earlier commit to show at least one point in time where it was working

[kristapsk]

That kinda makes no sense.... That commit is three months old and is included in both current master and v0.7.0. I will try to get my hands on a FreeBSD VM to test.

[ghost]

Hmm, I think you might be misunderstanding me…

If I try to run install.sh either at the current commit, or the v0.7.0 tag, I get the errors I included above. However, if I check out a39991a, the script will finish without errors. In other words, something changed between a39991a and now that broke things.

I can try bisecting (never used the bisect command, but I'll figure it out) and try to pinpoint the exact commit where things broke. I just wanted to start things off by simply stating what happened since things got so confusing last time I commented.

[kristapsk]

Yes, but nothing has been changed in regard of make/gmake and install.sh in general since that commit.

[ghost]

Hmm, then that error is probably unrelated. It probably just ignores the "-j" flag and builds with a single core.

I'm going to venture a guess that the actual show stopping error is probably something related to secp256k library. (due to this part of the output)

…
config.status: error: in `/usr/local/bin/joinmarket/deps/secp256k1-0d9540b13ffcd7cd44cc361b8744b93d88aa76ba':
config.status: error: Something went wrong bootstrapping makefile fragments
    for automatic dependency tracking.  If GNU make was not used, consider
    re-running the configure script with MAKE="gmake" (or whatever is
    necessary).  You can also try re-running configure with the
    '--disable-dependency-tracking' option to at least be able to build
    the package (albeit without support for automatic dependency tracking).
…

When I've attempted to install without using the install script I've been having issues due to secp256. Actually got it working a few hours ago, but in a hacky kind of a way. To get it to work I actually configure the venv before building and installing the secp256k library. Doing it that way allows me to install everything without errors (without using the install.sh script)

[ghost]

hmm, actually just remembered something… to get the secp256k library to build on FreeBSD I had to use the "--disable-dependency-tracking" flag, I completely forgot about that (added it to my script more than a month ago)

This is how I've been building it:

    git clone https://github.com/bitcoin-core/secp256k1 /tmp/secp256k1
    cd /tmp/secp256k1
    ./autogen.sh
    ./configure --disable-dependency-tracking \
                --enable-module-recovery \
                --disable-jni \
                --enable-experimental \
                --enable-module-ecdh \
                --enable-benchmark=no
    gmake 
    gmake check 
    gmake install

[kristapsk]

Your problem is that for some reason it uses make, not gmake, but current version of install.sh should always force gmake on FreeBSD.

[kristapsk]

Looks like I was able to reproduce the issue and found where is the problem.

[kristapsk]

@bektar , could you test #700? It solved this issue in my tests.

[ghost]

Excellent, worked without errors. Thanks mate!