Autotools build works only in the source directory

[jaganmn]

make fails immediately if configure was not run in the source directory.

$ tar -xf flint-3.1.3.tar.xz
$ cd flint-3.1.3
$ LIBTOOLIZE=glibtoolize ./bootstrap.sh
autoreconf: export WARNINGS=all
autoreconf: Entering directory '.'
autoreconf: configure.ac: not using Gettext
autoreconf: running: aclocal --force 
autoreconf: configure.ac: tracing
autoreconf: running: glibtoolize --copy --force
glibtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, 'config'.
glibtoolize: copying file 'config/ltmain.sh'
glibtoolize: Consider adding 'AC_CONFIG_MACRO_DIRS([m4])' to configure.ac,
glibtoolize: and rerunning glibtoolize and aclocal.
glibtoolize: Consider adding '-I m4' to ACLOCAL_AMFLAGS in Makefile.am.
autoreconf: configure.ac: not using Intltool
autoreconf: configure.ac: not using Gtkdoc
autoreconf: running: aclocal --force 
autoreconf: running: /usr/local/bin/autoconf --force
autoreconf: running: /usr/local/bin/autoheader --force
autoreconf: configure.ac: not using Automake
autoreconf: 'config/config.sub' is updated
autoreconf: 'config/config.guess' is updated
autoreconf: 'config/install-sh' is updated
autoreconf: Leaving directory '.'
$ mkdir _build && cd _build && ../configure && make
checking build system type... aarch64-apple-darwin23.6.0
checking host system type... aarch64-apple-darwin23.6.0
checking how to print strings... printf
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether the compiler supports GNU C... yes
checking whether gcc accepts -g... yes
checking for gcc option to enable C11 features... none needed
checking for a sed that does not truncate output... /usr/bin/sed
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for fgrep... /usr/bin/grep -F
checking for ld used by gcc... /Library/Developer/CommandLineTools/usr/bin/ld
checking if the linker (/Library/Developer/CommandLineTools/usr/bin/ld) is GNU ld... no
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 786432
checking how to convert aarch64-apple-darwin23.6.0 file names to aarch64-apple-darwin23.6.0 format... func_convert_file_noop
checking how to convert aarch64-apple-darwin23.6.0 file names to toolchain format... func_convert_file_noop
checking for /Library/Developer/CommandLineTools/usr/bin/ld option to reload object files... -r
checking for file... file
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for dlltool... no
checking how to associate runtime and link libraries... printf %s\n
checking for ar... ar
checking for archiver @FILE support... no
checking for strip... strip
checking for ranlib... ranlib
checking for gawk... no
checking for mawk... no
checking for nawk... no
checking for awk... awk
checking command to parse /usr/bin/nm -B output from gcc object... ok
checking for sysroot... no
checking for a working dd... /usr/local/bin/dd
checking how to truncate binary pipes... /usr/local/bin/dd bs=4096 count=1
checking for mt... no
checking if : is a manifest tool... no
checking for dsymutil... dsymutil
checking for nmedit... nmedit
checking for lipo... lipo
checking for otool... otool
checking for otool64... no
checking for -single_module linker flag... ld: warning: -single_module is obsolete
no
checking for -exported_symbols_list linker flag... yes
checking for -force_load linker flag... yes
checking for stdio.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for strings.h... yes
checking for sys/stat.h... yes
checking for sys/types.h... yes
checking for unistd.h... yes
checking for dlfcn.h... yes
checking for objdir... .libs
checking if gcc supports -fno-rtti -fno-exceptions... yes
checking for gcc option to produce PIC... -fno-common -DPIC
checking if gcc PIC flag -fno-common -DPIC works... yes
checking if gcc static flag -static works... no
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (/Library/Developer/CommandLineTools/usr/bin/ld) supports shared libraries... yes
checking dynamic linker characteristics... darwin23.6.0 dyld
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking for a race-free mkdir -p... /usr/local/bin/mkdir -p
checking for inline... inline
checking whether byte ordering is bigendian... no
checking if memory is strongly-ordered... unsure
checking how to run the C preprocessor... gcc -E
checking for gmp.h... yes
checking if version of GMP is greater than 6.2.1... yes
checking if GMP defines mp_limb_t as unsigned long long int... no
checking for mpfr.h... yes
checking if version of MPFR is greater than 4.1.0... yes
checking for desired ABI... 64
checking for stdarg.h... yes
checking for math.h... yes
checking for float.h... yes
checking for errno.h... yes
checking for fenv.h... yes
checking for alloca.h... yes
checking for malloc.h... no
checking for sys/param.h... yes
checking for windows.h... no
checking for pthread_np.h... no
checking for pthread.h... yes
checking for library containing atan2... none required
checking for library containing __gmpz_init... -lgmp
checking for library containing __gmpn_gcd_11... none required
checking for library containing __gmpn_div_q... none required
checking for library containing __gmpn_mul_basecase... none required
checking for library containing __gmpn_modexact_1_odd... no
checking for library containing __gmpn_addmul_2... no
checking for library containing mpfr_init... -lmpfr
checking for library containing mpfr_round_p... none required
checking for library containing mpfr_mulhigh_n... none required
checking for library containing mpfr_sqrhigh_n... none required
checking whether gcc accepts -pthread... yes
checking if cpu_set_t is supported... no
checking if alloca works... yes
checking for aligned_alloc... yes
checking for _aligned_malloc... no
checking whether gcc accepts -Werror=unknown-warning-option... yes
checking whether gcc accepts -Wall... yes
checking whether gcc accepts -Wno-stringop-overread... no
checking whether gcc accepts -Wno-stringop-overflow... no
checking whether gcc accepts -Werror=implicit-function-declaration... yes
checking whether gcc accepts -O3... yes
checking whether gcc accepts -march=native... yes
checking whether gcc accepts -std=c11... yes
checking whether gcc accepts -pedantic... yes
checking for host architecture... apple-m1
checking for arm_neon.h... yes
checking if system have required Arm instruction set for fft_small... yes
checking if system can use FLINT's fft_small module... yes
checking if system uses System V ABI... yes
checking if system can use FLINT's assembly... no
configure: WARNING: FLINT currently only have assembly routines for System V-type systems for x86_64 processors with ADX instruction set. Disabling assembly...
configure: creating ./config.status
config.status: creating src/gmpcompat.h
config.status: creating src/fft_tuning.h
config.status: creating src/fmpz/fmpz.c
config.status: creating Makefile
config.status: creating flint.pc
config.status: creating src/flint.h
config.status: creating src/config.h
config.status: creating src/flint-config.h
config.status: executing libtool commands
make: *** No rule to make target 'Makefile.in', needed by 'Makefile'.  Stop.

The cause seems to be kludge in Makefile.in:

config.status: configure
    ./config.status --recheck

Makefile: Makefile.in config.status
    ./config.status $@

flint.pc: flint.pc.in config.status
    ./config.status $@

src/flint.h: src/flint.h.in config.status
    ./config.status $@

libtool: config.status
    ./config.status $@

src/fft_tuning.h: @FFT_TUNING_IN@ config.status
    ./config.status $@

src/fmpz/fmpz.c: @FMPZ_C_IN@ config.status
    ./config.status $@

src/gmpcompat.h: src/@GMPCOMPAT_H_IN@ config.status
    ./config.status $@

Why are there rules for making x from x.in, when that is already done by configure before make is ever run? From an outsider perspective, the build system is nonconforming, if not broken.

For context, I am writing an R package that provides an R interface to FLINT. Due to the above issue and due to the implicit dependency on users or administrators having Autotools installed (in order to update configuration files themselves, when that is typically done by maintainers before tarballs are built), it is unlikely that CRAN will maintain installations of FLINT on its check platforms. That is a huge barrier to developers submitting to CRAN R packages that link FLINT, because it means that packages must bundle the FLINT sources.

[albinahlback]

Out of tree builds has been an issue for over 10 years (see #30).

Why are there rules for making x from x.in, when that is already done by configure before make is ever run? From an outsider perspective, the build system is nonconforming, if not broken.

These rules exists to update files like flint.h in case flint.h.in gets modified after running configure. They should not be in use if user is just running make after configuring.

For context, I am writing an R package that provides an R interface to FLINT. Due to the above issue and due to the implicit dependency on users or administrators having Autotools installed (in order to update configuration files themselves, when that is typically done by maintainers before tarballs are built), it is unlikely that CRAN will maintain installations of FLINT on its check platforms. That is a huge barrier to developers submitting to CRAN R packages that link FLINT, because it means that packages must bundle the FLINT sources.

There should be no need to have Autotools if you aim to just install FLINT. Any release available here on Github should not require it.

[albinahlback]

If out of tree builds is an issue for someone, let us know. I thought of fixing it this spring at the FLINT workshop, but no one at the time would really benefit from it, so I didn't fix it.

[jaganmn]

INSTALL explicitly lists Autotools as a build dependency. If that is not the case for building release tarballs, then that could be mentioned. The situation was particularly unclear to me because all of these recent release tarballs have configure containing PACKAGE_VERSION='3.1.0':

https://github.com/flintlib/flint/releases/download/v3.1.3/flint-3.1.3.tar.xz https://github.com/flintlib/flint/releases/download/v3.1.2/flint-3.1.2.tar.xz https://github.com/flintlib/flint/releases/download/v3.1.1/flint-3.1.1.tar.xz

and Autoconf was truly required to obtain an internally consistent build.

If out of tree builds is an issue for someone, let us know.

Yes, there are automated pipelines at CRAN that routinely build external libraries from sources out of tree (see, e.g., https://github.com/r-macos/recipes). Since Autotools was designed in the first place to handle out of tree builds, it has not been an issue before.

[albinahlback]

INSTALL explicitly lists Autotools as build dependency. If that is not the case for building release tarballs, then that could be mentioned.

Agreed! Thanks for bringing this up, will fix this.

Yes, there are automated build systems at CRAN that routinely build external libraries from sources out of tree (see, e.g., https://github.com/r-macos/recipes). Since Autotools was designed in the first place to handle out of tree builds, it has not been an issue before.

Alright. I believe I will fix this for FLINT v3.2.0 (I would guess it will be released after the next workshop in December or January, but we haven't really talked about the next release).

[edgarcosta]

The situation was particularly unclear to me because all of these recent release tarballs have configure containing PACKAGE_VERSION='3.1.0':

That issue has been fixed in our release script.

[albinahlback]

Closing this as #30 was solved. Please reopen if something was not settled.

[jaganmn]

Thanks. Is there any hope of creating a minimal diff against the latest release, supporting an out of tree build? That would allow me to move forward with getting FLINT installed on CRAN's machines. If there's been huge divergence, then I'll understand if I'm just out of luck here ...

[albinahlback]

It shouldn't be too hard to do, but I get conflicts in Makefile.in and configure.ac when trying to cherry-pick the conflict. Moreover, I believe we are moving towards v3.2.0, and I'm not so keen on maintaining older versions unless there are bugs. I would expect a new FLINT release within the next half year.

[jaganmn]

Under macOS, make check from a build directory different from the source directory fails with a linker error:

$ git clone git@github.com:flintlib/flint.git
$ cd flint
$ LIBTOOLIZE=glibtoolize ./bootstrap.sh
autoreconf: export WARNINGS=all
autoreconf: Entering directory '.'
autoreconf: configure.ac: not using Gettext
autoreconf: running: aclocal 
autoreconf: configure.ac: tracing
autoreconf: running: glibtoolize --copy
glibtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, 'config'.
glibtoolize: copying file 'config/ltmain.sh'
glibtoolize: Consider adding 'AC_CONFIG_MACRO_DIRS([m4])' to configure.ac,
glibtoolize: and rerunning glibtoolize and aclocal.
glibtoolize: Consider adding '-I m4' to ACLOCAL_AMFLAGS in Makefile.am.
autoreconf: configure.ac: not using Intltool
autoreconf: configure.ac: not using Gtkdoc
autoreconf: running: aclocal 
autoreconf: running: /usr/local/bin/autoconf
autoreconf: running: /usr/local/bin/autoheader
autoreconf: configure.ac: not using Automake
autoreconf: 'config/install-sh' is created
autoreconf: Leaving directory '.'
$ mkdir _build && cd _build && ../configure
checking build system type... applem1-apple-darwin23.6.0
checking host system type... applem1-apple-darwin23.6.0
checking how to print strings... printf
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether the compiler supports GNU C... yes
checking whether gcc accepts -g... yes
checking for gcc option to enable C11 features... none needed
checking for a sed that does not truncate output... /usr/bin/sed
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for fgrep... /usr/bin/grep -F
checking for ld used by gcc... /Library/Developer/CommandLineTools/usr/bin/ld
checking if the linker (/Library/Developer/CommandLineTools/usr/bin/ld) is GNU ld... no
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 786432
checking how to convert applem1-apple-darwin23.6.0 file names to applem1-apple-darwin23.6.0 format... func_convert_file_noop
checking how to convert applem1-apple-darwin23.6.0 file names to toolchain format... func_convert_file_noop
checking for /Library/Developer/CommandLineTools/usr/bin/ld option to reload object files... -r
checking for file... file
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for dlltool... no
checking how to associate runtime and link libraries... printf %s\n
checking for ar... ar
checking for archiver @FILE support... no
checking for strip... strip
checking for ranlib... ranlib
checking for gawk... no
checking for mawk... no
checking for nawk... no
checking for awk... awk
checking command to parse /usr/bin/nm -B output from gcc object... ok
checking for sysroot... no
checking for a working dd... /usr/local/bin/dd
checking how to truncate binary pipes... /usr/local/bin/dd bs=4096 count=1
checking for mt... no
checking if : is a manifest tool... no
checking for dsymutil... dsymutil
checking for nmedit... nmedit
checking for lipo... lipo
checking for otool... otool
checking for otool64... no
checking for -single_module linker flag... ld: warning: -single_module is obsolete
no
checking for -exported_symbols_list linker flag... yes
checking for -force_load linker flag... yes
checking for stdio.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for strings.h... yes
checking for sys/stat.h... yes
checking for sys/types.h... yes
checking for unistd.h... yes
checking for dlfcn.h... yes
checking for objdir... .libs
checking if gcc supports -fno-rtti -fno-exceptions... yes
checking for gcc option to produce PIC... -fno-common -DPIC
checking if gcc PIC flag -fno-common -DPIC works... yes
checking if gcc static flag -static works... no
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (/Library/Developer/CommandLineTools/usr/bin/ld) supports shared libraries... yes
checking dynamic linker characteristics... darwin23.6.0 dyld
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... no
checking for a race-free mkdir -p... /usr/local/bin/mkdir -p
checking how to run the C preprocessor... gcc -E
checking if compiler is GCC... no
checking if compiler is Clang... yes
checking for inline... inline
checking whether byte ordering is bigendian... no
checking if memory is strongly-ordered... unsure
checking for gmp.h... yes
checking if version of GMP is greater than 6.2.1... yes
checking if GMP defines mp_limb_t as unsigned long long int... no
checking for mpfr.h... yes
checking if version of MPFR is greater than 4.1.0... yes
checking for desired ABI... 64
checking for stdarg.h... yes
checking for math.h... yes
checking for float.h... yes
checking for errno.h... yes
checking for fenv.h... yes
checking for alloca.h... yes
checking for malloc.h... no
checking for sys/param.h... yes
checking for windows.h... no
checking for pthread_np.h... no
checking for pthread.h... yes
checking for library containing atan2... none required
checking for library containing __gmpz_init... -lgmp
checking for library containing __gmpn_mul_basecase... none required
checking for library containing __gmpn_gcd_11... none required
checking for library containing __gmpn_div_q... none required
checking for library containing __gmpn_invert_limb... none required
checking for library containing __gmpn_modexact_1_odd... no
checking for library containing __gmpn_addmul_2... no
checking for library containing __gmpn_add_n_sub_n... none required
checking for library containing __gmpn_add_nc... none required
checking for library containing __gmpn_sub_nc... none required
checking for library containing __gmpn_addlsh1_n... none required
checking for library containing __gmpn_addlsh1_n_ip1... no
checking for library containing __gmpn_rsh1sub_n... none required
checking for library containing __gmpn_rsh1add_n... none required
checking for library containing mpfr_init... -lmpfr
checking for library containing mpfr_round_p... none required
checking for library containing mpfr_mulhigh_n... none required
checking for library containing mpfr_sqrhigh_n... none required
checking whether gcc accepts -pthread... yes
checking if cpu_set_t is supported... no
checking if alloca works... yes
checking for aligned_alloc... yes
checking for _aligned_malloc... no
checking whether gcc accepts -Werror=unknown-warning-option... yes
checking whether gcc accepts -O3... yes
checking whether gcc accepts -pedantic... yes
checking whether gcc accepts -std=c11... yes
checking whether gcc accepts -Werror=implicit-function-declaration... yes
checking whether gcc accepts -Wall... yes
checking whether gcc accepts -Wno-stringop-overread... no
checking whether gcc accepts -Wno-stringop-overflow... no
checking whether gcc accepts -Wno-maybe-uninitialized... no
checking whether gcc accepts -march=armv8.5-a... yes
checking whether gcc accepts -funroll-loops... yes
checking for arm_neon.h... yes
checking if system have required ARM instruction set for fft_small... (cached) yes
checking if system can use FLINT's fft_small module... yes
checking for suitable m4... m4
checking whether assembler supports --noexecstack option... no
checking how to switch to text section... .text
checking how to switch to data section... .data
checking for assembler label suffix... :
checking for assembler global directive... .globl
checking for assembler global directive attribute... 
checking if globals are prefixed by underscore... yes
checking how to switch to read-only data section...     .section    __TEXT,__const
checking for assembler .type directive... 
checking for assembler .size directive... 
checking for assembler local label prefix... L
checking for assembler byte directive... .byte
checking if .align assembly directive is logarithmic... yes
creating config.m4
configure: creating ./config.status
config.status: creating Makefile
config.status: creating flint.pc
config.status: creating src/flint.h
config.status: creating src/config.h
config.status: creating src/flint-config.h
config.status: linking ../src/gmpcompat.h.in to src/gmpcompat.h
config.status: linking ../src/mpn_extras/arm64/flint-mparam.h to src/flint-mparam.h
config.status: linking ../src/fmpz/link/fmpz_single.c to src/fmpz/fmpz.c
config.status: executing libtool commands
$ make
$ make check
/usr/local/bin/mkdir -p build/test
  CC  test/main.c
Undefined symbols for architecture arm64:
  "__flint_rand_clear_gmp_state", referenced from:
      _test_add_ssaaaa in main-e5633d.o
      _test_add_sssaaaaaa in main-e5633d.o
      _test___builtin_clzl in main-e5633d.o
      _test___builtin_ctzl in main-e5633d.o
      _test_flint_fprintf in main-e5633d.o
      _test_memory_manager in main-e5633d.o
      _test_sdiv_qrnnd in main-e5633d.o
      ...
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [Makefile:675: build/test/main] Error 1
$