autotools toolchain

[jsharpe]

We should provide an autotools toolchain providing the common GNU tools, m4, autoconf and automake so that the configure_make rule can hermetically consume these components without relying on the host environment having working or new enough versions of these components.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days. Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_foreign_cc!

[uhthomas]

Hi @jsharpe. What is the status of this issue? Is there any guidance on how best to achieve this?

Thanks :)

[jsharpe]

@uhthomas there are toolchain definitions for each of the tools but the rules do not yet make use of them. The main issue with completing this is that autoconf is not relocatable in its build - it generates many files with hardcoded paths and so to build a hermetic toolchain of all the autotools components would require heavy patching of autoconf. How far I got is in examples/third_party/autotools but I don't currently have much time available to look at this.

I did experiment with hacking the configure script to pass in /proc/self/cwd as part of the install prefix to attempt to fix this but IIRC I ran into some limitations with resolving symlinks in this approach.

My current advise on this is to execute your build within a container with a known deployed toolchain to make this as hermetic as possible.

PRs are welcome though to move towards this goal!

[uhthomas]

Ah, I see, thank you.

Does the example you worked on also support autoreconf? I'd be really interested to help make this better.

[jsharpe]

I don't think we currently have any examples that are using autoreconf. If you have a small package that does use that it'd be useful to get that added to test the autoreconf functionality.

[uhthomas]

Not necessarily a "small" example, but as far as I understand capnproto is using autoreconf. Trying to get this working with rules_foreign_cc at the moment and struggling a bit.

https://github.com/capnproto/capnproto/tree/281aacc18003ae103b8a04ea83d7b6f814c97b8b/c%2B%2B

[jsharpe]

Its also possible that running autoreconf should be the default behaviour - Debian recommend this: https://wiki.debian.org/Autoreconf#Autoreconfing_packages_-_A_Maintainers.27_Guide.

I guess that the only issue is that the configure_make rules aren't necessarily explicitly an autotools project. Maybe we should have an autotools specific rule to distinguish between autotools specific configure_make projects and more generic ones that just follow that pattern e.g. openssl.

[uhthomas]

Sounds reasonable to me. Do you have a recommended path forward for this? As mentioned before, I'd be happy to help where possible. Though my knowledge of the C/C++ toolchains aren't too strong.

[jsharpe]

As far as I'm aware all the code paths should already be in place for autoreconf so likely what I'm suggesting is just a macro around the existing rule with some appropriate defaults set

[uhthomas]

Cool!

For what it's worth I tried building with the autotools example and didn't have too much luck.

+ autoreconf
configure.ac:70: warning: The macro `AC_LANG_C' is obsolete.
configure.ac:70: You should run autoupdate.
./lib/autoconf/c.m4:72: AC_LANG_C is expanded from...
m4/acx_pthread.m4:66: ACX_PTHREAD is expanded from...
lib/m4sugar/m4sh.m4:594: AS_CASE is expanded from...
configure.ac:70: the top level
configure.ac:70: warning: The macro `AC_TRY_LINK' is obsolete.
configure.ac:70: You should run autoupdate.
./lib/autoconf/general.m4:2920: AC_TRY_LINK is expanded from...
m4/acx_pthread.m4:66: ACX_PTHREAD is expanded from...
lib/m4sugar/m4sh.m4:594: AS_CASE is expanded from...
configure.ac:70: the top level
configure.ac:70: warning: The macro `AC_TRY_LINK' is obsolete.
configure.ac:70: You should run autoupdate.
./lib/autoconf/general.m4:2920: AC_TRY_LINK is expanded from...
lib/m4sugar/m4sh.m4:692: _AS_IF_ELSE is expanded from...
lib/m4sugar/m4sh.m4:699: AS_IF is expanded from...
./lib/autoconf/general.m4:2894: _AC_LINK_IFELSE is expanded from...
./lib/autoconf/general.m4:2911: AC_LINK_IFELSE is expanded from...
./lib/autoconf/general.m4:2920: AC_TRY_LINK is expanded from...
m4/acx_pthread.m4:66: ACX_PTHREAD is expanded from...
lib/m4sugar/m4sh.m4:594: AS_CASE is expanded from...
configure.ac:70: the top level
configure.ac:6: error: required directory ./build-aux does not exist
configure.ac:65: error: required file 'build-aux/compile' not found
configure.ac:65:   'automake --add-missing' can install 'compile'
configure.ac:70: error: required file 'build-aux/config.guess' not found
configure.ac:70:   'automake --add-missing' can install 'config.guess'
configure.ac:70: error: required file 'build-aux/config.sub' not found
autoreconf: error: automake failed with exit status: 1

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days. Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_foreign_cc!

[gozzarda]

@jsharpe sorry for the bump, but I was wondering if this was considered solved at this point? What is the current idiomatic way to build an autotools based dependency?

[jsharpe]

@gozzarda the rules rely on the tools being installed on the host machine; I'd started trying to get bazel to build the tools from source but got stuck on building a relocatable version of autoconf; it hardcodes too many paths internally so I got stuck there with getting a hermetic autotools toolchain built.

Any contributions in fixing this are gladly received as I have very limited time (if any!) to work on rules_foreign_cc at the moment.

[gozzarda]

@jsharpe Thanks for the update. That makes more sense to me. Sounds like builds should be working if you have the dependencies installed but not hermetic?

I too am pretty short on time lately, but have been looking into this because it could benefit a work project, so I may be able to convince work to allocate some time for me to look into this, but don't hold your breath.

For those looking for a combined solution in the meantime, do you still recommend running in a container? I assume a Nix shell would suffice, though the problems you encountered sound like they may also cause Nix headaches so I will have to check how autoconf behaves in Nix.

Thank you for taking what little time you have to respond to this thread. It is appreciated.