Open madebr opened 4 years ago
Working on my machine (after cloning your autoconf
branch):
docker run -it -v$(pwd):/home/ubuntu/ conan/gcc9 bash
cd /home/ubuntu/
conan export recipes/autoconf/all/conanfile.py autoconf/2.69@
Let's check package IDs first:
arch=x86
conan info autoconf/2.69@ --profile=default -s=arch=x86 -n id
autoconf/2.69
ID: 634482ae2711f292a960087dc3f4e0b99e4881dc
m4/1.4.18
ID: 9efecfd5d5097a3c59d3dcb372f978c2c3d2cee6
arch=x86_64
conan info autoconf/2.69@ --profile=default -s=arch=x86_64 -n id
autoconf/2.69
ID: 4db1be536558d833e52e862fd84d64d75c2b3656
m4/1.4.18
ID: 2a30b7d6ea2202e5393ebda51c8729dd2162b9f8
And now see which one is used in the conan create
command
conan create recipes/libcurl/all/conanfile.py libcurl/7.71.1@ --build=libcurl --profile:host=default --profile:build=default --settings:build=arch=x86
libcurl/7.71.1: Forced build from source
libcurl/7.71.1 (test package): Installing package
Requirements
libcurl/7.71.1 from local cache - Cache
openssl/1.1.1g from 'conan-center' - Cache
zlib/1.2.11 from 'conan-center' - Cache
Packages
libcurl/7.71.1:eb186ad30eac67be94e708b3027620138a2d5378 - Build
openssl/1.1.1g:6af9cc7cb931c5ad942174fd7838eb655717c709 - Download
zlib/1.2.11:6af9cc7cb931c5ad942174fd7838eb655717c709 - Download
Build requirements
autoconf/2.69 from local cache - Cache
automake/1.16.2 from 'conan-center' - Cache
libtool/2.4.6 from 'conan-center' - Cache
m4/1.4.18 from 'conan-center' - Cache
Build requirements packages
autoconf/2.69:634482ae2711f292a960087dc3f4e0b99e4881dc - Missing
automake/1.16.2:258f6f05ca54813b3d7180161753df7f9ccf7e1d - Download
libtool/2.4.6:1216c48521e2cd3c4b31b9f4fd771694e1f03bc7 - Missing
m4/1.4.18:9efecfd5d5097a3c59d3dcb372f978c2c3d2cee6 - Missing
Cross-build from 'Linux:x86' to 'Linux:x86_64'
Installing (downloading, building) binaries...
ERROR: Missing binary: autoconf/2.69:634482ae2711f292a960087dc3f4e0b99e4881dc
ERROR: Missing binary: libtool/2.4.6:1216c48521e2cd3c4b31b9f4fd771694e1f03bc7
ERROR: Missing binary: m4/1.4.18:9efecfd5d5097a3c59d3dcb372f978c2c3d2cee6
As you can see, it requires 634482ae2711f292a960087dc3f4e0b99e4881dc
which is the package-ID corresponding to arch=x86
as we have forced with the parameters --profile:build=default --settings:build=arch=x86
. At least settings are properly assigned, I haven't reviewed the recipe itself.
Note.- Pay attention, there is a bug in the output of the profile under
Configuration (profile_build):
, it doesn't update thearch
value from the command line, but it is properly applied to the recipe.
I think I need more information from your output in order to know what is happening there.
autoconf is not really a problem, because it is arch independent. It is a collection of bash, m4 and perl scripts.
I patched autoconf because its use of arch_build
and os_build
might clash with cross building.
The issue shows up with m4, which is a native executable.
Note the --target=x86-linux-gnu
gnu triplet at line 110 for m4 and line 3218 for libtool.
I checked the m4 and libtool build folder in the conan prefix, and they are both built using 64-bit (-m64 flag).
There was no -m32
flag present.
So it seems the gnu triplet is calculated wrongly.
The requested package-Id for m4
is correct as well, in my example it is using the 9efecfd...
which correspond to x86
which is the architecture for the build context. So the error must be in the triplet calculation or in the recipe itself.
Computed triplet is --target=x86-linux-gnu
. According to the m4
recipe it is calculated inside Conan, no value is overridden from the recipe, so the value is taken from the AutoToolsBuildEnvironment::_get_host_build_target_flags
:
if self._os_target and self._arch_target:
try:
target = get_gnu_triplet(self._os_target, self._arch_target, self._compiler)
except ConanException as exc:
self._conanfile.output.warn(str(exc))
target = None
else:
target = None
The values fro self._os_target
and self._arch_target
are taken from the host context, they Linux
and x86
respectively (from the POV of m4, it is generating binaries for the host context). From my understanding, a triplet like x86-linux-gnu
is expected.
What should be the input to the configure
script? Maybe the problem is that it is missing the --host
argument to tell that the resulting binary of m4
should be x86
? (I have very little experience with autotools).
@jgsogo
Those are the canonical triplets, which is checked/generated by autoconf, and can be checked by the configure script.
See https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.69/autoconf.html#Canonicalizing
There exists triplets for: build, host and target.
os/arch/compiler
should be used for the host triplet (--host xxx
),
os_build/arch_build/compiler_build
for the build triplet,
os_target/arch_target/compiler_target
for the target triplet.
For m4, only the host and build triplet matter, as there is no target.
So _os_target
and _arch_target
should not be used.
See my gnu-config pr. That is adding scripts to detect and verify triplets.
In my case I'm trying to get a 32-bit Windows build going on CI and I'm also running into this issue (I think). The build environment is 64-bit Windows. I'm using MSYS2 by setting CONAN_BASH_PATH
and then simply invoke conan -b missing -s arch=x86 -s build_arch=x86_64 -s build_type=Debug cunit/2.1-3@
.
The build_requirements lists libtool
. And so libtool
> automake
> autoconf
> m4
are required. Both libtool
and m4
are built for x86
, not automake
and autoconf
. To me it seems that somehow the build_requirements aren't propagated correctly. Since they're requirements for building, I'd expect conan to simply download the x86_64 versions and use those to compile cunit.
automake
and autoconf
don't contain binaries. So there is no difference between a 32-bit and 64-bit package.
In these recipes, settings.arch
and settings.compiler
are not taken into account for the package id.
https://github.com/conan-io/conan-center-index/blob/2f4af72088cb85eeea87868c908791022eb4fe71/recipes/autoconf/all/conanfile.py#L27-L31
Using the same autoconf
package for both intel architectures should not cause problems.
Entirely correct. autoconf
and automake
are simply downloaded. libtool
and m4
are not. I'd expect these tools simply be download as I'm doing a cross build from x86_64 to x86.
libtool
contains a library and m4
contains an executable.
libtool
has os, arch, compiler, build_type
in its settings (https://github.com/conan-io/conan-center-index/blob/2f4af72088cb85eeea87868c908791022eb4fe71/recipes/libtool/all/conanfile.py#L20), arch_build
is not in it.
Your -s build_arch
(or if you meant -s arch_build
) does not make it cross build.
You need to create a 32-bit profile and a 64-bit profile (see https://docs.conan.io/en/1.35/systems_cross_building/cross_building.html#cross-building-with-conan) and use -pr:b
/-pr:h
.
That does the trick regarding the downloading of packages, thanks! However, that does lead to:
cunit/2.1-3: Calling build()
**********************************************************************
** Visual Studio 2019 Developer Command Prompt v16.8.2
** Copyright (c) 2020 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x86'
cunit/2.1-3:
cunit/2.1-3: ERROR: Package '486ece6aed18ca03ed25554b84cacdd8db4d6674' build failed
cunit/2.1-3: WARN: Build folder C:\Users\k0ekk0ek\.conan\data\cunit\2.1-3\_\_\build\486ece6aed18ca03ed25554b84cacdd8db4d6674
ERROR: cunit/2.1-3: Error in build() method, line 117
with self._build_context():
AttributeError:
And that's caused by self.deps_user_info
being empty (automake
is the specific key it's looking for). Any idea what might be the issue there?
That's a bug in cunit
not supporting cross building. Please open a ticket on the cci bug tracker.
When cross building, a recipe also has self.user_info_build
which should have been used here.
We're currently in the process of reworking a lot of recipes.
@madebr : I am cross building on linux
for arm
. I am finding my m4
packages are not unique per host arch
setting which is the problem you fixed above. I believe I am using your merged fixes for autoconf
that should cause the gnu-triplet to be properly calculated. I am finding the m4
package being fetched is for the build arch
of x86_64
instead of the host arch
of armv7hf
.
Do you have any guidance as to what I might look at next?
I just pulled conan-center-index and manually created and uploaded the pkgs to my artifactory using this hash:
commit 48043fa744b9ffa4db3336cfcd9b8c6b1ea57e2e (HEAD -> master, origin/master)
Author: SpaceIm <30052553+SpaceIm@users.noreply.github.com>
Date: Mon Jul 26 23:04:16 2021 +0200
excerpts from my build
Configuration (profile_host):
[settings]
arch=armv7hf
arch_build=x86_64
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++11
compiler.version=8
os=Linux
os_build=Linux
[options]
[build_requires]
[env]
AR=arm-linux-gnueabihf-ar
AS=arm-linux-gnueabihf-as
CC=arm-linux-gnueabihf-gcc-8
CMAKE_CXX_COMPILER=arm-linux-gnueabihf-g++-8
CMAKE_C_COMPILER=arm-linux-gnueabihf-gcc-8
CONAN_CMAKE_TOOLCHAIN_FILE=/root/.conan/arm32v7_cmake.toolchain
CXX=arm-linux-gnueabihf-g++-8
FC=arm-linux-gnueabihf-gfortran-8
LD=arm-linux-gnueabihf-ld
RANLIB=arm-linux-gnueabihf-ranlib
STRIP=arm-linux-gnueabihf-strip
Configuration (profile_build):
[settings]
arch=x86_64
arch_build=x86_64
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++
compiler.version=7
os=Linux
os_build=Linux
[options]
[build_requires]
[env]
...
Requirements
libalsa/1.1.9 from 'nxc' - Downloaded
Python requires
boost_base/1.69.0@bincrafters/stable
Packages
libalsa/1.1.9:8becccfcd4d3af31e2ed07435c74e18c58a63c1d - Build
Build requirements
autoconf/2.71 from 'my_art' - Downloaded
automake/1.16.3 from 'nxc' - Downloaded
gnu-config/cci.20201022 from 'nxc' - Downloaded
libtool/2.4.6 from 'nxc' - Downloaded
m4/1.4.18 from 'nxc' - Downloaded
Build requirements packages
autoconf/2.71:cb054d0b3e1ca595dc66bc2339d40f1f8f04ab31 - Download
automake/1.16.3:258f6f05ca54813b3d7180161753df7f9ccf7e1d - Download
gnu-config/cci.20201022:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9 - Download
libtool/2.4.6:6fbe64f35b9d3eb3f534cbf1646451f40f718a31 - Build
m4/1.4.18:24647d9fe8ec489125dfbae4b3ebefaf7581674c - Download
Cross-build from 'Linux:x86_64' to 'Linux:armv7hf'
Installing (downloading, building) binaries...
...
libtool/2.4.6: Copying sources to build folder
libtool/2.4.6: Building your package in /root/.conan/data/libtool/2.4.6/_/_/build/6fbe64f35b9d3eb3f534cbf1646451f40f718a31
libtool/2.4.6: Generator txt created conanbuildinfo.txt
libtool/2.4.6: Calling build()
libtool/2.4.6: Calling:
> /root/.conan/data/libtool/2.4.6/_/_/build/6fbe64f35b9d3eb3f534cbf1646451f40f718a31/source_subfolder/configure '--datarootdir=/root/.conan/data/libtool/2.4.6/_/_/package/6fbe64f35b9d3eb3f534cbf1646451f40f718a31/bin/share' '--prefix=/root/.conan/data/libtool/2.4.6/_/_/package/6fbe64f35b9d3eb3f534cbf1646451f40f718a31' '--enable-shared' '--enable-static' '--enable-ltdl-install' '--bindir=${prefix}/bin' '--sbindir=${prefix}/bin' '--libexecdir=${prefix}/bin' '--libdir=${prefix}/lib' '--includedir=${prefix}/include' '--oldincludedir=${prefix}/include' --target=arm-linux-gnueabihf
## ------------------------- ##
## Configuring libtool 2.4.6 ##
## ------------------------- ##
checking for GNU M4 that supports accurate traces... configure: error: no acceptable m4 could be found in $PATH.
GNU M4 1.4.6 or later is required; 1.4.16 or newer is recommended.
GNU M4 1.4.15 uses a buggy replacement strstr on some systems.
Glibc 2.9 - 2.12 and GNU M4 1.4.11 - 1.4.15 have another strstr bug.
libtool/2.4.6:
libtool/2.4.6: ERROR: Package '6fbe64f35b9d3eb3f534cbf1646451f40f718a31' build failed
libtool/2.4.6: WARN: Build folder /root/.conan/data/libtool/2.4.6/_/_/build/6fbe64f35b9d3eb3f534cbf1646451f40f718a31
ERROR: libtool/2.4.6: Error in build() method, line 104
autotools = self._configure_autotools()
while calling '_configure_autotools', line 84
self._autotools.configure(args=conf_args, configure_dir=self._source_subfolder)
ConanException: Error 1 while executing /root/.conan/data/libtool/2.4.6/_/_/build/6fbe64f35b9d3eb3f534cbf1646451f40f718a31/source_subfolder/configure '--datarootdir=/root/.conan/data/libtool/2.4.6/_/_/package/6fbe64f35b9d3eb3f534cbf1646451f40f718a31/bin/share' '--prefix=/root/.conan/data/libtool/2.4.6/_/_/package/6fbe64f35b9d3eb3f534cbf1646451f40f718a31' '--enable-shared' '--enable-static' '--enable-ltdl-install' '--bindir=${prefix}/bin' '--sbindir=${prefix}/bin' '--libexecdir=${prefix}/bin' '--libdir=${prefix}/lib' '--includedir=${prefix}/include' '--oldincludedir=${prefix}/include' --target=arm-linux-gnueabihf
@TedSmithNXC
It might be a bug in the libtool
recipe.
libtool needs m4, which is not (currently) a build requirement (only a requirement through automake
-> autoconf
-> m4
).
Can you try applying the following patch to conan-center-index?
--- a/recipes/libtool/all/conanfile.py
+++ b/recipes/libtool/all/conanfile.py
@@ -53,6 +53,7 @@ class LibtoolConan(ConanFile):
if tools.os_info.is_windows and not tools.get_env("CONAN_BASH_PATH"):
self.build_requires("msys2/20200517")
self.build_requires("gnu-config/cci.20201022")
+ self.build_requires("m4/1.4.18")
@contextmanager
def _build_context(self):
Anyway, I think this bug can be closed as my initial problem was not there all along.
@madebr
libtool needs m4, which is not (currently) a build requirement (only a requirement through
automake
->autoconf
->m4
)
I'm trying to get my head around the host and build contexts and understand your answer here. So some of what I'm saying below might be flawed.
According to @TedSmithNXC's output, libtool
is a build_requirement
. So in a cross build, that means that, when building libtool
, it should have the build-context: self.settings
should equal the build profile (x86_64), self.setting_build
should also equal the build profile, and self.settings_target
should equal the host profile (armv7hf).
Since m4
is a requirement of libtool
(via automake
-> autoconf
-> m4
) it should share the same values for those settings.
@madebr suggests to make m4
a build requirement of libtool
. If we do that, then the change I'd expect to see in the m4
recipe is for it to take self.settings_target
to also be the build profile (x86_64). I believe this would essentially configure the m4
build like it is a native build on x86_64. @madebr is your theory that the cross-compiling libtool
specifically needs a native m4
? Can you explain this interaction?
I'm weak on autotools, so I don't fully understand what libtool
and m4
need to run. libtool
runs on the build system, but I believe libtool
needs to know it is cross-compiling so it can help link up staged libraries in the target rootfs. Therefore, I would expect libtool
should pay attention to its own self.settings_target
and send that in as the --host
argument when its build is being configured. According to conan's documentation, it tries to detect cross-compilations and populate the host argument. Is the issue that AutoToolsBuildEnvironment
is not yet paying attention to self.settings_target
?
Is the issue that
AutoToolsBuildEnvironment
is not yet paying attention toself.settings_target
?
Looking at the conan source code, I see that it is checking self.settings_target
in AutoToolsBuildEnvironment
.
When building libtool, the configure script prints the following:
## ------------------------- ##
## Configuring libtool 2.4.6 ##
## ------------------------- ##
checking for GNU M4 that supports accurate traces... /home/maarten/.conan/data/m4/1.4.18/_/_/package/24647d9fe8ec489125dfbae4b3ebefaf7581674c/bin/m4
checking whether /home/maarten/.conan/data/m4/1.4.18/_/_/package/24647d9fe8ec489125dfbae4b3ebefaf7581674c/bin/m4 accepts --gnu... yes
checking how m4 supports trace files... --debugfile
So yes, libtool
needs m4
as a build requirement.
I downloaded a gcc-arm toolchain at ARM's website and could reproduce @TedSmithNXC 's problem.
It looks like there is something fundamentally wrong with autoconf
and family.
Use the following default64
profile:
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++11
compiler.version=11
os=Linux
[options]
[build_requires]
[env]
AS=gcc
ASFLAGS=-m64
ASM=gcc
ASMFLAGS=-m64
CC=gcc
CFLAGS=-m64
CXX=g++
CXXFLAGS=-m64
LDFLAGS=-m64
Run
conan create recipes/autoconf/all autoconf/2.71@h -pr:b default64 -pr:b default64
Then m4 is not detected in autoconf
's configure
.
Adding m4/1.4.18
to the build requirements of autoconf
is no fix.
I've opened https://github.com/conan-io/conan-center-index/issues/6673 and https://github.com/conan-io/conan-center-index/pull/6674
This does not fix the problem, but perhaps we can pinpoint the cause better. I'm reopening this issue as I think again it is a problem with conan.
@madebr Many thanks for digging into this!
So yes,
libtool
needsm4
as a build requirement.
I agree. But my understanding is that requirements are also available at the time of building, so there is no benefit to making something a build requirement if it is already a requirement. The only exception to this is if you need to change self.settings_target
of that dependency, but I'm fuzzy on when that would be desired. Perhaps it is desired in this case?
You've proven this is not the fix, but I'm trying to solidify my understanding. :)
@TedSmithNXC Looks like @madebr is on to something over at https://github.com/conan-io/conan-center-index/issues/6673#issuecomment-892579356.
@madebr Adding that build_requires of m4
to libtool
does fix it. I understand now what is going on. Thank you again for your help!
When cross building a package and rebuilding all its dependencies, all its dependencies, including build requirements, are built using the host profile. It should use the build profile for the build requirements.
Environment Details (include every applicable attribute)
Steps to reproduce (Include if Applicable)
default32
:default64
:cd conan-center-index/recipes/libcurl/all
conan create . libcurl/7.71.1@ -pr:h default32 -pr:b default64 --build "*"
Logs (Executed commands with output) (Include/Attach if Applicable)
While rebuilding m4, the log contains:
which means m4 is rebuilt for x86 (=host) instead of x86_64 (=build profile)