search

conan-io / conan-center-index

Recipes for the ConanCenter repository
https://conan.io/center
MIT License
949 stars 1.74k forks source link

[package] boost/1.74.0: building with mingw with bash on Windows (e.g. MSYS2) failes #3196

Open Johnnyxy opened 3 years ago

Johnnyxy commented 3 years ago

Hi, I am trying to build Boost 1.74.0 with MinGW from Windows.

MinGW bin-folder is in the PATH and the GCC can be executed via gcc and g++.

Windows CMD

Now from a normal Windows cmd console I can build Boost successfully.

MSYS2 / MSYS2-MinGW

But when I am in a MSYS2 or a MSYS2-MinGW environment then Boost cannot be built because Boost cannot execute the detected gcc/g++-path. In the Boost recipe the path to the compiler is deduced by checking the environment variable CXX and if not successful by using the following path detection:

    def _cxx(self):
        ...
        if self.settings.compiler == "gcc":
            return tools.which("g++-%s" % compiler_version) or tools.which("g++-%s" % major) or tools.which("g++") or ""
        ...

In MSYS2/MSYS2-MinGW-console this results in /mingw64/bin/g++. Boost is not able to use this path as it needs a Windows-formatted path and stops execution with the following error:

warning: toolset gcc initialization:
warning: can not find user-provided command '/mingw64/bin/g++'

Problem

Conan uses the which command as this is the native way to find files in PATH in a Unix(-like) shell. But in this (maybe special) case the function where from Windows has to be employed as this would return a Windows-path like this: c:\msys64\mingw64\bin\gcc.exe.

Workaround

One could set the CXX environment variable to the full g++-path. Then Boost would use this path and build without problems. But I would like to avoid that as everything is already known at the point of the Boost-build invocation to detect the correct path automatically.

Question

Is there a quick fix that I did not think of until now? Otherwise I would just modify the recipe to execute where when a MSYS2 environment is detected.

Related

A similar problem has been discussed here which led to the tools.which implementation, I think: https://github.com/conan-community/community/issues/75

Tested Fix

My quickfix would be to use tools.which to detect the g++ binary and then checking if the compiler output contains "mingw".

    import re
    ...

    def _is_compiler_host_windows(self, cmd="gcc"):
        """Checks if the given compiler command is a MinGW variant."""
        res = False
        patterns = [
            re.compile(".*--host[=\\s]([^\\s]+)")
        ]
        command = "gcc -v"
        output = StringIO()
        try:
            self.run(command=command, output=output)
            output = output.getvalue().strip()
        except ConanException:
            output = ""
        matches = [pattern.match(l) for l in output.split("\n") for pattern in patterns]
        matches = [match.group(1) for match in matches if match]
        match = matches[0] if matches else ""
        if ("mingw" in match.lower()):
            res = True
        return res

    def _windows_where(self, cmd):
        """Executes Windows' *where* command to retrieve the path of a cmd."""
        output = StringIO()
        command = 'where "%s"' % cmd
        try:
            self.run(command=command, output=output)
        except ConanException:
            return ""
        output = output.getvalue().strip().splitlines()[0]
        return output

    @property
    def _cxx(self):
        if "CXX" in os.environ:
            return os.environ["CXX"]
        if tools.is_apple_os(self.settings.os) and self.settings.compiler == "apple-clang":
            return tools.XCRun(self.settings).cxx
        compiler_version = str(self.settings.compiler.version)
        major = compiler_version.split(".")[0]
        filenames = []
        if self.settings.compiler == "gcc":
            filenames = [
                "g++-%s" % compiler_version,
                "g++-%s" % major,
                "g++"
            ]
        if self.settings.compiler == "clang":
            filenames = [
                "clang++-%s" % compiler_version,
                "clang++-%s" % major,
                "clang++"
            ]
        for filename in filenames:            
            output = tools.which(filename)            
            if (output):
                if (self.settings.compiler in ["gcc"]) and \
                   (self._is_compiler_host_windows(output)):
                    output = self._windows_where(filename)
                return output
        return ""

EDIT: updated GCC output parsing to detect GCC target specifically

madebr commented 3 years ago

I tested current master's boost on msys2 on Windows. When running g++ -print-multiarch and g++ -v, the contents did not contain the string mingw.

maarten@maarten-acer2 MSYS ~
$ g++ -print-multiarch

maarten@maarten-acer2 MSYS ~
$ g++ -v
Using built-in specs.
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-pc-msys/9.1.0/lto-wrapper.exe
Target: x86_64-pc-msys
Configured with: /msys_scripts/gcc/src/gcc-9.1.0/configure --build=x86_64-pc-msys --prefix=/usr --libexecdir=/usr/lib --enable-bootstrap --enable-shared --enable-shared-libgcc --enable-static --enable-version-specific-runtime-libs --with-arch=x86-64 --with-tune=generic --disable-multilib --enable-__cxa_atexit --with-dwarf2 --enable-languages=c,c++,fortran,lto --enable-graphite --enable-threads=posix --enable-libatomic --enable-libgomp --enable-libitm --enable-libquadmath --enable-libquadmath-support --disable-libssp --disable-win32-registry --disable-symvers --with-gnu-ld --with-gnu-as --disable-isl-version-check --enable-checking=release --without-libiconv-prefix --without-libintl-prefix --with-system-zlib --enable-linker-build-id --with-default-libstdcxx-abi=gcc4-compatible --enable-libstdcxx-filesystem-ts
Thread model: posix
gcc version 9.1.0 (GCC)

The patch below does your suggestion in an alternative way. I don't think this is the correct approach because the build fails (for me at least). b2 probably requires some extra/different flags when built inside msys2. I didn't look into this further. For reference, this is my build log of boost/1.74.0 on msys2 (with conan inside msys2)

...failed gcc.archive C:\msys64\home\maarten\.conan\data\boost\1.74.0\_\_\build\872b5bae4311838e654cdb422a408106a1ef82fe\boost\bin.v2\libs\math\build\gcc-9.1.0\rls\lnk-sttc\pch-off\thrd-mlt\vsblt-hdn\libboost_math_tr1f.a...
...skipped <pC:\msys64\home\maarten\.conan\data\boost\1.74.0\_\_\package\872b5bae4311838e654cdb422a408106a1ef82fe\lib>libboost_math_tr1f.a for lack of <pC:\msys64\home\maarten\.conan\data\boost\1.74.0\_\_\build\872b5bae4311838e654cdb422a408106a1ef82fe\boost\bin.v2\libs\math\build\gcc-9.1.0\rls\lnk-sttc\pch-off\thrd-mlt\vsblt-hdn>libboost_math_tr1f.a...
...skipped <pC:\msys64\home\maarten\.conan\data\boost\1.74.0\_\_\build\872b5bae4311838e654cdb422a408106a1ef82fe\boost\bin.v2\libs\math\build\gcc-9.1.0\rls\lnk-sttc\pch-off\thrd-mlt\vsblt-hdn>libboost_math_tr1f-variant-static.cmake for lack of <pC:\msys64\home\maarten\.conan\data\boost\1.74.0\_\_\build\872b5bae4311838e654cdb422a408106a1ef82fe\boost\bin.v2\libs\math\build\gcc-9.1.0\rls\lnk-sttc\pch-off\thrd-mlt\vsblt-hdn>libboost_math_tr1f.a...
...skipped <pC:\msys64\home\maarten\.conan\data\boost\1.74.0\_\_\package\872b5bae4311838e654cdb422a408106a1ef82fe\lib\cmake\boost_math_tr1f-1.74.0>libboost_math_tr1f-variant-static.cmake for lack of <pC:\msys64\home\maarten\.conan\data\boost\1.74.0\_\_\build\872b5bae4311838e654cdb422a408106a1ef82fe\boost\bin.v2\libs\math\build\gcc-9.1.0\rls\lnk-sttc\pch-off\thrd-mlt\vsblt-hdn>libboost_math_tr1f-variant-static.cmake...
...failed updating 37 targets...
...skipped 113 targets...
...updated 16328 targets...
boost patch ```patch --- a/recipes/boost/all/conanfile.py +++ b/recipes/boost/all/conanfile.py @@ -5,6 +5,7 @@ from conans.errors import ConanException from conans.errors import ConanInvalidConfiguration import os +import platform import sys import shlex import shutil @@ -683,35 +684,51 @@ class BoostConan(ConanFile): return flags + def _normalize_compiler_path(self, path): + if not path: + return path + if tools.os_info.is_windows and platform.system() != "Windows": + # When running conan inside msys2, tools.which returns unix paths but Boost.Build expects Windows paths on Windows + output = StringIO() + self.run("cygpath -w '{}'".format(path), output=output) + path = output.getvalue().strip().split("\n")[-1] + return path + @property def _ar(self): - if "AR" in os.environ: - return os.environ["AR"] - if tools.is_apple_os(self.settings.os) and self.settings.compiler == "apple-clang": - return tools.XCRun(self.settings).ar - return None + def find_ar(): + if "AR" in os.environ: + return os.environ["AR"] + if tools.is_apple_os(self.settings.os) and self.settings.compiler == "apple-clang": + return tools.XCRun(self.settings).ar + return "" + return self._normalize_compiler_path(find_ar()) @property def _ranlib(self): - if "RANLIB" in os.environ: - return os.environ["RANLIB"] - if tools.is_apple_os(self.settings.os) and self.settings.compiler == "apple-clang": - return tools.XCRun(self.settings).ranlib - return None + def find_ranlib(): + if "RANLIB" in os.environ: + return os.environ["RANLIB"] + if tools.is_apple_os(self.settings.os) and self.settings.compiler == "apple-clang": + return tools.XCRun(self.settings).ranlib + return None + return self._normalize_compiler_path(find_ranlib()) @property def _cxx(self): - if "CXX" in os.environ: - return os.environ["CXX"] - if tools.is_apple_os(self.settings.os) and self.settings.compiler == "apple-clang": - return tools.XCRun(self.settings).cxx - compiler_version = str(self.settings.compiler.version) - major = compiler_version.split(".")[0] - if self.settings.compiler == "gcc": - return tools.which("g++-%s" % compiler_version) or tools.which("g++-%s" % major) or tools.which("g++") or "" - if self.settings.compiler == "clang": - return tools.which("clang++-%s" % compiler_version) or tools.which("clang++-%s" % major) or tools.which("clang++") or "" - return "" + def find_cxx(): + if "CXX" in os.environ: + return os.environ["CXX"] + if tools.is_apple_os(self.settings.os) and self.settings.compiler == "apple-clang": + return tools.XCRun(self.settings).cxx + compiler_version = str(self.settings.compiler.version) + major = compiler_version.split(".")[0] + if self.settings.compiler == "gcc": + return tools.which("g++-%s" % compiler_version) or tools.which("g++-%s" % major) or tools.which("g++") or "" + if self.settings.compiler == "clang": + return tools.which("clang++-%s" % compiler_version) or tools.which("clang++-%s" % major) or tools.which("clang++") or "" + return "" + return self._normalize_compiler_path(find_cxx()) def _create_user_config_jam(self, folder): """To help locating the zlib and bzip2 deps""" ```
Johnnyxy commented 3 years ago

Hi, be aware that you have to use the MinGW-GCC specifically targeting native Windows. Your GCC target is x86_64-pc-msys which implies that you are executing the GCC that builds for the MSYS environment (using msys.lib/.dll). Binaries built with this compiler only be executed when the msys.dll is available at runtime.

The MinGW-GCC is available under /mingw64/bin/g++. This one will build native Windows binaries which do not need anything from MSYS. Its target is x86_64-w64-mingw32.

abc@abc MINGW64 /c/tmp/build
$ gcc -v
Using built-in specs.
COLLECT_GCC=c:\msys64\mingw64\bin\gcc.exe
COLLECT_LTO_WRAPPER=c:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/lto-wrapper.exe
Target: x86_64-w64-mingw32
Configured with: ../gcc-10.2.0/configure --prefix=/mingw64 --with-local-prefix=/mingw64/local --build=x86_64-w64-mingw32 --host=x86_64-w64-mingw32 --target=x86_64-w64-mingw32
...

I did not test if building boost with the MSYS-targeting GCC needs Windows paths too. I do not think so but will test this tomorrow.

Additionally your patch uses cygpath for path conversion. This is a Cygwin specific tool. And I am not sure that this is available in a minimal MSYS environment. Windows' where command on the other hand is available and works as expected. My approach is specifically targeting the MinGW-GCC variant. Excluding all GCCs which do not target native Windows/mingw. Also my approach would work with only a bash environment (starting a plain Unix-like console in Windows; no MSYS).

madebr commented 3 years ago

Hi, be aware that you have to use the MinGW-GCC specifically targeting native Windows. Your GCC target is x86_64-pc-msys which implies that you are executing the GCC that builds for the MSYS environment (using msys.lib/.dll). Binaries built with this compiler only be executed when the msys.dll is available at runtime.

Indeed, you are right. Using one with Target: x86_64-w64-mingw32 makes it work. I used mingw_installer/1.0@conan/stable to test.

Additionally your patch uses cygpath for path conversion. This is a Cygwin specific tool. And I am not sure that this is available in a minimal MSYS environment. Windows' where command on the other hand is available and works as expected. My approach is specifically targeting the MinGW-GCC variant. Excluding all GCCs which do not target native Windows/mingw. Also my approach would work with only a bash environment (starting a plain Unix-like console in Windows; no MSYS).

I think cygpath is not cygwin specific and has becom the de facto standard script to convert between unix and windows paths. The reason I'm hesitant about your mingw detection routines is that it is too mingw specific.

Conan already has a tools.unix_path function, may it also needs a tools.windows_path function?

Johnnyxy commented 3 years ago

Yes, if cygpath is available in every MSYS installation by default this will not help in cases where you start a plain bash-port (like provided with Git-for-Windows). In any case where cygpath is not available it will not work, Windows' where does.

The reason why I made it MinGW-specific is that I did not see such a problem with Boost until now. I guess that Boost has this problem only if it has to use a GCC which runs on Windows under a Unix-like-shell. There are not that much GCC-ports around which are running on Windows. I know of some like TI-GCCs (Texas Instruments) and QCC (QNX). If they have those problems too we would adapt the recipe then. In general most people are using MinGW-GCC when compiling from Windows.

For the main scenario the recipe has to detect if the compiler is a GCC compiled for Windows and modify the path to a Windows path then. Initially I thought the recipe has to check the compiler's target-triplet but this was wrong. As the compiler target-triplet is irrelevant. The host-triplet of the compiler is what should be checked (where the compiler runs). I updated my approach accordingly.

madebr commented 3 years ago

@Johnnyxy To avoid a regression, does zlib still work with #3200 using your toolchain?

Johnnyxy commented 3 years ago

@madebr I have testet it and it will not compile at the moment. This has nothing to to with the proposed recipe changes.

zlib cannot be built in a MSYS / MinGW environment via autotools or CMake without adapting the zlib recipe. I have managed to compile it anyway and made additions to the recipe.

The zlib has to be built via a separate makefile located in the source folder win32\Makefile.gcc. One has to execute it like this:

...
    @property
    def _use_make_mingw(self):
        return tools.os_info.is_windows and tools.os_info.detect_windows_subsystem() in ["msys", "msys2"]
...
    def _build_zlib_mingw(self):
        build_folder = os.getcwd() # currently in source_folder/_build
        with tools.chdir(".."):
            self.run("make -fwin32/Makefile.gcc")
            self.run("make install -fwin32/Makefile.gcc SHARED_MODE={shared} DESTDIR={build_folder}".format(shared=("1" if self.options.shared else "0"), build_folder=build_folder))
            self.run("make test testdll -fwin32/Makefile.gcc")
...

    def _build_zlib(self):
        ...
                tools.mkdir("_build")
                with tools.chdir("_build"):
                    if self._use_autotools:
                        self._build_zlib_autotools()
                    elif self._use_make_mingw:
                        self._build_zlib_mingw()
                    else:
                        self._build_zlib_cmake()

This works as expected. What needs to be done is to make modifications to the test_package too. The test_package invokes only CMake. This is a problem as Conan uses the "MinGW Makefiles" generator but only the "real" CMake variant for windows knows this generator. The MSYS provided one or any other variant of CMake does not.

stevenwdv commented 5 months ago

A fix for this should probably go hand-in hand with proper handling of non-absolute user-specified paths in tools.build:compiler_executables (e.g. clang/clang++), see https://github.com/conan-io/conan/issues/15256#issuecomment-1864660463