[bug] Unable to run executable under msys2

jputcu commented 1 year ago

Environment details

Operating System+version: Windows 11 with msys2/mingw64 shell
Conan version:1.56.0
Python version:3.10.9

Steps to reproduce

In a msys2/mingw64 shell, create a basic library using a template:

> conan new dummy/1.3 --template=cmake_lib
> conan create .
...
dummy/1.3 (test package): Running test()
'.' is not recognized as an internal or external command,
operable program or batch file.
ERROR: dummy/1.3 (test package): Error in test() method, line 30
        self.run(cmd, env="conanrun")
        ConanException: Error 1 while executing ./example

In my own code I only got the following working:

    def test(self):
        if not cross_building(self):
            if self.settings.os == "Windows":
                self.run("example")
            else:
                self.run("./example")

Some relevant environment variables:

> env
SHELL=/usr/bin/bash
MINGW_PREFIX=/mingw64
MSYSTEM=MINGW64
OS=Windows_NT
_MSYS2_BASH=C:\msys64\usr\bin\bash.exe
_MSYS2_PREFIX=x86_64
MSYSTEM_CHOST=x86_64-w64-mingw32
MSYSTEM_PREFIX=/mingw64

Logs

No response

memsharded commented 1 year ago

Hi @jputcu

Thanks for your question. The new tools design tries hard to not make any guesses or auto-detect, but rely on explicit user inputs. In this case, if Conan is running already inside a subsystem, it is necessary to define:

tools.microsoft.bash:active=True
tools.microsoft.bash:subsystem=msys2

To indicate that Conan is already inside the subsystem, and such command executions do not need to add the bash -c ... to run inside bash. Said that, it is still probable that there are some rough edges about running Conan inside the subsystem. In general Conan is tested and also heavily used in ConanCenter to build things inside msys2, but running from Windows native, and then running the builds inside the subsystem defining win_bash=True in the recipe.

Please let us know how it goes when you try the above.

jputcu commented 1 year ago

I've defined the parameters in global.conf but still the same error. Playing with win_bash in the recipe doesn't feel correct because the creator doesn't need to know. Removing the join with self.cpp.build.bindirs[0] it is able to run the example:

dummy/1.3 (test package): Running test()
dummy/1.3: Hello World Debug!
  dummy/1.3: _M_X64 defined
  dummy/1.3: __x86_64__ defined
  dummy/1.3: _GLIBCXX_USE_CXX11_ABI 1
  dummy/1.3: __cplusplus201703
  dummy/1.3: __GNUC__12
  dummy/1.3: __GNUC_MINOR__2
  dummy/1.3: __MINGW32__1
  dummy/1.3: __MINGW64__1

I understand the position towards guesses and auto-detect, but having the basic hello world working on all platforms would lower the threshold.

memsharded commented 1 year ago

Playing with win_bash in the recipe doesn't feel correct because the creator doesn't need to know.

This is what hundreds of recipes in ConanCenter do. And the recipes really need to know, those are libraries that only build using autotools, and then need to run in a subsystem. So the recipe explicitly tells win_bash=True to execute self.run() commands (which are also used by tools like autotools.configure()) inside the subsystem bash with bash -c .... This is the intended usage of the win_bash feature.

A different story is for a normal recipe, lets say the -m=cmake_lib one, that is going to be built inside msys2. That is what the conf tools.microsoft.bash:active=True is for.

Removing the join with self.cpp.build.bindirs[0] it is able to run the example:

This is something that looks like we would like to improve, thanks!

I understand the position towards guesses and auto-detect, but having the basic hello world working on all platforms would lower the threshold.

The basic hello world example aims to build in all platforms. But the platform has to be correctly defined in the profile, that is the basic assumption. And for the profile settings os=Windows, it is necessary something to differentiate between the native Windows and the msys2 subsystem, and that is the above conf. So maybe the conversation would be around if the Conan profile auto-detect should be able to automatically define that conf identifying that it is running inside a subsystem. It might be doable, but it is also true that profile auto-detect is not recommended for production anyway, and instead using your own profiles (can be managed with conan config install), so this would still be a very corner case feature. Not saying that it would be rejected, but probably very unlikely to be prioritized.

jputcu commented 1 year ago

I just noticed that adding the global.conf items triggered the same error but earlier, during cmake configure:

DEBUG :build.py       [11]: Call conanfile.build() with files in build folder: [] [2023-01-11 15:51:32,956]
dummy/1.3 (test package): Calling build()
dummy/1.3 (test package): CMake command: cmake -G "MinGW Makefiles" -DCMAKE_TOOLCHAIN_FILE="C:/Users/joris/Downloads/test_conan/test_package/build/generators/conan_toolchain.cmake" -DCMAKE_SH="CMAKE_SH-NOTFOUND" -DCMAKE_POLICY_DEFAULT_CMP0091="NEW" -DCMAKE_BUILD_TYPE="Debug" "C:/Users/joris/Downloads/test_conan/test_package/."
'.' is not recognized as an internal or external command,
operable program or batch file.
ERROR: dummy/1.3 (test package): Error in build() method, line 21
        cmake.configure()
        ConanException: Error 1 while executing cmake -G "MinGW Makefiles" -DCMAKE_TOOLCHAIN_FILE="C:/Users/joris/Downloads/test_conan/test_package/build/generators/conan_toolchain.cmake" -DCMAKE_SH="CMAKE_SH-NOTFOUND" -DCMAKE_POLICY_DEFAULT_CMP0091="NEW" -DCMAKE_BUILD_TYPE="Debug" "C:/Users/joris/Downloads/test_conan/test_package/."

There is also a possibility to add os.subsystem=msys2 to the profile.

memsharded commented 1 year ago

There is also a possibility to add os.subsystem=msys2 to the profile.

This is related to the binary model. If the binary is going to be linked with the subsystem compilers and runtime, then it is necessary to be added, because the binary will be a "msys2" binary. But it is also a possibility running inside msys2 and still targeting the MSVC compiler with the VS runtime, in that case it would be a native windows binary, and os.subsystem shouldn't be defined.

I can see "MinGW Makefiles" in the output of the error above. Depending on the compiler used inside msys2, which can be the msys2 gcc or the MinGW subsystem gcc, it might be necessary one generator or the other. The runtime for them is also different. In this blog post related to Clang some related issues are described: https://blog.conan.io/2022/10/13/Different-flavors-Clang-compiler-Windows.html.

As I commented, running Conan inside a subsystem is not that common. I can try to polish a bit, it would be great if you let me know:

Confirm it is msys2/mingw64 (that is, running inside the mingw64 terminal, not the msys2 and using the mingw compiler
CMake version and installation (which cmake)
Conan installation (how did you install it for running inside mingw64) Many thanks!

jputcu commented 1 year ago

Everything is done from within the mingw64. Working in a Windowsterminal:

cmd.exe /c "set MSYSTEM=MINGW64&& C:\msys64\usr\bin\bash.exe --login"

> env
SHELL=/usr/bin/bash
MINGW_PREFIX=/mingw64
MSYSTEM=MINGW64
OS=Windows_NT
_MSYS2_BASH=C:\msys64\usr\bin\bash.exe
_MSYS2_PREFIX=x86_64
MSYSTEM_CHOST=x86_64-w64-mingw32
MSYSTEM_PREFIX=/mingw64

CMake installed using pacman

> which cmake && cmake --version
/mingw64/bin/cmake
cmake version 3.25.1

CMake suite maintained and supported by Kitware (kitware.com/cmake).

Conan was installed using pip.

> which conan && conan --version
/mingw64/bin/conan
Conan version 1.56.0

Compiler detected by cmake: Check for working CXX compiler: C:/msys64/mingw64/bin/c++.exe

> which c++ && c++ --version
/mingw64/bin/c++
c++.exe (Rev7, Built by MSYS2 project) 12.2.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Strange thing, I have changed my generator to ninja in conan.conf

[general]
cmake_generator=Ninja

jputcu commented 1 year ago

The resulting binary:

$ file test_package/build/Debug/example.exe
test_package/build/Debug/example.exe: PE32+ executable (console) x86-64, for MS Windows, 21 sections

joris@JORIS-DELL MINGW64 ~/Downloads/test_conan
$ ldd test_package/build/Debug/example.exe
        ntdll.dll => /c/WINDOWS/SYSTEM32/ntdll.dll (0x7ffead6f0000)
        KERNEL32.DLL => /c/WINDOWS/System32/KERNEL32.DLL (0x7ffeab750000)
        KERNELBASE.dll => /c/WINDOWS/System32/KERNELBASE.dll (0x7ffeaaef0000)
        msvcrt.dll => /c/WINDOWS/System32/msvcrt.dll (0x7ffeabc50000)
        libstdc++-6.dll => /mingw64/bin/libstdc++-6.dll (0x7ffe3dc70000)
        libwinpthread-1.dll => /mingw64/bin/libwinpthread-1.dll (0x7ffe7e880000)
        libgcc_s_seh-1.dll => /mingw64/bin/libgcc_s_seh-1.dll (0x7ffe7dd60000)

jputcu commented 1 year ago

By putting the cmake generator to Ninja in global.conf instead of conan.conf I get another output:

dummy/1.3 (test package): Calling build()
dummy/1.3 (test package): CMake command: cmake -G "Ninja" -DCMAKE_TOOLCHAIN_FILE="C:/Users/joris/Downloads/test_conan/test_package/build/generators/conan_toolchain.cmake" -DCMAKE_POLICY_DEFAULT_CMP0091="NEW" -DCMAKE_BUILD_TYPE="Debug" "C:/Users/joris/Downloads/test_conan/test_package/."
ERROR: dummy/1.3 (test package): Error in build() method, line 21
        cmake.configure()
        ConanException: Cannot wrap command with different envs, ['C:/Users/joris/Downloads/test_conan/test_package/build/generators/conanbuild.bat'] - ['C:/Users/joris/Downloads/test_conan/test_package/build/generators/conanbuild.sh'] - []

memsharded commented 1 year ago

I am investigating this a bit, and I have struggled for hours due to an apparent bug in previous versions of CMake, that couldn't handle some Msys2 paths for find_package(), find_library(), etc.

I am updating to latest CMake 3.25 from msys2 repos, I'll keep reporting

memsharded commented 1 year ago

Another thing that I am struggling is the resolution of the Conan home, and how it affects behavior of things like paths (like the execution of Release/example.exe not found, but needing Release\example.exe instead (or .\Release\example.exe) with the typical Windows native backslash instead of the msys2 regular ones.

And it is related to the fact that inside msys2 os.path.expanduser("~") in Python do not return the /home/myuser msys2 home (I would have expected that, given that I am running a msys2 python, not the windows python), and then Conan operates with the windows cache

If I relocate the cache to the subsystem home (CONAN_USER_HOME=/home/myuser in 1.X, CONAN_HOME=/home/myuser/.conan2 in 2.0), then automatically the paths are allowed and it works. I still don't understand how this is related, but it seems to affect.

Quick question: do you intend to use a Conan cache in the Windows user (C:/Users/myuser) or in the Msys2 user (/home/myuser, or the full <path/to/msys64>/home/myuser)?

At this point, I think that I would try to skip running inside the msys2, but run in Windows prompt and use the win_bash=True in recipes that need to build inside bash (autotools ones). Can you please elaborate a bit more about your use case:

Do you and your developers always work, build things, etc inside msys2? Is that your default terminal in Windows?
Are you using CMake or other build systems?

jputcu commented 1 year ago

I'm the only embedded C/C++ at the company. Using CMake, Conan, GCC toolchains (for AVR, ARM, Linux, Windows). Mostly use msys2 because I'm used to linux and have all the command line tooling (e.g. grep). I only use the conan in the msys2 environment but my Windows user directory is my HOME. I could try CONAN_HOME=/c/Users/joris/.conan to perhaps force that path notation. (Can test further on monday)

memsharded commented 1 year ago

Thanks, some test with the other home would be nice too. Just have in mind that it is CONAN_USER_HOME for conan 1.X and CONAN_HOME env-vars for 2.0.

conan-io / conan