[bug] cannot build boost (and some other package) when using tools.env.virtualenv:powershell=True

fdgStilla commented 1 year ago

Environment details

Operating System+version: Windows 11
Compiler+version: VS16
Conan version: 1.55
Python version: 3.10

Steps to reproduce

Run conan install boost/1.78.0@ --build boost -c tools.env.virtualenv:powershell=True => an error is displayed:

ConanException: Cannot wrap command with different envs, ['C:\.conan\578f60\1\build-release\conan\conanbuild.bat'] - [] - ['C:\.conan\578f60\1\build-release\conan\conanbuild.ps1']

In reality it occurs in our CI using powershell and the global.conf contains tools.env.virtualenv:powershell=True, and we also need the virtual env for powershell, so it is not as easy as simply removing the config option from the command line to fix the issue.

Logs

No response

fdgStilla commented 1 year ago

I know that I had the issue with other packages but I don't remember which one exactly, I think it was meson

b1ackviking commented 1 year ago

I am facing the same issue with fmt/9.1.0:

OS: Windows 11
Compiler: MSVC 19.35
Conan: 1.58 and 2.0-beta9
Python: 3.11

According to this comment, Conan tries to activate both conanbuild.bat and conanbuild.ps1 if they exist.

memsharded commented 1 year ago

Hi there!

I am trying to reproduce this with Conan 2.0, and so far found a bug in escaping of spaces with powershell, I will submit a PR.

The error that you are seeing, I can only reproduce in the test_package side, if I mix the usage of tools.env.virtualenv:powershell=True with previous invocations of the same command without it, because that might reuse the same folder and find there the .bat files (I need to look at this too).

It would be useful if you share your whole output logs, to see if you are experiencing this in the test_package side or not. Thanks!

memsharded commented 1 year ago

If it is a test_package issue, I am proposing https://github.com/conan-io/conan/pull/13084 to fix this for 2.0 (in 1.X the solution would be to clean the temporary test_package/build or equivalent)

b1ackviking commented 1 year ago

Hi @memsharded

Here is the output from conan 2.0.0-beta9 run in powershell for this repository:

PS C:\Users\B1ack\projects\cpp-project-template> conan --version
Conan version 2.0.0-beta9

PS C:\Users\B1ack\projects\cpp-project-template> conan remove *
Remove the recipe and all the packages of 'fmt/9.1.0#e747928f85b03f48aaf227ff897d9634'? (yes/no): yes
Remove the recipe and all the packages of 'gtest/1.12.1#3ed1009f8004c2ac29ab5b1d9fbfbf53'? (yes/no): yes

PS C:\Users\B1ack\projects\cpp-project-template> conan install -pr:b default -pr:h default -c tools.cmake.cmaketoolchain:generator=Ninja -s build_type=Debug -b missing . -c tools.microsoft.msbuild:installation_path="C:\Program Files\Microsoft Visual Studio\2022\Preview" -c tools.env.virtualenv:powershell=True

-------- Input profiles --------
Profile host:
[settings]
arch=x86_64
build_type=Debug
compiler=msvc
compiler.cppstd=23
compiler.runtime=static
compiler.runtime_type=Release
compiler.version=193
os=Windows
[conf]
tools.cmake.cmaketoolchain:generator=Ninja
tools.microsoft.msbuild:installation_path=C:\Program Files\Microsoft Visual Studio\2022\Preview
tools.env.virtualenv:powershell=True

Profile build:
[settings]
arch=x86_64
build_type=Release
compiler=msvc
compiler.cppstd=23
compiler.runtime=static
compiler.runtime_type=Release
compiler.version=193
os=Windows

-------- Computing dependency graph --------
fmt/9.1.0: Not found in local cache, looking in remotes...
fmt/9.1.0: Checking remote: conancenter
fmt/9.1.0: Trying with 'conancenter'...
Downloading conanmanifest.txt
Downloading conanfile.py
Downloading conan_export.tgz
Decompressing conan_export.tgz
fmt/9.1.0: Downloaded recipe revision e747928f85b03f48aaf227ff897d9634
gtest/1.12.1: Not found in local cache, looking in remotes...
gtest/1.12.1: Checking remote: conancenter
gtest/1.12.1: Trying with 'conancenter'...
Downloading conan_export.tgz
Downloading conanfile.py
Downloading conanmanifest.txt
Decompressing conan_export.tgz
gtest/1.12.1: Downloaded recipe revision 3ed1009f8004c2ac29ab5b1d9fbfbf53
Graph root
    conanfile.py: C:\Users\B1ack\projects\cpp-project-template\conanfile.py
Requirements
    fmt/9.1.0#e747928f85b03f48aaf227ff897d9634 - Downloaded (conancenter)
    gtest/1.12.1#3ed1009f8004c2ac29ab5b1d9fbfbf53 - Downloaded (conancenter)

-------- Computing necessary packages --------
fmt/9.1.0: Checking 3 compatible configurations:
fmt/9.1.0: '3ece157c89538e78c1993e4ac4b8ae459cd713e9': compiler.cppstd=14
fmt/9.1.0: 'f6d20493a27b06f0c9692d99109dabc35b3e9f7e': compiler.cppstd=17
fmt/9.1.0: '76717822334f745cc7642b44b8dd062394222567': compiler.cppstd=20
gtest/1.12.1: Checking 3 compatible configurations:
gtest/1.12.1: 'ca71b74982d972494e80364302d7994aee0f933a': compiler.cppstd=14
gtest/1.12.1: '3d187f1f0245ab55c0f75d7bed463a819ea8cfa0': compiler.cppstd=17
gtest/1.12.1: 'ba0e7fc7a954ddff3cf4adeea26441d8c4aecce6': compiler.cppstd=20
Requirements
    fmt/9.1.0#e747928f85b03f48aaf227ff897d9634:fd4295b1197d392d3235159c430acb8d3e41b2bf - Build
    gtest/1.12.1#3ed1009f8004c2ac29ab5b1d9fbfbf53:258e605002596f9a04069cdf6089e8df0d9d258b - Build

-------- Installing packages --------

-------- Installing (downloading, building) binaries... --------
fmt/9.1.0: Sources downloaded from 'conancenter'
fmt/9.1.0: Calling source() in C:\Users\B1ack\.conan2\p\fmt7a5444c469322\s\src
Downloading fmt-9.1.0.zip

fmt/9.1.0: Unzipping 4.5MB, this can take a while
Unzipping 100 %
fmt/9.1.0: Copying sources to build folder
fmt/9.1.0: Building your package in C:\Users\B1ack\.conan2\p\t\fmte9a0a885af0e1\b
fmt/9.1.0: Calling generate()
fmt/9.1.0: Preset 'debug' added to CMakePresets.json. Invoke it manually using 'cmake --preset debug'
fmt/9.1.0: If your CMake version is not compatible with CMakePresets (<3.19) call cmake like: 'cmake <path> -G Ninja -DCMAKE_TOOLCHAIN_FILE=C:\Users\B1ack\.conan2\p\t\fmte9a0a885af0e1\b\build\Debug\generators\conan_toolchain.cmake -DFMT_DOC=OFF -DFMT_TEST=OFF -DFMT_INSTALL=ON -DFMT_LIB_DIR=lib -DFMT_OS=ON -DCMAKE_POLICY_DEFAULT_CMP0091=NEW -DCMAKE_BUILD_TYPE=Debug'
fmt/9.1.0: Aggregating env generators
fmt/9.1.0: Calling build()
fmt/9.1.0: apply_conandata_patches(): No patches defined in conandata
fmt/9.1.0: CMake command: cmake -G "Ninja" -DCMAKE_TOOLCHAIN_FILE="C:/Users/B1ack/.conan2/p/t/fmte9a0a885af0e1/b/build/Debug/generators/conan_toolchain.cmake" -DCMAKE_INSTALL_PREFIX="C:/Users/B1ack/.conan2/p/t/fmte9a0a885af0e1/p" -DFMT_DOC="OFF" -DFMT_TEST="OFF" -DFMT_INSTALL="ON" -DFMT_LIB_DIR="lib" -DFMT_OS="ON" -DCMAKE_POLICY_DEFAULT_CMP0091="NEW" -DCMAKE_BUILD_TYPE="Debug" "C:\Users\B1ack\.conan2\p\t\fmte9a0a885af0e1\b\src"
fmt/9.1.0: ERROR:
Package 'fd4295b1197d392d3235159c430acb8d3e41b2bf' build failed
fmt/9.1.0: WARN: Build folder C:\Users\B1ack\.conan2\p\t\fmte9a0a885af0e1\b\build\Debug
ERROR: fmt/9.1.0: Error in build() method, line 93
        cmake.configure()
        ConanException: Cannot wrap command with different envs, ['C:\\Users\\B1ack\\.conan2\\p\\t\\fmte9a0a885af0e1\\b\\build\\Debug\\generators\\conanbuild.bat'] - [] - ['C:\\Users\\B1ack\\.conan2\\p\\t\\fmte9a0a885af0e1\\b\\build\\Debug\\generators\\conanbuild.ps1']
PS C:\Users\B1ack\projects\cpp-project-template>

fdgStilla commented 1 year ago

It does not seem to be a test_package issue:

>conan install boost/1.79.0@ --build boost -c tools.env.virtualenv:powershell=True
Configuration:
[settings]
arch=x86_64
arch_build=x86_64
build_type=Release
compiler=Visual Studio
compiler.runtime=MD
compiler.version=16
os=Windows
os_build=Windows
[options]
[build_requires]
[env]
[conf]
tools.cmake.cmaketoolchain:generator=Ninja
tools.cmake.cmaketoolchain.presets:max_schema_version=2
tools.system.package_manager:mode=install
tools.env.virtualenv:powershell=True

boost/1.79.0: Not found in local cache, looking in remotes...
boost/1.79.0: Trying with 'artifactory-conancenter'...
Downloading conanmanifest.txt completed [0.79k]
Downloading conanfile.py completed [80.12k]
Downloading conan_export.tgz completed [1.74k]
Decompressing conan_export.tgz completed [0.00k]
boost/1.79.0: Downloaded recipe revision 65fac7fc9c7d1f1df53a69aabc09d2d4
boost/1.79.0: Forced build from source
b2/4.9.3: Not found in local cache, looking in remotes...
b2/4.9.3: Trying with 'artifactory-conancenter'...
Downloading conanmanifest.txt completed [0.10k]
Downloading conanfile.py completed [6.85k]
Downloading conan_export.tgz completed [0.24k]
Decompressing conan_export.tgz completed [0.00k]
b2/4.9.3: Downloaded recipe revision 5445210661950ce568cfd76158d9e117
Installing package: boost/1.79.0
Requirements
    boost/1.79.0 from 'artifactory-conancenter' - Downloaded
    bzip2/1.0.8 from 'artifactory-conancenter' - Cache
    zlib/1.2.13 from 'artifactory-conancenter' - Cache
Packages
    boost/1.79.0:37050c911e5419517ec6e0761c16c6e262b3e2a3 - Build
    bzip2/1.0.8:d16a91eadaaf5829b928b12d2f836ff7680d3df5 - Cache
    zlib/1.2.13:3fb49604f9c2f729b85ba3115852006824e72cab - Cache
Build requirements
    b2/4.9.3 from 'artifactory-conancenter' - Downloaded
Build requirements packages
    b2/4.9.3:ca33edce272a279b24f87dc0d4cf5bbdcffbc187 - Download

Installing (downloading, building) binaries...
b2/4.9.3: Retrieving package ca33edce272a279b24f87dc0d4cf5bbdcffbc187 from remote 'artifactory-conancenter'
Downloading conanmanifest.txt completed [14.27k]
Downloading conaninfo.txt completed [0.24k]
Downloading conan_package.tgz completed [505.63k]
Decompressing conan_package.tgz completed [0.00k]
b2/4.9.3: Package installed ca33edce272a279b24f87dc0d4cf5bbdcffbc187
b2/4.9.3: Downloaded package revision bee211b00c687df275f75974c79b5827
bzip2/1.0.8: Already installed!
zlib/1.2.13: Already installed!
boost/1.79.0: Applying build-requirement: b2/4.9.3
Downloading conan_sources.tgz completed [3.80k]
Decompressing conan_sources.tgz completed [0.00k]
boost/1.79.0: Configuring sources in C:\.conan\902bfd\1\src
Downloading boost_1_79_0.tar.bz2 completed [110797.67k]                                  boost/1.79.0: /1.79.0:
boost/1.79.0:
boost/1.79.0: Apply patch (conan): Fails the build when there is no iconv backend
boost/1.79.0: Apply patch (conan): Fails the build when mpi is not configured
boost/1.79.0: Apply patch (conan): This library links to boost_system, even though that library is header-only
boost/1.79.0: Apply patch (conan): fix layout=versioned for clang@Macos + mingw@Windows
boost/1.79.0: Apply patch (official): json::array::erase(it) seg fault on linux
boost/1.79.0: Apply patch (portability): Use msync for PowerPC architectures
boost/1.79.0: Apply patch (portability): Fix access specifier preventing use of experimental iterators. Allow more granular control over enabled experimental features.
boost/1.79.0: Building your package in C:\.conan\db106a\1
boost/1.79.0: Generator txt created conanbuildinfo.txt
boost/1.79.0: Calling generate()
boost/1.79.0: Aggregating env generators
boost/1.79.0: Calling build()
boost/1.79.0: WARN: replace_in_file didn't find pattern '/* thread_local */' in 'C:\.conan\902bfd\1\src\boost\stacktrace\detail\libbacktrace_impls.hpp' file.
boost/1.79.0: WARN: replace_in_file didn't find pattern '/* static __thread */' in 'C:\.conan\902bfd\1\src\boost\stacktrace\detail\libbacktrace_impls.hpp' file.
boost/1.79.0: WARN: replace_in_file didn't find pattern 'local generic-os = [ set.difference $(all-os) : aix darwin vxworks solaris osf hpux ] ;' in 'C:\.conan\902bfd\1\src\tools\build\src\tools\gcc.jam' file.
boost/1.79.0: WARN: replace_in_file didn't find pattern 'local no-threading = android beos haiku sgi darwin vxworks ;' in 'C:\.conan\902bfd\1\src\tools\build\src\tools\gcc.jam' file.
boost/1.79.0: WARN: Patching user-config.jam
boost/1.79.0: WARN: Using the new toolchains and generators without specifying a build profile (e.g: -pr:b=default) is discouraged and might cause failures and unexpected behavior
boost/1.79.0: WARN:
using zlib : 1.2.13 : <include>"C:/Users/user/.conan/data/zlib/1.2.13/_/_/package/3fb49604f9c2f729b85ba3115852006824e72cab/include" <search>"C:/Users/user/.conan/data/zlib/1.2.13/_/_/package/3fb49604f9c2f729b85ba3115852006824e72cab/lib" <name>zlib ;
using bzip2 : 1.0.8 : <include>"C:/Users/user/.conan/data/bzip2/1.0.8/_/_/package/d16a91eadaaf5829b928b12d2f836ff7680d3df5/include" <search>"C:/Users/user/.conan/data/bzip2/1.0.8/_/_/package/d16a91eadaaf5829b928b12d2f836ff7680d3df5/lib" <name>bz2 ;
using "msvc" : 14.2 :  :
 ;
boost/1.79.0: WARN: b2 -q numa=on target-os=windows architecture=x86 address-model=64 binary-format=pe abi=ms --layout=system --user-config=C:\.conan\902bfd\1\src\tools\build\user-config.jam -sNO_ZLIB=0 -sNO_BZIP2=0 -sNO_LZMA=1 -sNO_ZSTD=1 boost.locale.icu=off --disable-icu boost.locale.iconv=off --disable-iconv runtime-link=shared runtime-debugging=off threading=multi visibility=hidden link=static variant=release --with-atomic --with-chrono --with-container --with-context --with-contract --with-coroutine --with-date_time --with-exception --with-fiber --with-filesystem --with-graph --with-iostreams --with-json --with-locale --with-log --with-math --with-nowide --with-program_options --with-random --with-regex --with-serialization --with-stacktrace --with-system --with-test --with-thread --with-timer --with-type_erasure --with-wave toolset=msvc pch=on linkflags="" cxxflags="-fPIC" install --prefix=C:\.conan\6e718d\1 -j8 --abbreviate-paths -d0 --debug-configuration --build-dir="C:\.conan\db106a\1\build-release"
boost/1.79.0:
boost/1.79.0: ERROR: Package '37050c911e5419517ec6e0761c16c6e262b3e2a3' build failed
boost/1.79.0: WARN: Build folder C:\.conan\db106a\1\build-release
ERROR: boost/1.79.0: Error in build() method, line 872
        self.run(full_command)
        ConanException: Cannot wrap command with different envs, ['C:\\.conan\\db106a\\1\\build-release\\conan\\conanbuild.bat'] - [] - ['C:\\.conan\\db106a\\1\\build-release\\conan\\conanbuild.ps1']

memsharded commented 1 year ago

Thanks very much for sharing. I think I see what is happening. When some build systems need the vcvars.bat of Visual Studio to activate the environment (Ninja, b2), then a conanbuild.bat to launch vcvars.bat is created, which collides with powershell.

I will have a look at this, but mixing bats and powershell is probably challenging. I'd recommend in the meantime to not use powershell for dependencies, and builds in the cache, just for developer experience as consumer conan install without building dependencies. I will probably mark powershell support as experimental or preview in 2.0, I see it is still problematic.

memsharded commented 1 year ago

I keep having a look at this.

I am also interested in high level feedback: what problems are you trying to solve by using -c tools.env.virtualenv=powershell? To build packages from ConanCenter recipes it should be pretty unnecessary, I think it might only make sense in the consumer side, to integrate with other tooling?

fdgStilla commented 1 year ago

Our CI is running on powershell so we have this tools.env.virtualenv=powershell in our shared conan config file to be able to run the unit test etc.

Our job that build the missing package also use this shared config file, that's why we are building with this parameter.

b1ackviking commented 1 year ago

I found this issue by accident. My problem is more about activating vcvars in general. Maybe you can advise me on that.

My use case is to build a C++ application with CMake+Ninja on Windows (in PowerShell) and Linux like this:

conan install . -c tools.cmake.cmaketoolchain:generator=Ninja -s build_type=Debug
cmake --preset debug
cmake --build --preset debug

On Windows, I also pass -c tools.microsoft.msbuild:installation_path="C:\Program Files\Microsoft Visual Studio\2022\Preview" to the conan install command, so Conan can find the MSVC compiler.

To run the cmake --preset debug command, I have to activate vcvars somehow. Conan generates ./build/Debug/generators/conanvcvars.bat and ./build/Debug/generators/conanbuild.bat files, but I can not use them from PowerShell. I thought using -c tools.env.virtualenv:powershell=True might get me .ps1 versions of these scripts, but it is not.

In fact, the VcVars generator (invoked inside CMakeToolchain generator) only outputs the conanvcvars.bat file, which is obviously not used in conanbuild.ps1 file. And because there are no prebuilt dependencies for my configuration, when I pass -b missing to conan install, I get the error that both conanbuild.bat and conanbuild.ps1 are used.

So -c tools.env.virtualenv:powershell=True does not solve my initial problem and not required for me to use anyway. Instead, I found these ways to manage vsvars:

Activate vcvars manually with & 'C:\Program Files\Microsoft Visual Studio\2022\Preview\Common7\Tools\Launch-VsDevShell.ps1' -Arch amd64 -HostArch amd64
Use Visual Studio generator instead of Ninja
Execute cmake configuration step in cmd.exe: cmd.exe /c '.\build\Debug\generators\conanbuild.bat && cmake --build --preset debug'
Run all commands in the Visual Studio dev prompt (the one that is installed with Visual Studio and has vcvars activated already)

memsharded commented 1 year ago

If you want to try to run Conan from source from my branch, I am trying there a bit of hacky approach, but it might work.

In any case, it is intended to run the powershell scripts, but not sure if you mean that you are running Conan inside a powershell terminal, that might still not work with my branch, because it still relies on running a vcvars.bat...

memsharded commented 1 year ago

cmd.exe /c '.\build\Debug\generators\conanbuild.bat && cmake --build --preset debug'

I have been trying this in my PR and I couldn't make this works in the command line ``powershell .... ; cmd /c '...bat && somecmd' was not working, either crashing or the environment launched bey .bat was temporary and not applied to somecmd, like powershell is launching different suprocesses or something. But I can check this approach again

Run all commands in the Visual Studio dev prompt (the one that is installed with Visual Studio and has vcvars activated already)

Running in a regular cmd terminal, and activating the conanbuild.bat has the same effect, this is the intention, but yeah, this can work too

memsharded commented 1 year ago

The original issue has been fixed in https://github.com/conan-io/conan/pull/13084, for next beta.10 (or Conan 2.0 final, we will see... :) ), so closing this is as solved, testing and feedback very welcome.

I see there could still be issues for running Conan inside powershell. I have been trying a bit, and with those changes made things work. Please test, and if you have any further issue, I'd recommend to open a new ticket, and I strongly recommend to test things with the conan new cmake_lib -d name=pkg -d version=1.0 (2.0 syntax) template, and the commands you are using, that helps a lot to quickly reproduce things. Thanks!

conan-io / conan