[question/feature] conan.tools.env » VirtualRunEnv does not generate sh scripts on windows

boussaffawalid commented 2 years ago

Hello,

I'm trying to switch to the new VirtualRunEnv instead of virtualrunenv, but I've noticed that the new generator does not generate sh scripts for windows when conan command is being called from bash (git bash or MSYS2)

This was not the case before with the old virtualrunenv generator. Not sure if this is an intended behavior or a bug, but it would be really nice to support sh scripts for windows as well.

Thanks.

franramirez688 commented 2 years ago

Hi @boussaffawalid

Just to be sure, is that happening even adding os.subsystem=msys2 into your profile?

boussaffawalid commented 2 years ago

@franramirez688 I tried to add os.subsystem=msys2 to my profile as follow

[settings]
os=Windows
os_build=Windows
os.subsystem=msys2
arch=x86_64
arch_build=x86_64
compiler=Visual Studio
compiler.version=16
compiler.runtime=MD
build_type=RelWithDebInfo

[options]

[tool_requires]
*: ninja/1.10.2-gb935597@pmi/stable

[env]
CC=cl.exe
CXX=cl.exe

[conf]
tools.cmake.cmaketoolchain:generator=Visual Studio 16 2019

but I got this error

ninja/1.10.2-gb935597@pmi/stable: WARN: Can't find a 'ninja/1.10.2-gb935597@pmi/stable' package for the specified settings, options and dependencies:
- Settings: arch=x86_64, build_type=RelWithDebInfo, compiler=Visual Studio, compiler.runtime=MD, compiler.version=16, os=Windows, os.subsystem=msys2

it seems like os.subsystem is not included in the package id

memsharded commented 2 years ago

I think it is because running in bash is not the same as having a os.subsystem different binary. That subsetting is intended for having specific different binaries for msys/cygwin platforms. In this case, it seems that you want to run in bash, but the build is still a native Windows binary, so no os.subsystem should be defined.

Instead, the confs:

"tools.microsoft.bash:subsystem"
"tools.microsoft.bash:path"

Is what will be used to tell Conan that it is running bash on that specific subsystem, and Conan should act accordingly and generate .sh files

franramirez688 commented 2 years ago

Yeah, that's fine. It only means that the precompiled package does not exist for your current settings (see this Troubleshooting section). You'll have to add the --build missing option into your conan install or conan create command.

boussaffawalid commented 2 years ago

@memsharded I'm running native windows binary, but using git bash to run commands/scripts instead of windows cmd.exe. I tired all of options below but nothing worked, only conanrun.bat are being generated

export CONAN_BASH_PATH="C:\\Program Files\\Git\\bin\\bash.exe" export CONAN_BASH_PATH="C:/Program Files/Git/bin/bash.exe" export CONAN_BASH_PATH=C:/Program Files/Git/bin/bash.exe export CONAN_BASH_PATH=/C/Program Files/Git/bin/bash.exe

[conf]
tools.microsoft.bash:path="C:/Program Files/Git/bin/bash.exe"

[conf]
tools.microsoft.bash:path="C:/Program Files/Git/bin/bash.exe"

[conf]
tools.microsoft.bash:path=/C/Program Files/Git/bin/bash.exe

[conf]
tools.microsoft.bash:path="C:\Program Files\Git\bin\bash.exe"

memsharded commented 2 years ago

But aren't you defining tools.microsoft.bash:subsystem too? That is the main one to be defined, as it conditions the path format and other stuff to be used

boussaffawalid commented 2 years ago

I tried that as well but still the same issue, here are some of the combinations I tired

[conf]
tools.cmake.cmaketoolchain:generator=Ninja
tools.microsoft.bash:subsystem=msys
tools.microsoft.bash:path=C:/Program Files/Git/bin/bash.exe

[conf]
tools.cmake.cmaketoolchain:generator=Ninja
tools.microsoft.bash:subsystem=msys2
tools.microsoft.bash:path=C:/Program Files/Git/bin/bash.exe

[conf]
tools.cmake.cmaketoolchain:generator=Ninja
tools.microsoft.bash:subsystem=cygwin
tools.microsoft.bash:path=C:/Program Files/Git/bin/bash.exe

[conf]
tools.cmake.cmaketoolchain:generator=Ninja
tools.microsoft.bash:subsystem=cygwin
tools.microsoft.bash:path="C:/Program Files/Git/bin/bash.exe"

memsharded commented 2 years ago

Oh, I think what it could be. Have you added the win_bash = True to the recipe? This is necessary to enable the Windows subsystem bash. I have tried it, and I managed to get .sh files with the above conf (the first one)

boussaffawalid commented 2 years ago

still I cannot get the sh scripts even with win_bash = True I will share a git repo with a basic example to reproduce the issue.

boussaffawalid commented 2 years ago

@memsharded here is a basic example to reproduce the issue https://github.com/boussaffawalid/conan_issue_10900

memsharded commented 2 years ago

I see. It seems the issue is that you are using VirtualRunEnv. As it is a Windows native binary, it doesn't need the subsystem, and its target run environment is the Windows native machine. So it is generating a .bat according to the target run environment. It works when you are using VirtualBuildEnv and using tool_requires dependencies.

For Conan VirtualRunEnv to generate a .sh for the run environment, it should be targeting the subsystem environment, like being a msys2 or cygwin executable.

Maybe it would be good to understand a bit better what you are trying to accomplish, and your setup. Conan so far models 2 ways of working:

Building native Windows binaries, but needing to use some bash for running the build. This is the conf way, and it doesn't need to run Conan inside bash, it can perfectly run in cmd terminal, and it will still use the bash terminal for the build whenever necessary. But other tasks, as running executables are still done in the native Windows. (with .bat files)
Building subsystem Windows binaries (like cygwin or msys2), binaries that somehow must run inside the subsystem. Those use the os.subsystem setting definition (they are different binaries than the native ones). Conan must run inside the subsystem directly, not natively.

boussaffawalid commented 2 years ago

Our use case:

Build native Windows binaries that does not necessarily need bash for running the build.
Use bash (from msys or git bash) as main terminal instead of windows cmd.exe to benefit from the bash utilities
I would like to be able generate conanrun.sh for native windows, so that we can start executable directly from bash.exe without the need to switch to windows cmd.exe.

memsharded commented 2 years ago

Just in case: I think explicitly generating the file is always possible with something like:

def generate(self):
    vrun = VirtualRunEnv(self)
    vvars = vrun.vars()
    vvars.save_script("myvars.sh")

As long as the file has the .sh extension, it will use the .sh output format and not bat.

franramirez688 commented 2 years ago

Did you get to solve the issue @boussaffawalid ? Or still, having problems?

boussaffawalid commented 2 years ago

@franramirez688 I just tested it and it works as charm! Thanks

boussaffawalid commented 2 years ago

Doing some more testing I found another issue. here is what is in my conanrunenv.sh

export PATH="C:\.conan\ccd9f5\1\bin;C:\.conan\04a518\1\bin;C:\.conan\3de530\1\bin;$PATH"

This wont work from git bash, it has to be something like

export PATH="C:\.conan\ccd9f5\1\bin":"C:\.conan\04a518\1\bin":"C:\.conan\3de530\1\bin":"$PATH"

memsharded commented 2 years ago

This is true @franramirez688 , the code inside has a comment:

    if subsystem is None and not scope.startswith("build"):  # "run" scope do not follow win_bash
        return WINDOWS

Most of the internals are not designed to run inside bash, but without not targeting a subsystem.

friendlyanon commented 2 years ago

The Virtual*Env generators should just generate all 3 flavors of scripts, like they did with the v1 generators. On Windows especially, one can end up using cmd, bash and powershell, while on *nix powershell is also used quite a bit.

memsharded commented 2 years ago

The VirtualEnv generators should just generate all 3 flavors of scripts, like they did with the v1 generators. On Windows especially, one can end up using cmd, bash and powershell, while on nix powershell is also used quite a bit.

Even if it generated the 3 flavors, now Conan relies on using those files for the injection of the environment to commands. So defining which system/flavor is using is necessary. Generating the 3 only delays the decision, and introduces the chance of discrepancies, like working in the user side because the user manually uses one, but then failing at conan create because Conan uses a different one. It would be preferred that the generated and used file is selected consistently.

friendlyanon commented 2 years ago

I don't think someone who uses Conan just to fetch dependencies and spit out config files to make those importable wants to use conan create or other machinery provided by Conan. In fact, it looks like someone who wants to use Conan as such needs to do more upfront Conan specific plumbing with v2.

memsharded commented 2 years ago

I don't think someone who uses Conan just to fetch dependencies and spit out config files to make those importable wants to use conan create or other machinery provided by Conan. In fact, it looks like someone who wants to use Conan as such needs to do more upfront Conan specific plumbing with v2.

Even if you don't want to, it is necessary to be consistent, otherwise it is a mess for users wanting to create packages. And when the configuration files need to be aware of things like subsystem, the configuration is necessary. It is not enough to generate .sh files, it is necessary to know which subsystem is targeting (msys, cygwin...) because the paths do change. Conan v2 is indeed more explicit, it will no longer try to guess things, it has always been a very problematic behavior of Conan 1.X, and a source of many support tickets and users struggling. So now the idea is that Conan 2.0 will generate a .sh script for you, for example if you tell it "I am using msys2, and I need a .sh" file for it. It might be via a -s os.subsystem=msys2 or maybe with -c tools.build:subsystem=cygwin, depending on the use case. And it is totally possible that we are missing some cases that we haven't contemplated yet in the new model, but for sure, Conan 2.0 will require the user to tell what it wants. If you are in Windows, and target Visual compiler, and don't specify anything else, it will likely resort the the default of generating a .bat for you and not a .sh, because the likelihood that the .sh that it will be guessing is not right is not low.

friendlyanon commented 2 years ago

I guess what I'm missing is very clear step-by-step guides for scenarios like the case I presented (just using Conan to fetch deps and spit out the files for CMake to use) and how get more involved with Conan.

It would be terrific if I could just link people wishing to contribute to a Conan and CMake managed project in the contribution docs to a Conan managed documentation page that describes setting up a profile, toolchain settings, etc, to the point where they can just conan install . and carry on with their day.

memsharded commented 2 years ago

I guess what I'm missing is very clear step-by-step guides for scenarios like the case I presented (just using Conan to fetch deps and spit out the files for CMake to use) and how get more involved with Conan.

Yes, that is true. We are in the process of creating whole new docs for Conan 2.0, and they are more tutorial-like, with more examples, etc.

memsharded commented 2 years ago

Hi all,

I am investigating/proposing in https://github.com/conan-io/conan/pull/11066, to define a new conf that says "I am already running inside bash", and that would trigger the creation of .sh files, both for "buildenv" and "runenv"

paulharris commented 2 years ago

Even if you don't want to, it is necessary to be consistent, otherwise it is a mess for users wanting to create packages. And when the configuration files need to be aware of things like subsystem, the configuration is necessary. It is not enough to generate .sh files, it is necessary to know which subsystem is targeting (msys, cygwin...) because the paths do change. Conan v2 is indeed more explicit, it will no longer try to guess things, it has always been a very problematic behavior of Conan 1.X, and a source of many support tickets and users struggling.

So how can someone build their own in-source recipe with "conan install . ; conan source ." in msys (and also probably "conan build ." and then switch to MSVC to compile and debug? We are still pumping out the .sln files, right? Won't those be building on the DOS MSVC Command Prompt environment?

I don't know all the problems you are talking about (apart from the generation-of-scripts), but it seems to be pretty unavoidable that you have to be able to switch between different path-environments to use the full set of tools available on Windows.

paulharris commented 2 years ago

Actually in my situation, I'd be happy with Windows paths in a environment.sh file. I'm in msys, I'm using bash, but building Windows-native - ie does not need to run on the msys subsystem.

This already behaves this way when you run "conan install . -g VirtualRunEnv" -- you get the .sh files only. So I can't use that build and switch to a DOS / MSVC environment to continue working there ...

It is only when you run "conan install package/1.0@ -g VirtualRunEnv" that you get the .bat files instead of the .sh files.

paulharris commented 2 years ago

I got VirtualRunEnv to finally work, but I had to: add to the profile:

tools.microsoft.bash:subsystem=msys
tools.microsoft.bash:path=None (I suppose could be left out)

And to the recipe: win_bash = True

It is very clunky compared to the old virtualenv, I would have to edit the recipe, adjust the profile, "conan install" and then finally be able to use (eg) the DOS prompt.

I hope #11066 makes this easier ...

memsharded commented 1 year ago

https://github.com/conan-io/conan/pull/12178 merged, this will be in next 1.54

conan-io / conan

[question/feature] conan.tools.env » VirtualRunEnv does not generate sh scripts on windows #10900