vcpkg ignores msvc toolset version set on the command-line

[patrikhuber]

Describe the bug

When run in manifest mode "manually" from the command-line, say I have a directory with only a vcpkg.json and vcpkg-configuration.json, and then run ."$env:VCPKG_ROOT/vcpkg.exe" install (where VCPKG_ROOT isC:\Program Files\Microsoft Visual Studio\2022\Community\VC\vcpkg), vcpkg always finds the latest compiler / toolchain version: For example here it will proceed building the packages with msvc 14.40:

C:\a> ."$env:VCPKG_ROOT/vcpkg.exe" install
Detecting compiler hash for triplet x64-windows...
Compiler found: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.40.33807/bin/Hostx64/x64/cl.exe
The following packages will be built and installed:
    opencv4[contrib,core,cuda,jpeg,png]:x64-windows@4.8.0#19 -- C:\Users\PatrikHuber\AppData\Local\vcpkg\registries\git-trees\85685a5e45ef916a21769da98c8346462ef179d0

even though I have run the command from a terminal where I've explicitly opened a VS developer prompt with MSVC version 14.39 using .\vcvarsall.bat amd64 -vcvars_ver="14.39".

vcpkg should not just ignore this and use 14.40 (the latest installed). If I'm in a VS dev prompt with 14.39 activated, vcpkg needs to respect that, or offer some other way to specify the buildtools version. (I know this could be done with a CMakePresets.json that contains a toolchain=14.39 setting but that doesn't help in this case as I am not actually building something with CMake myself, I'm just running vcpkg on a terminal on a vcpkg.json file in manifest mode.

Environment

• OS: Windows 11 23H2, latest vcpkg as of 22/06/2024
• Compiler: 14.39/14.40

To Reproduce See above

Expected behavior See above

Failure logs See above

Additional context

[Neumann-A]

even though I have run the command from a terminal where I've explicitly opened a VS developer prompt with MSVC version 14.39 using .\vcvarsall.bat amd64 -vcvars_ver="14.39".

vcpkg does not use that environment unless you manually set a toolchain in the triplet. You can also manually set the toolset in the triplet https://learn.microsoft.com/en-us/vcpkg/users/triplets#vcpkg_platform_toolset_version to select the toolset.

[patrikhuber]

@Neumann-A Thanks for the quick reply. I'm not sure triplets are an option. After building a package in the above way, I'm storing it using vcpkg's binary caching mechanism, and the created binary cache file then needs to work on other dev machines who are all using the default triplet. Is the triplet part of the hash that vcpkg computes for its binary cache artefacts? I remember seeing a table in the vcpkg docs of what parts go into the hash, but I can't find it anymore now. I have a feeling though it will be - in which case this won't work. In the above example, how could I set VCPKG_PLATFORM_TOOLSET_VERSION, without creating a custom triplet?

[Neumann-A]

who are all using the default triplet.

Why?

triplet part of the hash that vcpkg computes for its binary cache artefacts?

Yes

In the above example, how could I set VCPKG_PLATFORM_TOOLSET_VERSION, without creating a custom triplet?

There is non.

I find it funny: People want fine control but then somehow want that fine control to magically to transfer to others without being specific somehow.

[patrikhuber]

Why?

Simplicity of dev setup and workflow. Don't make things more complex than they need to be. C++ software development and dependency management is complex enough as it is.

People want fine control but then somehow want that fine control to magically to transfer to others without being specific somehow.

I think the intuitive thing (which I would have expected) would have been for vcpkg to pick up the vcvarsall settings. But I think I understand why it doesn't / can't - it sandboxes its environment (which is good, after all).

From a user perspective, it's a shame that the following won't work:

• On a dev machine, have a CMakePresets.json with "toolset": { "value": "host=x64,version=14.39" } (this is needed because CUDA <12.5 won't build with msvc-14.40, and some dependencies need CUDA <12.5 - so we're stuck on 14.39); while at the same time being able to use the x64-windows default triplet, which works just fine.
• One of the dependencies in vcpkg requires 2hrs to build, so we don't want to do that on local dev machines, but we want to build that on CI and use vcpkg's binary caching on a shared drive. So we build that using a simple vcpkg.json with that dependency (without any CMake) and just call vcpkg in Manifest mode on that vcpkg.json (via vcpkg install in that directory). Now it seems like unfortunately we can't do that as that will always use the latest compiler and there is no way to use CMakePresets.json here (unless we create a "dummy" project just to compile the dependency). So we have to create a custom triplet, but then all the devs will have to use that custom triplet too on their local machines. Unnecessary complexity.

This would actually all "magically" work if we installed an older version of VS on the CI machine which only contains 14.39 (unfortuantely, on the latest version, it's not possible to uninstall 14.40, as it's the latest/default package, and trying to uninstall it comes with a warning that the whole CMake and other workloads have to be uninstalled too). Then vcpkg would have no choice but to pick that up, with the default triplet, hash & upload the binary file, and then vcpkg on our dev machines would pick that package up as the hash should match. So for this use case, it would be better if the triplet is actually not part of the binary package hash (as the compiler version already differentiates between the two), or if there was another way of injecting the toolset version, like via CMakePresets.json, but only for building the dependency.

So in a way, no, I don't really want the fine control in the first place (I wish things would just work...), but unfortunately the C++ ecosystem often forces you to need it, as in this example, newer compiler version breaks CUDA, breaks dependencies, etc... And it's also not as simple as upgrading to CUDA 12.5 (I wish) because then it's OpenCV-4.8 in vcpkg that doesn't work yet with CUDA 12.5 and/or msvc 14.40. That's the not-so-fun side of C++ sadly.

Anyway, thank you very much for quickly replying and for the advice on what is and what's not possible (and how)!

[Neumann-A]

From your answer:

On a dev machine, have a CMakePresets.json

Just control vcpkg related vars like the triplet also from the preset? (You need to control vcpkg install options anyway to get cache hits....)

[patrikhuber]

Just control vcpkg related vars like the triplet also from the preset? (You need to control vcpkg install options anyway to get cache hits....)

That's perhaps a good point. We haven't historically committed CMakePresets.json to git because there are some dev-local paths in there (some dependencies are not in vcpkg, and different devs have it in different locations). I think I'll have to look into CMakeUserPresets.json and whether, if both CMakePresets.json and CMakeUserPresets.json are present, VS reads both of these, or only one of them. If we can store the toolchain and triplet info in CMakePresets.json and then devs can set/override machine-local paths in CMakeUserPresets.json, and VS reads from both, that could work quite well!

The cache we can actually configure with environment variables, but you're right, these are a bit of a hassle to set up as well, if those could also go into CMakePresets.json into git, that would be amazing, but I don't think they can? VCPKG_BINARY_SOURCES is an environment variable (https://learn.microsoft.com/en-us/vcpkg/reference/binarycaching), not a variable that can be injected via CMakePresets.json, or can it? (And what do you mean by "control vcpkg install options anyway to get cache hits"?)

[Neumann-A]

And what do you mean by "control vcpkg install options anyway to get cache hits"?

--x-abi-tools-use-exact-versions; https://learn.microsoft.com/en-us/vcpkg/users/binarycaching-troubleshooting#cause-2-the-library-was-built-on-a-different-machine-than-the-machine-used-to-consume-it-1

[patrikhuber]

--x-abi-tools-use-exact-versions; https://learn.microsoft.com/en-us/vcpkg/users/binarycaching-troubleshooting#cause-2-the-library-was-built-on-a-different-machine-than-the-machine-used-to-consume-it-1

That is useful to know, thank you! However, out of all these https://github.com/Microsoft/vcpkg/blob/master/scripts/vcpkgTools.xml, only the CMake version is part of the binary cache package hash, if I remember correctly from reading it a couple weeks ago (as mentioned I can't find the docs link to this anymore...). And VS/MSVC is not part of that XML file, so the build tools needs to be handled via triplet (or CMakePresets toolchain variable) anyway.