[powershell] vcpkg doesn't work with PowerShell installed from Windows Store.

[zomgrolf]

Describe the bug It seems that vcpkg doesn't work with the version of PowerShell Core installed from the Windows Store. The initial check seems to pass and vcpkg doesn't download its local copy, but vcpkg_copy_tool_dependencies issues an error when trying to build a port that uses Powershell.

Environment

• OS: Windows 10 Pro 20H2
• Compiler: 19.28.29914 (VS 2019 16.9.4)

To Reproduce Steps to reproduce the behavior:

1. Install Powershell from the Windows Store (version 7.1.3)
2. Sync the vcpkg repo (e.g. to c:/tools/vcpkg); run bootstrap-vcpkg.bat
3. In a directory of your choice set the following environment variables

   set VCPKG_TOOLCHAIN_FILE=C:/tools/vcpkg/scripts/buildsystems/vcpkg.cmake
   set VCPKG_FEATURE_FLAGS=manifests,versions
   set VCPKG_MANIFEST_DIR=%~dp0

   And edit the vcpkg.json file:

   {
   "$schema": "https://raw.githubusercontent.com/microsoft/vcpkg/master/scripts/vcpkg.schema.json",
   "name": "brotli-test",
   "version": "0.0.1",
   "builtin-baseline": "5a10163c44526b5d42a67480ce7a692a36948761",
   "dependencies": [
       "brotli"
   ]
   }

4. run vcpkg install --binarysource=clear

Expected behavior Build succeeding

Failure logs

Detecting compiler hash for triplet x86-windows...
The following packages will be built and installed:
    brotli[core]:x86-windows -> 1.0.9#1 -- C:\tools\vcpkg\buildtrees\versioning\versions\brotli\8f55fe158d8bd753a6e6908164e03ae4f0b73cea
Starting package 1/1: brotli:x86-windows
Building package brotli[core]:x86-windows...
-- Installing port from location: C:\tools\vcpkg\buildtrees\versioning\versions\brotli\8f55fe158d8bd753a6e6908164e03ae4f0b73cea
-- Downloading https://github.com/google/brotli/archive/e61745a6b7add50d380cfd7d3883dd6c62fc2c71.tar.gz -> google-brotli-e61745a6b7add50d380cfd7d3883dd6c62fc2c71.tar.gz...
-- Extracting source C:/tools/vcpkg/downloads/google-brotli-e61745a6b7add50d380cfd7d3883dd6c62fc2c71.tar.gz
-- Applying patch install.patch
-- Applying patch fix-arm-uwp.patch
-- Applying patch pkgconfig.patch
-- Applying patch fix-ios.patch
-- Using source at C:/tools/vcpkg/buildtrees/brotli/src/6c62fc2c71-f2afd323e3.clean
-- Found external ninja('1.10.2').
-- Configuring x86-windows
-- Building x86-windows-dbg
-- Building x86-windows-rel
CMake Error at scripts/cmake/vcpkg_copy_tool_dependencies.cmake:29 (message):
  Could not find PowerShell Core; please open an issue to report this.
Call Stack (most recent call first):
  buildtrees/versioning/versions/brotli/8f55fe158d8bd753a6e6908164e03ae4f0b73cea/portfile.cmake:24 (vcpkg_copy_tool_dependencies)
  scripts/ports.cmake:142 (include)

Error: Building package brotli:x86-windows failed with: BUILD_FAILED
Please ensure you're using the latest portfiles with `.\vcpkg update`, then
submit an issue at https://github.com/Microsoft/vcpkg/issues including:
  Package: brotli:x86-windows
  Vcpkg version: 2021-01-13-d67989bce1043b98092ac45996a8230a059a2d7e

Additionally, attach any relevant sections from the log files above.

Additional context where pwsh.exe issued in the same command prompt prints out: c:\Users\zomgrolf\AppData\Local\Microsoft\WindowsApps\pwsh.exe

Running vcpkg with the --debug flag produces this:

[DEBUG] Found path: c:\users\zomgrolf\AppData\Local\Microsoft\WindowsApps\pwsh.exe
[DEBUG] Found path: C:\Users\zomgrolf\AppData\Local\Microsoft\WindowsApps\pwsh.exe
[DEBUG] CreateProcessW("c:\users\zomgrolf\AppData\Local\Microsoft\WindowsApps\pwsh.exe" --version)
[DEBUG] cmd_execute_and_stream_data() returned 0 after   330190 us

There is no copy of PowerShell-7.1.0-win-x86.zip in c:/tools/vcpkg/downloads

If I replace the version from Windows Store with a version installed from the *.msi installer the build succeeds. It seems that CMake is unable to find the Windows Store version of PowerShell?

[JackBoosY]

Did you use powershell or cmd?

[zomgrolf]

cmd

[JackBoosY]

@zomgrolf Please use powershell instead. Ignore my nonsense.

[zomgrolf]

You have to be joking.

[JackBoosY]

@zomgrolf Forgive me... I didn't understand clearly.

[JackBoosY]

@strega-nil Can you please take a look? I think this is a bug of vcpkg.

[zomgrolf]

My guess is it's an issue with CMake (or CMake scripts used by vcpkg?). The method vcpkg uses to test if there's a suitable version of PowerShell available seems to succeed (as expected), however the call to find_program(PWSH_EXE pwsh) in vcpkg_copy_tool_dependencies.cmake fails (unexpectedly). I would expect that call to succeed, not sure why it doesn't. I've just tried this simple CMake script in a separate command prompt:

cmake_minimum_required(VERSION 3.19)

project(TEST)

find_program(PWSH_EXE pwsh)
if (NOT PWSH_EXE)
    message(FATAL_ERROR "Could not find PowerShell Core; please open an issue to report this.")
endif()

add_executable(test test.cpp)

And it fails with the Windows Store version of PowerShell Core. However, CMake clearly is the lowest common denominator here -- so until this is fixed, I would suggest making vcpkg a bit more eager to download a local copy of PowerShell.

[zomgrolf]

All right, this is getting odd: I've re-run the script with CMAKE_FIND_DEBUG_MODE set to true, and CMake reports that C:/Users/zomgrolf/AppData/Local/Microsoft/WindowsApps/pwsh.exe is on the list of locations considered by find_program -- that's the location reported by where pwsh.exe for a Windows Store version of PowerShell.

[JackBoosY]

The code for detecting pwsh is c++, and the code for finding and using pwsh is cmake.

[BillyONeal]

Does the C:/Users/zomgrolf/AppData/Local/Microsoft/WindowsApps/pwsh.exe need some kind of environment variable we're stripping or something like that?

[zomgrolf]

I don't think so. All I did was install PowerShell from the Windows Store, open a new command prompt in Windows Terminal, and then try to install the libraries. Same thing, if I simply open a new command prompt and try to run CMake using that simple script above, find_program simply fails, so I think ultimately this is an issue with CMake, unless that call to find_program is wrong and a different method should be used for the Store apps. The call to find_program seems to fail in the exact same way whether it's made by CMake executed by vcpkg or manually by me from the command prompt, so I don't think it's anything that vcpkg does.

This is with CMake 3.20.0

[zomgrolf]

It looks very similar to #15681

[JackBoosY]

@zomgrolf Yes, almost same issue.

[zomgrolf]

I think the underlying issue here is that, when installed from the Microsoft Store, C:/Users/zomgrolf/AppData/Local/Microsoft/WindowsApps/pwsh.exe is a symlink to a file inside a Windows app package, rather than a stand-alone executable.

I've dug a little deeper into how find_program in CMake works: for any candidate it performs two tests: it checks if the file at the given location exists and if it is executable.

The code that checks if the given file exists is here: SystemTools::FileExists

It calls GetFileAttributesW on the given path, checks for the presence of the FILE_ATTRIBUTE_REPARSE_POINT to detect a symlink, then calls CreateFileW. On my machine that second call fails with what looks like a security error, which shouldn't be surprising, I guess, given how tightly locked down Windows app packages are. This is what most likely makes CMake ignore the seemingly valid PowerShell executable.

[cjwijtmans]

I have the same issue, failing to build brotli so it seems brotli specific. The annoying part is other packages that build just fine while detecting powershell(from the windows store).

C:\usr\src\vcpkg>vcpkg list
fmt:x64-windows                                    7.1.3#4          Formatting library for C++. It can be used as a ...
fmt:x86-windows                                    7.1.3#4          Formatting library for C++. It can be used as a ...
jsoncpp:x64-windows                                1.9.4            jsoncpp is an implementation of a JSON reader an...
jsoncpp:x86-windows                                1.9.4            jsoncpp is an implementation of a JSON reader an...
sdl2:x64-windows                                   2.0.14#4         Simple DirectMedia Layer is a cross-platform dev...
sdl2:x86-windows                                   2.0.14#4         Simple DirectMedia Layer is a cross-platform dev...
vcpkg-cmake-config:x64-windows                     2021-05-22
vcpkg-cmake-config:x86-windows                     2021-05-22
vcpkg-cmake:x64-windows                            2021-02-28#2
vcpkg-cmake:x86-windows                            2021-02-28#2

[thomasgt]

I've also experienced this problem. In my case, it happens when installing brotli on a project that uses a vcpkg.json manifest file. I am working in VS Code with the C++ and CMake extensions, and I notice the error when configuring the project, which triggers vcpkg to build and install my dependencies because I am using a manifest file. If I launch VS Code from my desktop or Start menu, the problem manifests. If I launch VS Code from my PowerShell Core shell, everything works.

[yuriy-odonnell-epic]

This problem is indeed caused by a bug in cmake (as zomgrolf pointed out). A merge request is now open against cmake's platform abstraction library to fix it: https://gitlab.kitware.com/utils/kwsys/-/merge_requests/236.

[yuriy-odonnell-epic]

Unfortunately, it does not seem like the fix is going to land after all. Allowing find_program to find execution aliases / Store apps immediately breaks python test case, as Windows creates python.exe alias out of the box, which is not actually a real python. At this point I have no energy or will to continue looking at this or trying to submit further patches to cmake, as it seems like a somewhat self-inflicted wound by Microsoft :(

[yuriy-odonnell-epic]

The CMake issue was addressed after all. The dummy python alias (AppInstallerPythonRedirector) is explicitly banned. Theoretically, other store apps should work in the future CMake release.

[emptyVoid]

Will this ever be fixed for older CMake versions? (I cannot upgrade CMake beyond 3.19)

[yuriy-odonnell-epic]

@emptyVoid please CMake developers directly in their issue tracker.

[emptyVoid]

@emptyVoid please CMake developers directly in their issue tracker.

My question has nothing to do with CMake.

vcpkg @ 59a8a9c6248ed230b19028a44484ae5f06db697d worked flawlessly without requiring specific flavors of third-party utilities installed in the system (by downloading all the required tools by itself). I would stay with that revision if it worked with VS 2022, sadly it's unable to find VS with VCPKG_VISUAL_STUDIO_PATH and VCPKG_PLATFORM_TOOLSET specified (the latest revision has no issue with the same triplet file).

Thus the question is about fixing this issue in this repository.

[BillyONeal]

@emptyVoid please CMake developers directly in their issue tracker.

My question has nothing to do with CMake.

vcpkg @ 59a8a9c worked flawlessly without requiring specific flavors of third-party utilities installed in the system (by downloading all the required tools by itself). I would stay with that revision if it worked with VS 2022, sadly it's unable to find VS with VCPKG_VISUAL_STUDIO_PATH and VCPKG_PLATFORM_TOOLSET specified (the latest revision has no issue with the same triplet file).

We have never downloaded all required utilities if usable versions were present on the system. Powershell is a little different here because of the "powershell" vs "pwsh" thing of course.

Thus the question is about fixing this issue in this repository.

Depending on the path by which you get to a place, our options are limited. If we are invoked via CMake integration with manifests (as in #21203 ) we have no opportunity to run first and download things. (That said the powershell in ##21203 is very much not coming from the Windows Store)

Ultimately the best fix would be to get powershell out of this dependency chain. I had a prototype of something like this but ran into the snag that some ports explicitly depend on powershell, for example: https://github.com/microsoft/vcpkg/blob/master/ports/qt5-base/qtdeploy.ps1 --- and I ran out of time attempting to come up with a solution at that time. Maybe something that can be picked up later...

[BillyONeal]

@emptyVoid For now can you confirm that installing powershell-core on the system from https://github.com/PowerShell/PowerShell/releases/tag/v7.2.0 temporarily resolves the problem?

[emptyVoid]

We have never downloaded all required utilities if usable versions were present on the system.

Yeah, I do realize that, but still that revision did download some newer PowerShell (at least I remember seeing it in the build log).

@emptyVoid For now can you confirm that installing powershell-core on the system from https://github.com/PowerShell/PowerShell/releases/tag/v7.2.0 temporarily resolves the problem?

Depends on the install options: vcpkg is able to locate it only if I agree to add PS to PATH.

[BillyONeal]

Depends on the install options: vcpkg is able to locate it only if I agree to add PS to PATH.

Makes sense.

If anyone wants to take a crack at designing the way ports like qt5-base should work without powershell and wants to start from my work-in-progress, here you go: https://github.com/BillyONeal/vcpkg-tool/tree/replace-powershell

I suspect we need to let ports create some kind of a marker file next to their DLLs that says 'if this DLL is applocally deployed, also applocal deploy these files/directories/DLLs' or similar.

[BillyONeal]

(I want to resolve this problem but don't expect I'll have time to look again before mid-December)

[dg0yt]

@BillyONeal Wrt to applocal.ps1 I would still suggest to try CMake first.

[BillyONeal]

@BillyONeal Wrt to applocal.ps1 I would still suggest to try CMake first.

When applocal.ps1 there is not necessarily any cmake available.