[vcpkg-tool] x-download Downloading files with extra headers (basic auth) returns error 2

[robertobernabe]

Describe the bug Hi there! I need your help. We're getting strange download errors since we switched from latest release tag (2021.05) to the latest master in vcpkg. This applies only to Windows Server 2016. On newer Windows version this doesn't happen.

The crazy thing here...when i'm executing the curl command itself it works...but executed trough vcpkg (vcpkg x-download command) it fails. Important to know is that we are passing a basic authentication header into the cmake function vcpk_download_distfile to download our own overlay ports which are hosted on artifactory. Downloading files without passing in any basic auth headers works fine on Windows Server 2016.

[DEBUG] cmd_execute_and_stream_data() returned 2

Environment

• OS: Only on Windows Server 2016

To Reproduce Steps to reproduce the behavior:

PS C:\Users\Administrator> (Get-WmiObject -class Win32_OperatingSystem).Caption
Microsoft Windows Server 2016 Standard Evaluation
PS C:\Users\Administrator> [Environment]::OSVersion

Platform ServicePack Version      VersionString
-------- ----------- -------      -------------
 Win32NT             10.0.14393.0 Microsoft Windows NT 10.0.14393.0

PS C:\Users\Administrator\vcpkg> .\bootstrap-vcpkg.bat
Downloading https://github.com/microsoft/vcpkg-tool/releases/download/2021-11-02/vcpkg.exe -> C:\Users\Administrator\vcp
kg\vcpkg.exe... done.
Validating signature... done.

Telemetry
---------
vcpkg collects usage data in order to help us improve your experience.
The data collected by Microsoft is anonymous.
You can opt-out of telemetry by re-running the bootstrap-vcpkg script with -disableMetrics,
passing --disable-metrics to vcpkg on the command line,
or by setting the VCPKG_DISABLE_METRICS environment variable.

Read more about vcpkg telemetry at docs/about/privacy.md
PS C:\Users\Administrator\vcpkg> .\vcpkg.exe x-download  "c:\some\file.part" --skip-sha512 --url="https://some/server/file.zip" --debug --feature-flags=-manifests --header="Authorization: Basic ******"
[DEBUG] Feature flag 'binarycaching' unset
[DEBUG] Feature flag 'manifests' = off
[DEBUG] Feature flag 'compilertracking' unset
[DEBUG] Feature flag 'registries' unset
[DEBUG] Feature flag 'versions' unset
[DEBUG] CreateProcessW(curl --fail -L https://some/server/file.zip --create-dirs --output "c:\some\file.part"
-H "Authorization: Basic **********")
[DEBUG] cmd_execute_and_stream_data() returned 2 after     2008 us
Error: Failed to download from mirror set:
https://some/server/file.zip:

[DEBUG] D:\a\_work\1\s\src\vcpkg\base\downloads.cpp(656)
[DEBUG] Exiting after %s (%d us)

CMake Error at scripts/cmake/vcpkg_download_distfile.cmake:83 (message):

      Failed to download file with error: 1
      If you use a proxy, please check your proxy setting. Possible causes are:

      1. You are actually using an HTTP proxy, but setting HTTPS_PROXY variable
         to `https://address:port`. This is not correct, because `https://` prefix
         claims the proxy is an HTTPS proxy, while your proxy (v2ray, shadowsocksr
         , etc..) is an HTTP proxy. Try setting `http://address:port` to both
         HTTP_PROXY and HTTPS_PROXY instead.

      2. You are using Fiddler. Currently a bug (https://github.com/microsoft/vcpkg/issues/17752)
         will set HTTPS_PROXY to `https://fiddler_address:port` which lead to problem 1 above.
         Workaround is open Windows 10 Settings App, and search for Proxy Configuration page,
         Change `http=address:port;https=address:port` to `address`, and fill the port number.

      3. You proxy´s remote server is out of service.

      In future vcpkg releases, if you are using Windows, you no longer need to set
      HTTP(S)_PROXY environment variables. Vcpkg will simply apply Windows IE Proxy
      Settings set by your proxy software. See (https://github.com/microsoft/vcpkg-tool/pull/49)
      and (https://github.com/microsoft/vcpkg-tool/pull/77)

      Otherwise, please submit an issue at https://github.com/Microsoft/vcpkg/issues

Call Stack (most recent call first):
  scripts/cmake/vcpkg_download_distfile.cmake:291 (z_vcpkg_download_distfile_show_proxy_and_fail)
  C:/bamboo-build/PWPPSDK-WPVCPKGPORTS57-BAPS/base-portfiles/someportfile.cmake:32 (vcpkg_download_distfile)
  C:/bamboo-build/PWPPSDK-WPVCPKGPORTS57-BAPS/release/someport/someportfile.cmake:2 (include)
  scripts/ports.cmake:142 (include)

Expected behavior

PS C:\Users\Administrator\vcpkg> .\vcpkg.exe x-download "c:\some\file.part" --skip-sha512 --url="https://some/server/file.zip" --debug --feature-flags=-manifests --header="Authorization: Basic ******"
[DEBUG] Feature flag 'binarycaching' unset
[DEBUG] Feature flag 'manifests' = off
[DEBUG] Feature flag 'compilertracking' unset
[DEBUG] Feature flag 'registries' unset
[DEBUG] Feature flag 'versions' unset
[DEBUG] Downloading https://some/server/file.zip
[DEBUG] D:\a\_work\1\s\src\vcpkg\commands.xdownload.cpp(143)

Failure logs -(please attached failure logs)

Additional context Add any other context about the problem here.

[JackBoosY]

Can't reproduce this issue using the following sample command:

 .\vcpkg.exe x-download f:\\tls12-download-arm64.exe --skip-sha512 --url="https://github.com/microsoft/vcpkg-tool/releases/download/2021-11-02/tls12-download-arm64.exe"

[JackBoosY]

Can you please add --x-cmake-args=--trace-expand to the x-download command, run it and provide the output?

Thanks.

[BillyONeal]

Curl error 1 is "unsupported protocol" but the protocol is clearly https here..... does your server 2016 system have a proxy configured? We attempt to get curl to use the same proxy servers that WinHTTP would use by setting HTTP_PROXY and HTTPS_PROXY environment variables...

[robertobernabe]

@JackBoosY without adding any basic authentication headers it works. As soon as i add --header="Authorization: Basic ******" if fails - ONLY on Windows Server 2016! It works on Windows Server 2019/ Windows 10. I will provide the output passing the --x-cmake-args=--trace-expand to x-download command.

@BillyONeal We didn't set any HTTP_PROXY. [DEBUG] cmd_execute_and_stream_data() returned 2 is 2 the exit code of curl itself?

[robertobernabe]

I've found the root cause. vcpkg x-download relies on system-wide curl installation. Since curl is shipped starting on Windows 2019, and we are using Windows 2016 - lucky me. After installing curl manually and adding it to the PATH env variable - it worked. Unfortunately Windows Server 2016 ships with a curl alias in PowerShell - so that was the reason why it worked without passing in any headers.

PS C:\Users\Administrator\vcpkg> get-command curl

CommandType     Name                                               Version    Source
-----------     ----                                               -------    ------
Alias           curl -> Invoke-WebRequest

which is just an alias for Invoke-WebRequest.

Really nice one.

Maybe it would make sense to rely on your own copy of curl and not on a system-wide one? Git for windows (which you are already relying on ships with a curl binary as far as I know).

[BillyONeal]

I've found the root cause. vcpkg x-download relies on system-wide curl installation. Since curl is shipped starting on Windows 2019, and we are using Windows 2016 - lucky me.

That doesn't explain why you get different behavior if and only if you add extra headers though.

Unfortunately Windows Server 2016 ships with a curl alias in PowerShell - so that was the reason why it worked without passing in any headers.

That shouldn't work because when vcpkg invokes the thing it doesn't do that from a powershell prompt, it does a CreateProcess. So if this were the issue it shouldn't ever work, not depend on whether headers are used.

Maybe it would make sense to rely on your own copy of curl and not on a system-wide one?

That's kind of a catch-22, considering it's what we're doing to download things :). Although I was under the impression that we usually tried to use WinHTTP rather than curl.exe on Windows....

[robertobernabe]

@BillyONeal You're right - it has nothing to do with the PowerShell alias.

That doesn't explain why you get different behavior if and only if you add extra headers though.

For me, it looks like that curl gets only invoked if I pass in extra headers.

[BillyONeal]

For me, it looks like that curl gets only invoked if I pass in extra headers.

Ah, maybe the WinHTTP backend doesn't understand extra headers, that would do it

[github-actions[bot]]

This is an automated message. Per our repo policy, stale issues get closed if there has been no activity in the past 180 days. The issue will be automatically closed in 14 days. If you wish to keep this issue open, please add a new comment.