search

microsoft / vcpkg

C++ Library Manager for Windows, Linux, and MacOS
MIT License
23.2k stars 6.39k forks source link

vcpkg_from_gitlab fails to download but project exists / URL is correct #38687

Closed patrikhuber closed 5 months ago

patrikhuber commented 6 months ago

Describe the bug vcpkg-custom-overlay/my-test-package/portfile.cmake with vcpkg_from_gitlab:

vcpkg_from_gitlab(
    GITLAB_URL https://gitlab.com
    OUT_SOURCE_PATH SOURCE_PATH
    REPO myorg/mysubgroup/my-test-package
    REF "${VERSION}"
    SHA512 0
    HEAD_REF main
)

vcpkg_cmake_configure(
    SOURCE_PATH "${SOURCE_PATH}"
)

vcpkg_cmake_install()

vcpkg_cmake_config_fixup(PACKAGE_NAME "my-test-package")

file(REMOVE_RECURSE "${CURRENT_PACKAGES_DIR}/debug/include")

# file(INSTALL "${SOURCE_PATH}/LICENSE" DESTINATION "${CURRENT_PACKAGES_DIR}/share/${PORT}" RENAME copyright)
configure_file("${CMAKE_CURRENT_LIST_DIR}/usage" "${CURRENT_PACKAGES_DIR}/share/${PORT}/usage" COPYONLY)

Then installing it from a custom overlay with:

.\vcpkg.exe install my-test-package --overlay-ports=C:\Users\MyUser\vcpkg-custom-overlay --head

It fails at:

Downloading https://gitlab.com/api/v4/projects/%6d%79%6f%72%67%2f%6d%79%73%75%62%67%72%6f%75%70%2f%6d%79%2d%74%65%73%74%2d%70%61%63%6b%61%67%65/repository/branches/main
error: Failed to download from mirror set
error: https://gitlab.com/api/v4/projects/%6d%79%6f%72%67%2f%6d%79%73%75%62%67%72%6f%75%70%2f%6d%79%2d%74%65%73%74%2d%70%61%63%6b%61%67%65/repository/branches/main: failed: status code 404

and

Downloading https://gitlab.com/api/v4/projects/%6d%79%6f%72%67%2f%6d%79%73%75%62%67%72%6f%75%70%2f%6d%79%2d%74%65%73%74%2d%70%61%63%6b%61%67%65/repository/branches/main
error: Failed to download from mirror set
error: https://gitlab.com/api/v4/projects/%6d%79%6f%72%67%2f%6d%79%73%75%62%67%72%6f%75%70%2f%6d%79%2d%74%65%73%74%2d%70%61%63%6b%61%67%65/repository/branches/main: failed: status code 404

BUT: If I put the above "encoded" URL into the browser (https://gitlab.com/api/v4/projects/%6d%79%6f%72%67%2f%6d%79%73%75%62%67%72%6f%75%70%2f%6d%79%2d%74%65%73%74%2d%70%61%63%6b%61%67%65/repository/branches/main), it resolves to https://gitlab.com/api/v4/projects/%6d%79%6f%72%67%2f%6d%79%73%75%62%67%72%6f%75%70%2f%6d%79%2d%74%65%73%74%2d%70%61%63%6b%61%67%65/repository/branches/main, and this results in a valid reply, not a 404:

image

(Note this is a private repo and I've changed the names, so if you run the above line in your browser, you will get a 404 in the browser too obviously as you won't have access to our internal project - but I don't get a 404 in the browser if I try it with the real internal name of the project.)

This should be easily reproducible by creating your own (even empty) GitLab repo and trying vcpkg_from_gitlab. It looks like something within vcpkg_from_gitlab is going wrong?

Environment

To Reproduce See above.

Expected behavior As the URL is clearly correct when manually putting it into the browser, I'd expect vcpkg_from_gitlab to be able to not display a 404, or give me a more descriptive error message.

Failure logs See above & below output.

Additional context Full cmdline output:

C:\Users\MyUser\vcpkg-clean [master ≡]> .\vcpkg.exe install my-test-package --overlay-ports=C:\Users\MyUser\Projects\vcpkg-custom-overlay --head
Computing installation plan...
The following packages will be built and installed:
    my-test-package:x64-windows@1.0.0 (+HEAD) -- C:\Users\MyUser\Projects\vcpkg-custom-overlay\my-test-package
Installing 1/1 my-test-package:x64-windows@1.0.0...
Building my-test-package:x64-windows@1.0.0 from HEAD...
C:\Users\MyUser\Projects\vcpkg-custom-overlay\my-test-package: info: installing overlay port from here
-- Downloading https://gitlab.com/api/v4/projects/%6d%69%6d%65%74%72%69%6b%2f%66%61%63%65%62%6f%77%2f%66%61%63%65%62%6f%77%66%69%6c%65%72%65%61%64%65%72/repository/branches/main -> myorg-mysubgroup-my-test-package-main.tar.gz.version...
[DEBUG] To include the environment variables in debug output, pass --debug-env
[DEBUG] Trying to load bundleconfig from C:\Users\MyUser\vcpkg-clean\vcpkg-bundle.json
[DEBUG] Failed to open: C:\Users\MyUser\vcpkg-clean\vcpkg-bundle.json
[DEBUG] Bundle config: readonly=false, usegitregistry=false, embeddedsha=nullopt, deployment=Git, vsversion=nullopt
[DEBUG] Metrics enabled.
[DEBUG] Feature flag 'binarycaching' unset
[DEBUG] Feature flag 'compilertracking' unset
[DEBUG] Feature flag 'registries' unset
[DEBUG] Feature flag 'versions' unset
[DEBUG] Feature flag 'dependencygraph' unset
Downloading https://gitlab.com/api/v4/projects/%6d%79%6f%72%67%2f%6d%79%73%75%62%67%72%6f%75%70%2f%6d%79%2d%74%65%73%74%2d%70%61%63%6b%61%67%65/repository/branches/main
error: Failed to download from mirror set
error: https://gitlab.com/api/v4/projects/%6d%79%6f%72%67%2f%6d%79%73%75%62%67%72%6f%75%70%2f%6d%79%2d%74%65%73%74%2d%70%61%63%6b%61%67%65/repository/branches/main: failed: status code 404
[DEBUG] D:\a\_work\1\s\src\vcpkg\base\downloads.cpp(997):
[DEBUG] Time in subprocesses: 0us
[DEBUG] Time in parsing JSON: 2us
[DEBUG] Time in JSON reader: 0us
[DEBUG] Time in filesystem: 872us
[DEBUG] Time in loading ports: 0us
[DEBUG] Exiting after 233 ms (216853us)

CMake Error at scripts/cmake/vcpkg_download_distfile.cmake:32 (message):

      Failed to download file with error: 1
      If you are using a proxy, please check your proxy setting. Possible causes are:

      1. You are actually using an HTTP proxy, but setting HTTPS_PROXY variable
         to `https://address:port`. This is not correct, because `https://` prefix
         claims the proxy is an HTTPS proxy, while your proxy (v2ray, shadowsocksr
         , etc..) is an HTTP proxy. Try setting `http://address:port` to both
         HTTP_PROXY and HTTPS_PROXY instead.

      2. If you are using Windows, vcpkg will automatically use your Windows IE Proxy Settings
         set by your proxy software. See https://github.com/microsoft/vcpkg-tool/pull/77
         The value set by your proxy might be wrong, or have same `https://` prefix issue.

      3. Your proxy's remote server is out of service.

      If you've tried directly download the link, and believe this is not a temporary
      download server failure, please submit an issue at https://github.com/Microsoft/vcpkg/issues
      to report this upstream download server failure.

Call Stack (most recent call first):
  scripts/cmake/vcpkg_download_distfile.cmake:270 (z_vcpkg_download_distfile_show_proxy_and_fail)
  scripts/cmake/vcpkg_from_gitlab.cmake:97 (vcpkg_download_distfile)
  C:/Users/MyUser/Projects/vcpkg-custom-overlay/my-test-package/portfile.cmake:1 (vcpkg_from_gitlab)
  scripts/ports.cmake:175 (include)

error: building my-test-package:x64-windows failed with: BUILD_FAILED
Elapsed time to handle my-test-package:x64-windows: 1.2 s
patrikhuber commented 6 months ago

In case this is an authentication issue (as our repository is private):

I can successfully do: git clone https://gitlab.com/myorg/mysubgroup/my-test-package.git from the same command-line I'm running vcpkg, without having to enter any username/password. I'm using Windows 11 and git-for-Windows with the (git/Windows) credentials manager. So presumably vcpkg's git should be able to have access to this repo, or does it use a separate git or credentials store?

I've read through https://learn.microsoft.com/en-us/vcpkg/users/authentication as well. I've tried "pre-seeding" (not sure if this is needed in my setup...):

"url=https://gitlab.com`npath=myorg/mysubgroup/my-test-package`nusername=myusername`npassword=mypassword`n"
 | git credential approve

but this didn't help either.

Osyotr commented 6 months ago

vcpkg_from_gitlab does not use git. Use AUTHORIZATION_TOKEN argument or fall back to vcpkg_from_git.

Unfortunately, AUTHORIZATION_TOKEN is not documented yet.

github-actions[bot] commented 5 months ago

This is an automated message. Per our repo policy, stale issues get closed if there has been no activity in the past 28 days. The issue will be automatically closed in 14 days. If you wish to keep this issue open, please add a new comment.

patrikhuber commented 5 months ago

@Osyotr Thank you! I haven't gotten back to trying this again, but that's very useful to know.

I would suggest to keep this issue open and improve the vcpkg_from_gitlab documentation & document AUTHORIZATION_TOKEN?

Osyotr commented 5 months ago

Please open an issue in docs repo. Go to https://learn.microsoft.com/en-us/vcpkg/maintainers/functions/vcpkg_from_gitlab and click "Open a documentation issue".

patrikhuber commented 5 months ago

I've created https://github.com/microsoft/vcpkg-docs/issues/338, I'll close this.