search

microsoft / vcpkg

C++ Library Manager for Windows, Linux, and MacOS
MIT License
23.33k stars 6.45k forks source link

[libogg] server down #41802

Open dsvensson opened 4 weeks ago

dsvensson commented 4 weeks ago

Describe the bug Sometimes servers are down. Today xiph.org is not reachable. Wouldn't require much storage for Microsoft to mirror all vcpkg source tarballs and likely would be infinitely more reliable. If Azure Blob Storage is anything like AWS S3, it would probably be hard to find a natural disaster that takes it down.

Environment Not applicable

To Reproduce

  1. Depend on ogg.

Expected behavior Ogg source is downloaded and everything works fine.

Failure logs

  -- Downloading https://gitlab.xiph.org/xiph/ogg/-/archive/v1.3.5/ogg-v1.3.5.tar.gz -> xiph-ogg-v1.3.5.tar.gz...
  [DEBUG] To include the environment variables in debug output, pass --debug-env
  [DEBUG] Trying to load bundleconfig from /Users/runner/work/ezquake-source/ezquake-source/vcpkg/vcpkg-bundle.json
  [DEBUG] Failed to open: /Users/runner/work/ezquake-source/ezquake-source/vcpkg/vcpkg-bundle.json
  [DEBUG] Bundle config: readonly=false, usegitregistry=false, embeddedsha=nullopt, deployment=Git, vsversion=nullopt
  [DEBUG] Metrics enabled.
  [DEBUG] Feature flag 'binarycaching' unset
  [DEBUG] Feature flag 'compilertracking' unset
  [DEBUG] Feature flag 'registries' unset
  [DEBUG] Feature flag 'versions' unset
  [DEBUG] Feature flag 'dependencygraph' unset
  [DEBUG] Detected CI environment: GitHub_Actions
  [DEBUG] 1000: execute_process(curl --fail -L https://gitlab.xiph.org/xiph/ogg/-/archive/v1.3.5/ogg-v1.3.5.tar.gz --create-dirs --output /Users/runner/work/ezquake-source/ezquake-source/vcpkg/downloads/xiph-ogg-v1.3.5.tar.gz.5422.part)
  [DEBUG] 1000: cmd_execute_and_stream_data() returned 7 after  5037453 us
  error: Missing xiph-ogg-v1.3.5.tar.gz and downloads are blocked by x-block-origin.
  error: https://gitlab.xiph.org/xiph/ogg/-/archive/v1.3.5/ogg-v1.3.5.tar.gz: curl failed to download with exit code 7
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed

  curl: (7) Failed to connect to gitlab.xiph.org port 443 after 5028 ms: Couldn't connect to server

Additional context N/A

pierr3 commented 4 weeks ago

Concur, this is not the first time this occurs with this specific repo too.

interloper98 commented 4 weeks ago

Can't single-point-of-failures fallback to a try from a list of mirrors?

For example, this same ogg-v1.3.5.tar.gz file is available on xiph's GitHub repo: https://github.com/xiph/ogg/releases

(I wouldn't doubt if the same tarball exists on a whole lot more servers, too)

dg0yt commented 4 weeks ago

For example, this same ogg-v1.3.5.tar.gz file is available on xiph's GitHub repo: https://github.com/xiph/ogg/releases

The port uses on-the-fly tarballs generated by gitlab, not the mirrored release tarballs. Different SHA512.

If you need it in CI, setup proper source asset caching. If you need it locally, there is local asset caching. Otherwise wait for remote maintenance to complete.

interloper98 commented 4 weeks ago

I appreciate the tips, @dg0yt - and to confirm, yes, our project is using extensive CI caching (at almost every level, from what I can see).

The problem in this case happened after rolling our baseline forward - so we didn't have a source cache nor the resulting binary cache to pull from. We only pull and build dependency sources one time, and they're cached fully going forward.

Caching is a different problem and concept than mirroring.

Robust system design permits and benefits from cache (on the user-side), but does not require it. However, robust repository design requires the ample use of mirroring (on the provider side).

It sounds like the vcpkg team has declared that "ports that use on-the-fly tarballs shall be single points of failure!" (I hope I'm wrong!)

In any case.. hopefully maintainers in the future reject these types of ports and require concrete release tarballs such that mirrors can be leveraged for their sources.

dg0yt commented 4 weeks ago

In any case.. hopefully maintainers in the future reject these types of ports and require concrete release tarballs such that mirrors can be leveraged for their sources.

You might need to raise this more explicitly if it is important to you. What I see is that vcpkg encourages the use of the maintainer functions which rely on the on-the-fly tarballs (vcpkg_from_github, vcpkg_from_gitlab). This has pros and cons. It isolated vcpkg from the tainted liblzma tarballs. It works well for github (unless they unexpectedly change the operation), but might be less robust for "small" gitlab installations.