search

microsoft / vcpkg

C++ Library Manager for Windows, Linux, and MacOS
MIT License
22.19k stars 6.16k forks source link

A MinGW package dependency of PCL seems to be missing from mirror servers, causing MingGW to fail to downloadload #23828

Closed andreimr closed 9 months ago

andreimr commented 2 years ago

Package: bzip2:x86-windows Vcpkg version: 2020.06.15-nohash

I'm using WIndows10 build 21h1.

I'm running vcpkg install pcl and am failing to download the package mingw/i686/mingw-w64-i686-pkg-config-0.29.2-1-any.pkg.tar.xz from all mirrors:

-- Downloading https://repo.msys2.org/mingw/i686/mingw-w64-i686-pkg-config-0.29.2-1-any.pkg.tar.xz...
-- Downloading https://repo.msys2.org/mingw/i686/mingw-w64-i686-pkg-config-0.29.2-1-any.pkg.tar.xz... Failed. Status: 22;"HTTP response code said error"
-- Downloading https://www2.futureware.at/~nickoe/msys2-mirror/mingw/i686/mingw-w64-i686-pkg-config-0.29.2-1-any.pkg.tar.xz...
-- Downloading https://www2.futureware.at/~nickoe/msys2-mirror/mingw/i686/mingw-w64-i686-pkg-config-0.29.2-1-any.pkg.tar.xz... Failed. Status: 22;"HTTP response code said error"
-- Downloading https://mirror.yandex.ru/mirrors/msys2/mingw/i686/mingw-w64-i686-pkg-config-0.29.2-1-any.pkg.tar.xz...
-- Downloading https://mirror.yandex.ru/mirrors/msys2/mingw/i686/mingw-w64-i686-pkg-config-0.29.2-1-any.pkg.tar.xz... Failed. Status: 22;"HTTP response code said error"
-- Downloading https://mirrors.tuna.tsinghua.edu.cn/msys2/mingw/i686/mingw-w64-i686-pkg-config-0.29.2-1-any.pkg.tar.xz...
-- Downloading https://mirrors.tuna.tsinghua.edu.cn/msys2/mingw/i686/mingw-w64-i686-pkg-config-0.29.2-1-any.pkg.tar.xz... Failed. Status: 22;"HTTP response code said error"
-- Downloading https://mirrors.ustc.edu.cn/msys2/mingw/i686/mingw-w64-i686-pkg-config-0.29.2-1-any.pkg.tar.xz...
-- Downloading https://mirrors.ustc.edu.cn/msys2/mingw/i686/mingw-w64-i686-pkg-config-0.29.2-1-any.pkg.tar.xz... Failed. Status: 22;"HTTP response code said error"
-- Downloading https://mirror.bit.edu.cn/msys2/mingw/i686/mingw-w64-i686-pkg-config-0.29.2-1-any.pkg.tar.xz...
-- Downloading https://mirror.bit.edu.cn/msys2/mingw/i686/mingw-w64-i686-pkg-config-0.29.2-1-any.pkg.tar.xz... Failed. Status: 6;"Couldn't resolve host name"
-- Downloading https://mirror.selfnet.de/msys2/mingw/i686/mingw-w64-i686-pkg-config-0.29.2-1-any.pkg.tar.xz...
-- Downloading https://mirror.selfnet.de/msys2/mingw/i686/mingw-w64-i686-pkg-config-0.29.2-1-any.pkg.tar.xz... Failed. Status: 35;"SSL connect error"
-- Downloading https://mirrors.sjtug.sjtu.edu.cn/msys2/mingw/i686/mingw-w64-i686-pkg-config-0.29.2-1-any.pkg.tar.xz...
-- Downloading https://mirrors.sjtug.sjtu.edu.cn/msys2/mingw/i686/mingw-w64-i686-pkg-config-0.29.2-1-any.pkg.tar.xz... Failed. Status: 22;"HTTP response code said error"
CMake Error at scripts/cmake/vcpkg_download_distfile.cmake:182 (message):

      Failed to download file.
      If you use a proxy, please set the HTTPS_PROXY and HTTP_PROXY environment
      variables to "https://user:password@your-proxy-ip-address:port/".
      Otherwise, please submit an issue at https://github.com/Microsoft/vcpkg/issues

I took a look at a number of the webpages on the above list, and saw that the package in question is absent from these servers (but other MingGW packages are present)

Can you please check where thes issuereport.txt e packages are supposed to be located, or update the vcpkg dependencies?

etherlore commented 2 years ago

Similar with SDL2: Package: sdl2:x86-windows Vcpkg version: 2021-05-05-e8977e69d9a3fb462d9ad42013d83a7682706659

Error: Failed to download from mirror set:
https://repo.msys2.org/mingw/i686/mingw-w64-i686-libwinpthread-git-8.0.0.5906.c9a21571-1-any.pkg.tar.zst: failed: status code 404
https://www2.futureware.at/~nickoe/msys2-mirror/mingw/i686/mingw-w64-i686-libwinpthread-git-8.0.0.5906.c9a21571-1-any.pkg.tar.zst: failed: status code 404
https://mirror.yandex.ru/mirrors/msys2/mingw/i686/mingw-w64-i686-libwinpthread-git-8.0.0.5906.c9a21571-1-any.pkg.tar.zst: failed: status code 404
https://mirrors.tuna.tsinghua.edu.cn/msys2/mingw/i686/mingw-w64-i686-libwinpthread-git-8.0.0.5906.c9a21571-1-any.pkg.tar.zst: failed: status code 404
https://mirrors.ustc.edu.cn/msys2/mingw/i686/mingw-w64-i686-libwinpthread-git-8.0.0.5906.c9a21571-1-any.pkg.tar.zst: failed: status code 404
https://mirror.bit.edu.cn/msys2/mingw/i686/mingw-w64-i686-libwinpthread-git-8.0.0.5906.c9a21571-1-any.pkg.tar.zst: WinHttpSendRequest() failed: 12007
https://mirror.selfnet.de/msys2/mingw/i686/mingw-w64-i686-libwinpthread-git-8.0.0.5906.c9a21571-1-any.pkg.tar.zst: WinHttpSendRequest() failed: 12175
https://mirrors.sjtug.sjtu.edu.cn/msys2/mingw/i686/mingw-w64-i686-libwinpthread-git-8.0.0.5906.c9a21571-1-any.pkg.tar.zst: failed: status code 404
JackBoosY commented 2 years ago

The msys components pkg-config and libwinpthread-git was updated. Please update your vcpkg using git pull then try again.

Thanks.

andreimr commented 2 years ago

Thank you Jack! Deeply appreciated!

On Tue, Mar 29, 2022 at 3:10 AM Jack·Boos·Yu @.***> wrote:

The msys components pkg-config and libwinpthread-git was updated. Please update your vcpkg using git pull then try again.

Thanks.

— Reply to this email directly, view it on GitHub https://github.com/microsoft/vcpkg/issues/23828#issuecomment-1081496482, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABXLMKL72O2E2YBPZMKVCPTVCKUGXANCNFSM5R4T7OEQ . You are receiving this because you authored the thread.Message ID: @.***>

andreimr commented 2 years ago

Hi again,

It turns out that git pull had no effect. Even a fresh install/build of vcpkg tag 2020-11 had no effect. The issue persists.

On Tue, Mar 29, 2022 at 22:29 Jack·Boos·Yu @.***> wrote:

Closed #23828 https://github.com/microsoft/vcpkg/issues/23828.

— Reply to this email directly, view it on GitHub https://github.com/microsoft/vcpkg/issues/23828#event-6331319901, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABXLMKKQBWK22IIBU4WZRDDVCO4BHANCNFSM5R4T7OEQ . You are receiving this because you authored the thread.Message ID: @.***>

JackBoosY commented 2 years ago

As you can see: https://github.com/microsoft/vcpkg/blob/9d9a6f486cc7da7664117a75d01440db0088634a/scripts/cmake/vcpkg_acquire_msys.cmake#L336-L340

pkg-config was updated to 0.29.2-4 in PR https://github.com/microsoft/vcpkg/pull/15732 last year. Maybe you need to remove folder _VCPKGROOT/downloads/tools/msys2 and check the same file content in your machine.

dg0yt commented 2 years ago

Hi again, It turns out that git pull had no effect. Even a fresh install/build of vcpkg tag 2020-11 had no effect. The issue persists. …

git pull doesn't help if you locked yourself on an old tag. If you don't want to move to the master branch, you must checkout a newer tag.

andreimr commented 2 years ago

It’s not unreasonable to expect to be able to rely on a tag not changing. Why is the tag still available if it can’t be used anymore? Why bother with tags at all?

By the way, I tried very recent tags, and they broke, as well, for the same reason: packages are missing.

Thanks for your attention.

On Wed, Mar 30, 2022 at 22:33 Kai Pastor @.***> wrote:

Hi again, It turns out that git pull had no effect. Even a fresh install/build of vcpkg tag 2020-11 had no effect. The issue persists. … <#m-8785074615803251011>

git pull doesn't help if you locked yourself on an old tag. If you don't want to move to the master branch, you must checkout a newer tag.

— Reply to this email directly, view it on GitHub https://github.com/microsoft/vcpkg/issues/23828#issuecomment-1084008812, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABXLMKKQHBOZLVK6H4DWSL3VCUFGBANCNFSM5R4T7OEQ . You are receiving this because you authored the thread.Message ID: @.***>

dg0yt commented 2 years ago

t’s not unreasonable to expect to be able to rely on a tag not changing. Why is the tag still available if it can’t be used anymore? Why bother with tags at all? By the way, I tried very recent tags, and they broke, as well, for the same reason: packages are missing.

The tags didn't change. The environment changed: The MSYS2 team continuously updates their binary packages. Neither does Microsoft hold copies of the binary MSYS2 packages, nor does the MSYS2 community receive support to keep the outdated binary packages available for download. So it must break after some time.

However, you should be able to use your desired versions of vcpkg ports with the latest HEAD of vcpkg: with manifest mode.

andreimr commented 2 years ago

Thanks for your reply and explanation and identifying my misunderstanding: that I believed incorrectly the tags were connected to package versions.Now I see from the VCPKG documentation how versions are specified. I happened to have inherited a project that depends on VCPKG but did not do that correctly. Thanks for your help. Sorry for being bitchy.

On Thu, Mar 31, 2022 at 10:52 AM Kai Pastor @.***> wrote:

t’s not unreasonable to expect to be able to rely on a tag not changing. Why is the tag still available if it can’t be used anymore? Why bother with tags at all? By the way, I tried very recent tags, and they broke, as well, for the same reason: packages are missing.

The tags didn't change. The environment changed: The MSYS2 team continuously updates their binary packages. Neither does Microsoft hold copies of the binary MSYS2 packages, nor does the MSYS2 community receive support to keep the outdated binary packages available for download. So it must break after some time.

However, you should be able to use your desired versions of vcpkg ports with the latest HEAD of vcpkg: with manifest mode.

— Reply to this email directly, view it on GitHub https://github.com/microsoft/vcpkg/issues/23828#issuecomment-1084697093, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABXLMKNQE6JYHIWDZ42MJ43VCW34RANCNFSM5R4T7OEQ . You are receiving this because you authored the thread.Message ID: @.***>

JackBoosY commented 2 years ago

@dg0yt Maybe we should make a feature to vcpkg_acquire_msys to record and use the newerst components.

dg0yt commented 2 years ago

@andreimr I'm glad we could sort it out.

@JackBoosY I do have an unfinished experiment which uses a more regular approach to using MSYS2 with pacman. Basically I disable update of the database in the installer. It will take a moment to form a draft PR. Key questions:

JackBoosY commented 2 years ago

@dg0yt My initial thought was to create a cache file in the path _VCPKGROOT/tools/msys2 and use the urls and hashs from the cache by default (if validation is required). This way users can always use the latest msys components when rolling back vcpkg versions. For the issues of msys's official frequent update of components, I think we only have two ways:

  1. Print a special message to the failure section to remind the user to open the issue and attach the message.
  2. Use search mode and disable hash detection (because both msys official and mirrors provide hash files to verify data integrity).
JackBoosY commented 2 years ago

Modify vcpkg_acquire_msys (and use a triplet variable), or start with a versioned port (and use a non-default feature)?

The ports always needs the components. We can choose this way if we can dowload the components using msys2 itself.

Keep SHA512-checked downloads of packages (managed and cached by vcpkg) for the default configuration?

I think currently we don't have a plan to provide the storage feature.

Offer asset caching for the GPG-signed online updates (managed by pacman)?

As I said in the previous comment, the official and mirrors provide the checksum file, I think we can use that.

dg0yt commented 2 years ago

Modify vcpkg_acquire_msys (and use a triplet variable), or start with a versioned port (and use a non-default feature)?

The ports always needs the components. We can choose this way if we can dowload the components using msys2 itself.

I don't fully understand. It is clear that there must be a way to download MSYS2 packages. And of course you can download components with MSYS2: Use pacman.

Keep SHA512-checked downloads of packages (managed and cached by vcpkg) for the default configuration?

I think currently we don't have a plan to provide the storage feature.

vcpkg currently explicitly downloads each MSYS2 package, in a way which is covered by asset caching. That's why CI doesn't notice when MSYS2 removes packages. This breaks for users which don't have (access to) a cache hit when the package is no longer available.

Offer asset caching for the GPG-signed online updates (managed by pacman)?

As I said in the previous comment, the official and mirrors provide the checksum file, I think we can use that.

Asset caching also removes load from MSYS2 mirrors. This must remain the default for CI. The question is if it is mandatory when using new versions of MSYS2 packages.

JackBoosY commented 2 years ago

I think we should have a discuss with this situation.

JackBoosY commented 2 years ago
  1. If manifest mode is used, only the port will fall back to the specified version, and the msys2 component is always up-to-date.
  2. If we use the command line mode, using vcpkg release or using git reset to a certain version will cause the msys2 component version to roll back. Since msys2 official and all its mirrors only store the latest version and not the historical version, the download of these components must fail.

If vcpkg_acquire_msys needs to get rid of the constraints of vcpkg version control and always download the latest components, it should automatically query the version number on the official website or mirror address and download it, and then use the matching verification file to verify the correctness of the binary. But this will definitely increase the time-consuming of the whole process significantly.

We can also add the msys2 component cache, and use the version number in the cache to download and verify first. But unfortunately this cache file will not be generated if vcpkg_acquire_msys is not executed in the latest vcpkg.

@ras0219-msft @BillyONeal @strega-nil-ms @vicroms what's your opinion?

dg0yt commented 2 years ago
mprather commented 2 years ago

I like to add some thoughts in addition to those in https://github.com/microsoft/vcpkg/issues/24006#issuecomment-1091784504. If the core of the problem is that vcpkg has taken a dependency on an external binary, then I think vcpkg should maintain a copy of the binaries it needs.

We've been using classic mode for the better part of two years. In that time, we've shipped 6 versions of our product and have used exactly 2 vcpkg commits to support those releases. We support clients that utilize air-gapped systems, which means we have to verify every version of incoming code. There is a high burden to upgrading code, whether we create it or rely on a system such as vcpkg. We can't just upgrade vcpkg on a whim. It has to be planned and costed out.

We have been bitten by a few vcpkg oddities over time and those oddities also impact what we can and cannot consider as viable upgrade candidates. For example, our latest upgrade placed us on an Oct 29, 2021 commit because anything newer suddenly resulted in platform bugs with vcpkg that blocked any use on our target and, if I remember correctly, anything older than a couple of weeks resulted in dependent library failures. Finding the right set of working pieces takes effort. To make matters more interesting, it's possible that vcpkg will just stop working months after we've invested in testing, validation, documentation, and even shipped two releases and ... poof... we hit a mysterious wall?

So, now we have the unfortunate luck that we can't recreate the environment that is only a few months old. I get that msys2 might be solely concerned with their primary use case and hence drive their storage as they see fit. If they won't/don't care about the fact others are using them as dependencies (ala vcpkg) to establish other dependencies, then it's incumbent on tools like vcpkg to mitigate the loss of those dependencies by improving the algorithm, saving the dependencies, etc.

We never considered that vcpkg would just lose functionality one day after-the-fact b/c it was never promoted that way. It's been "find a working commit and you're good". Not everyone can update and live on the edge. The message to just upgrade to latest is a non-starter for a lot of folks. Telling folks to use manifest mode also doesn't really work if we are still tied to using a specific version of vcpkg (manifest mode has other issues that preclude use for us as well). If vcpkg wants to be edge-centric, then messaging has to change so that your customers will know what to expect. Otherwise, please figure out the infrastructure, code changes, etc. to avoid these type of tooling breaks.

ras0219-msft commented 2 years ago

@mprather Thanks for describing your concrete scenario, this really helps.

We've introduced a feature that's directly targeted at your situation to ensure you never hit this mysterious wall: asset caching[1]. This enables you to locally cache and preserve everything[2] that vcpkg needs to operate so you can be completely air-gapped and reproducible over time. If you're interested in sending us a mail (vcpkg@microsoft.com) with more details about what specific issues you've hit, we'd appreciate it.

This need for air-gapped operation and caching is one of the motivations to move away from calling MSYS's pacman in the first place -- we found that pacman was often a source of instability in our CI and it adds significant additional burden for "offline/airgapped" users. I don't think we should regress back to that for default operation, though I could perhaps see some optional "please get me the latest packages from msys, I know that this is unstable and might break my build".

I think the long term solution to this problem either needs to be:

  1. Someone provides stable rehosting of the MSYS packages. It's challenging for us (Microsoft) to do this because it introduces significant legal challenges. A single package with a license that we can't rehost defeats this approach, so it's not clear up-front that relying entirely on us to provide the mirror is a viable solution.

    • Fortunately, because we do pin the SHA512 hashes of the packages, we don't need to put significant trust into such a mirror. This could even be as simple as someone creating a GitHub repo and attaching the handful of packages that vcpkg uses to a release.

  2. vcpkg learns to build any needed msys pieces from source as host packages (or we find alternatives). We've started making this transition in some places (pkgconf[3]), but some ports will always remain a challenge. Ffmpeg requires MSYS for every documented build procedure on Windows[4].

[1] https://github.com/microsoft/vcpkg/blob/master/docs/users/assetcaching.md [2] Asset caching covers everything we can download over http/https/ftp -- there are a very small number of packages which require us to run git clone that are missed by asset caching, but there are additional steps you can take if you need those packages. [3] https://github.com/microsoft/vcpkg/blob/master/ports/pkgconf/ [4] https://trac.ffmpeg.org/wiki/CompilationGuide

dg0yt commented 2 years ago

In addition to asset caching, it is also possible to manually transfer previous downloads directly via the downloads folder. The key barrier remains to not possess the downloads for other past revisions of vcpkg.

My playground is currently frozen at the step where I need to decide how to couple pacman downloads with vcpkg's asset caching. It is doable, but you have to define and freeze the versions and the SHA512 revisions.

In any case, I want that a --only-downloads run captures all required downloads.

github-actions[bot] commented 10 months ago

This is an automated message. Per our repo policy, stale issues get closed if there has been no activity in the past 28 days. The issue will be automatically closed in 14 days. If you wish to keep this issue open, please add a new comment.