search

microsoft / vcpkg

C++ Library Manager for Windows, Linux, and MacOS
MIT License
22.87k stars 6.31k forks source link

Vcpkg fails to install anything - File does not have the expected hash #4663

Closed parsa closed 3 years ago

parsa commented 5 years ago

Steps to reproduce:

Output:

A suitable version of 7zip was not found (required v18.1.0). Downloading portable 7zip v18.1.0...
Downloading 7zip...
  https://www.nuget.org/api/v2/package/7-Zip.CommandLine/18.1.0 -> C:\Repos\vcpkg\downloads\7-zip.commandline.18.1.0.nupkg
File does not have the expected hash:
             url : [ https://www.nuget.org/api/v2/package/7-Zip.CommandLine/18.1.0 ]
       File path : [ C:\Repos\vcpkg\downloads\7-zip.commandline.18.1.0.nupkg.part ]
   Expected hash : [ 8c75314102e68d2b2347d592f8e3eb05812e1ebb525decbac472231633753f1d4ca31c8e6881a36144a8da26b2571305b3ae3f4e2b85fc4a290aeda63d1a13b8 ]
     Actual hash : [ a9dfaaafd15d98a2ac83682867ec5766720acf6e99d40d1a00d480692752603bf3f3742623f0ea85647a92374df405f331afd6021c5cf36af43ee8db198129c0 ]

Detail:

Full Output:

PS C:\Repos\vcpkg> vcpkg install --triplet x64-windows boost
The following packages will be built and installed:
    boost[core]:x64-windows
  * boost-accumulators[core]:x64-windows
  * boost-algorithm[core]:x64-windows
  * boost-align[core]:x64-windows
  * boost-any[core]:x64-windows
  * boost-array[core]:x64-windows
  * boost-asio[core]:x64-windows
  * boost-assert[core]:x64-windows
  * boost-assign[core]:x64-windows
  * boost-atomic[core]:x64-windows
  * boost-beast[core]:x64-windows
  * boost-bimap[core]:x64-windows
  * boost-bind[core]:x64-windows
  * boost-build[core]:x64-windows
  * boost-callable-traits[core]:x64-windows
  * boost-chrono[core]:x64-windows
  * boost-circular-buffer[core]:x64-windows
  * boost-compatibility[core]:x64-windows
  * boost-compute[core]:x64-windows
  * boost-concept-check[core]:x64-windows
  * boost-config[core]:x64-windows
  * boost-container[core]:x64-windows
  * boost-container-hash[core]:x64-windows
  * boost-context[core]:x64-windows
  * boost-contract[core]:x64-windows
  * boost-conversion[core]:x64-windows
  * boost-convert[core]:x64-windows
  * boost-core[core]:x64-windows
  * boost-coroutine[core]:x64-windows
  * boost-coroutine2[core]:x64-windows
  * boost-crc[core]:x64-windows
  * boost-date-time[core]:x64-windows
  * boost-detail[core]:x64-windows
  * boost-disjoint-sets[core]:x64-windows
  * boost-dll[core]:x64-windows
  * boost-dynamic-bitset[core]:x64-windows
  * boost-endian[core]:x64-windows
  * boost-exception[core]:x64-windows
  * boost-fiber[core]:x64-windows
  * boost-filesystem[core]:x64-windows
  * boost-flyweight[core]:x64-windows
  * boost-foreach[core]:x64-windows
  * boost-format[core]:x64-windows
  * boost-function[core]:x64-windows
  * boost-function-types[core]:x64-windows
  * boost-functional[core]:x64-windows
  * boost-fusion[core]:x64-windows
  * boost-geometry[core]:x64-windows
  * boost-gil[core]:x64-windows
  * boost-graph[core]:x64-windows
  * boost-graph-parallel[core]:x64-windows
  * boost-hana[core]:x64-windows
  * boost-hana-msvc[core]:x64-windows
  * boost-heap[core]:x64-windows
  * boost-hof[core]:x64-windows
  * boost-icl[core]:x64-windows
  * boost-integer[core]:x64-windows
  * boost-interprocess[core]:x64-windows
  * boost-interval[core]:x64-windows
  * boost-intrusive[core]:x64-windows
  * boost-io[core]:x64-windows
  * boost-iostreams[core]:x64-windows
  * boost-iterator[core]:x64-windows
  * boost-lambda[core]:x64-windows
  * boost-lexical-cast[core]:x64-windows
  * boost-local-function[core]:x64-windows
  * boost-locale[core]:x64-windows
  * boost-lockfree[core]:x64-windows
  * boost-log[core]:x64-windows
  * boost-logic[core]:x64-windows
  * boost-math[core]:x64-windows
  * boost-metaparse[core]:x64-windows
  * boost-modular-build-helper[core]:x64-windows
  * boost-move[core]:x64-windows
  * boost-mp11[core]:x64-windows
  * boost-mpl[core]:x64-windows
  * boost-msm[core]:x64-windows
  * boost-multi-array[core]:x64-windows
  * boost-multi-index[core]:x64-windows
  * boost-multiprecision[core]:x64-windows
  * boost-numeric-conversion[core]:x64-windows
  * boost-odeint[core]:x64-windows
  * boost-optional[core]:x64-windows
  * boost-parameter[core]:x64-windows
  * boost-phoenix[core]:x64-windows
  * boost-poly-collection[core]:x64-windows
  * boost-polygon[core]:x64-windows
  * boost-pool[core]:x64-windows
  * boost-predef[core]:x64-windows
  * boost-preprocessor[core]:x64-windows
  * boost-process[core]:x64-windows
  * boost-program-options[core]:x64-windows
  * boost-property-map[core]:x64-windows
  * boost-property-tree[core]:x64-windows
  * boost-proto[core]:x64-windows
  * boost-ptr-container[core]:x64-windows
  * boost-python[core]:x64-windows
  * boost-qvm[core]:x64-windows
  * boost-random[core]:x64-windows
  * boost-range[core]:x64-windows
  * boost-ratio[core]:x64-windows
  * boost-rational[core]:x64-windows
  * boost-regex[core]:x64-windows
  * boost-scope-exit[core]:x64-windows
  * boost-serialization[core]:x64-windows
  * boost-signals[core]:x64-windows
  * boost-signals2[core]:x64-windows
  * boost-smart-ptr[core]:x64-windows
  * boost-sort[core]:x64-windows
  * boost-spirit[core]:x64-windows
  * boost-stacktrace[core]:x64-windows
  * boost-statechart[core]:x64-windows
  * boost-static-assert[core]:x64-windows
  * boost-system[core]:x64-windows
  * boost-test[core]:x64-windows
  * boost-thread[core]:x64-windows
  * boost-throw-exception[core]:x64-windows
  * boost-timer[core]:x64-windows
  * boost-tokenizer[core]:x64-windows
  * boost-tti[core]:x64-windows
  * boost-tuple[core]:x64-windows
  * boost-type-erasure[core]:x64-windows
  * boost-type-index[core]:x64-windows
  * boost-type-traits[core]:x64-windows
  * boost-typeof[core]:x64-windows
  * boost-ublas[core]:x64-windows
  * boost-units[core]:x64-windows
  * boost-unordered[core]:x64-windows
  * boost-utility[core]:x64-windows
  * boost-uuid[core]:x64-windows
  * boost-variant[core]:x64-windows
  * boost-vcpkg-helpers[core]:x64-windows
  * boost-vmd[core]:x64-windows
  * boost-wave[core]:x64-windows
  * boost-winapi[core]:x64-windows
  * boost-xpressive[core]:x64-windows
  * boost-yap[core]:x64-windows
  * bzip2[core]:x64-windows
  * liblzma[core]:x64-windows
  * openssl[core]:x64-windows
  * openssl-windows[core]:x64-windows
  * python3[core]:x64-windows
  * zlib[core]:x64-windows
Additional packages (*) will be modified to complete this operation.
Starting package 1/143: boost-vcpkg-helpers:x64-windows
Building package boost-vcpkg-helpers[core]:x64-windows...
A suitable version of cmake was not found (required v3.11.4). Downloading portable cmake v3.11.4...
Extracting cmake...
A suitable version of 7zip was not found (required v18.1.0). Downloading portable 7zip v18.1.0...
Downloading 7zip...
  https://www.nuget.org/api/v2/package/7-Zip.CommandLine/18.1.0 -> C:\Repos\vcpkg\downloads\7-zip.commandline.18.1.0.nupkg
File does not have the expected hash:
             url : [ https://www.nuget.org/api/v2/package/7-Zip.CommandLine/18.1.0 ]
       File path : [ C:\Repos\vcpkg\downloads\7-zip.commandline.18.1.0.nupkg.part ]
   Expected hash : [ 8c75314102e68d2b2347d592f8e3eb05812e1ebb525decbac472231633753f1d4ca31c8e6881a36144a8da26b2571305b3ae3f4e2b85fc4a290aeda63d1a13b8 ]
     Actual hash : [ a9dfaaafd15d98a2ac83682867ec5766720acf6e99d40d1a00d480692752603bf3f3742623f0ea85647a92374df405f331afd6021c5cf36af43ee8db198129c0 ]
Rastaban commented 5 years ago

That's annoyhing, I just tried the above command on a clean machine and was unable to reproduce the issue.

 A suitable version of 7zip was not found (required v18.1.0). Downloading portable 7zip v18.1.0...
Downloading 7zip...
  https://www.nuget.org/api/v2/package/7-Zip.CommandLine/18.1.0 -> E:\vcpkg\vcpkg\downloads\7-zip.commandline.18.1.0.nupkg
Downloading 7zip... done.
Extracting 7zip...

Does it still repro? I ask only because it could have been an intermittent issue in the nuget service that is now resolved so it worked for me but not for you earlier.

szx917 commented 5 years ago

same here OS: win10 x64

>vcpkg install curl
The following packages will be built and installed:
    curl[core,ssl]:x86-windows
  * openssl[core]:x86-windows
  * openssl-windows[core]:x86-windows
  * zlib[core]:x86-windows
Additional packages (*) will be modified to complete this operation.
Starting package 1/4: zlib:x86-windows
Building package zlib[core]:x86-windows...
A suitable version of cmake was not found (required v3.11.4). Downloading portable cmake v3.11.4...
Extracting cmake...
A suitable version of 7zip was not found (required v18.1.0). Downloading portable 7zip v18.1.0...
Downloading 7zip...
  https://www.nuget.org/api/v2/package/7-Zip.CommandLine/18.1.0 -> C:\Users\Jane Wang\vcpkg\downloads\7-zip.commandline.18.1.0.nupkg
File does not have the expected hash:
             url : [ https://www.nuget.org/api/v2/package/7-Zip.CommandLine/18.1.0 ]
       File path : [ C:\Users\****\vcpkg\downloads\7-zip.commandline.18.1.0.nupkg.part ]
   Expected hash : [ 8c75314102e68d2b2347d592f8e3eb05812e1ebb525decbac472231633753f1d4ca31c8e6881a36144a8da26b2571305b3ae3f4e2b85fc4a290aeda63d1a13b8 ]
     Actual hash : [ a9dfaaafd15d98a2ac83682867ec5766720acf6e99d40d1a00d480692752603bf3f3742623f0ea85647a92374df405f331afd6021c5cf36af43ee8db198129c0 ]
anders-wind commented 5 years ago

I'm experiencing the same issue. Tried vcpkg commit: 9ad166279c913cf0becc56865d05ed5806fad8aa Fails at 7-Zip.Commandline/18.1.0 extraction.

hjabird commented 5 years ago

This can be temporarily and foolishly 'fixed' by replacing the hash value in vcpkg/scripts/vcpkgTools (tool name="7zip" os="windows") with the actual hash. I'm pretty sure that it such an act ranks highly in the list of security sins.

ijean commented 5 years ago

I can confirm, same problem, on a clean Windows 10 x64 machine.

@hjabird Instead of changing the hash, probably a better idea is to bump the version to 18.05 which is the latest stable version and solves some Windows 10 bugs.

parsa commented 5 years ago

@Rastaban tried again on a clean VM and it does still reproduce. Same exact hashes. Also I my machine I just changed the hash in scripts/vcpkgTools.xml and the issue went away and Boost installed just fine.

derekseiple commented 5 years ago

I can confirm the same failure as well.

To work around this I copied the original 7-zip.commandline.18.1.0.nupkg (the one with the correct hash) that I had on hand from another install location into the vcpkg/downloads directory. It was then able to install packages as normal.

This doesn't help though if you can't get your hands on the right file...

ras0219-msft commented 5 years ago

Thank you everyone for quickly jumping in and reporting this!

I've just pushed 068032bc to master, which downgrades the version of 7-zip to a previous version while we work on some internal changes to fix this more permanently for 18.1.0.

The root cause of the issue is that NuGet.org decided to modify the existing published nupkg files and inject signature files into them, which changes their hash. Our system to protect users from malicious files detected this change and correctly refused to use the un-validated executable.

Unfortunately, it isn't as simple as changing the expected hash, because existing users will still have the older file downloaded. We also can't just rename the file, because nuget.exe demands that the file be named in a very specific way.

As mentioned above, we're working on fixing this properly as a high priority, but 068032bc will fix the issue for now.

Thanks again!

jherico commented 5 years ago

Since applying the hotfix I'm seeing a new problem

Expected C:\Users\bdavi\Git\vcpkg\downloads\tools\cmake-3.11.4-windows\cmake-3.11.4-win32-x86\bin\cmake.exe to exist after fetching

This didn't make sense to me because the file it's looking for exists. I dug into the code and discovered that fs::stdfs::exists was returning false for the executable.

I created a brand new local C++ project and verified the same result, not just for the executable, but for all the files in the archive. However, if I wiped the folder and then re-extracted the files using WinRAR, the files suddenly visible as far as fs::stdfs::exists was concerned.

Something about the 16 version of 7zip is setting file attributes so that even though they're visible in the command line, can be executed, and visible in explorer, the C++ library's filesystem exists function thinks they're not present. This actually feels like a bug in the C++ library, but it's blocking my use of vcpkg.

ras0219-msft commented 5 years ago

Thanks for the further investigation.

I was able to confirm some strange behavior (though not this particular issue) when using the old 7zip as well. I've pushed a different workaround (273b8ce3d0d3533f3b959a7ecf4b0aa1eef22cab) that's a bit more permanent by mapping the new hash in the tool back to the old hash.

The reason to map the new hash to the old hash is to improve compatibility for existing users which might git pull but forget to ./bootstrap-vcpkg.bat -- they'll see everything keep working as before, because they already have the old version downloaded. For new users, they will definitely be bootstrapping, so the new file's hash will get (hackily) mapped to the old hash and the check will pass.

A proper fix is still desirable, but this particular duct tape should hold for longer since it's using the same (latest available) version of 7zip as before.

Farwaykorse commented 5 years ago

This issue still appears for anyone who has not ran .\bootstrap-vcpkg.bat since the patch. It is not something I'd do when there is no update warning.

Why not increase the vcpkg version?

degski commented 5 years ago

@ras0219-msft @Farwaykorse Yes, the version should be increased so at least a message to run .\bootstrap-vcpkg.bat pops up.

@ras0219-msft The nuget versioning is wacky, 7zip versioning is format YY,MM, i.e. 18.01, and not number.number.

Adraesh commented 5 years ago

Hello, I have the same issue, I am trying to run .\bootstrap-vcpkg.bat but it failed due to toolsrc/ folder not existing.

I got vcpkg 1.50 through VS 2017 Nuget Package Manager and by this way you don't get the toolsrc/ folder...

Thank you in advance.

aybassiouny commented 5 years ago

This seems to be an issue especially when going back in time to get earlier versions of libraries.

rhvarrier commented 4 years ago

This issue is still present in 2020.01 version

ghost commented 3 years ago

For me it was solved by running the command with admin privileges...

malidiab commented 3 years ago

updating vcpkg to 2020.06.15- fixes the problem to me

Nikmeh922 commented 3 years ago

I can confirm the same failure as well.

To work around this I copied the original 7-zip.commandline.18.1.0.nupkg (the one with the correct hash) that I had on hand from another install location into the vcpkg/downloads directory. It was then able to install packages as normal.

This doesn't help though if you can't get your hands on the right file...

Thank you. This helped a lot!!! @derekseiple

parsa commented 3 years ago

Thank you everyone for quickly jumping in and reporting this!

I've just pushed 068032b to master, which downgrades the version of 7-zip to a previous version while we work on some internal changes to fix this more permanently for 18.1.0.

The root cause of the issue is that NuGet.org decided to modify the existing published nupkg files and inject signature files into them, which changes their hash. Our system to protect users from malicious files detected this change and correctly refused to use the un-validated executable.

Unfortunately, it isn't as simple as changing the expected hash, because existing users will still have the older file downloaded. We also can't just rename the file, because nuget.exe demands that the file be named in a very specific way.

As mentioned above, we're working on fixing this properly as a high priority, but 068032b will fix the issue for now.

Thanks again!

This issue has been long solved for me, and I don't think the follow-ups describe the same issue. If my opinion as the person who opened this issue counts, this issue can be closed.

PhoebeHui commented 3 years ago

I agree, thank you parsa for your response! closing this issue now.

fly2sky2018 commented 2 years ago

I solve the preblem 1)download this files https://raw.githubusercontent.com/boostorg/boost/boost-1.75.0/LICENSE_1_0.txt https://raw.githubusercontent.com/boostorg/boost/boost-1.75.0/boostcpp.jam 2)skip the hash test add skip code at the vcpkg_download_distfile.camke