Chocolatey is using checksum.exe which is slow and can't cope with larger than 2GB files

flcdrg commented 3 months ago

Checklist

[X] I confirm there are no unresolved issues reported on the Chocolatey Status page.
[X] I have verified this is the correct repository for opening this issue.
[X] I have verified no other issues exist related to my problem.
[X] I have verified this is not an issue for a specific package.
[X] I have verified this issue is not security related.
[X] I confirm I am using official, and not unofficial, or modified, Chocolatey products.

What You Are Seeing?

Package fails to validate or install because download is larger than 2GB

What is Expected?

Chocolatey should be able to cope with large files for calculating checksum/filehash

How Did You Get This To Happen?

Tried to install package with downloaded file larger than 2GB eg- https://community.chocolatey.org/packages/resharper-platform/241.0.20240528.180551

System Details

As per verifier environment

Installed Packages

n/a

Output Log

2024-06-10 01:03:39,185 4952 [INFO ] - Download of JetBrains.dotUltimate.2024.1.3.exe (2.55 GB) completed.
2024-06-10 01:03:42,544 4952 [DEBUG] - No runtime virus checking built into FOSS Chocolatey. Check out Pro/Business - https://chocolatey.org/compare
2024-06-10 01:03:42,637 4952 [DEBUG] - Verifying package provided checksum of '5342ede24c95a30261c0d43402c5cd3687d4b208233f028d7cb1ef7fe99a5a25' for 'C:\ProgramData\chocolatey\lib\resharper-platform\JetBrains.dotUltimate.2024.1.3.exe'.
2024-06-10 01:03:42,793 4952 [DEBUG] - Running Get-ChecksumValid -file 'C:\ProgramData\chocolatey\lib\resharper-platform\JetBrains.dotUltimate.2024.1.3.exe' -checksum '5342ede24c95a30261c0d43402c5cd3687d4b208233f028d7cb1ef7fe99a5a25' -checksumType 'sha256' -originalUrl 'https://download.jetbrains.com/resharper/dotUltimate.2024.1.3/JetBrains.dotUltimate.2024.1.3.exe' 
2024-06-10 01:03:43,091 4952 [DEBUG] - checksum.exe found at 'C:\ProgramData\chocolatey\helpers\..\tools\checksum.exe'
2024-06-10 01:03:43,123 4952 [DEBUG] - Executing command ['C:\ProgramData\chocolatey\helpers\..\tools\checksum.exe' -c="5342ede24c95a30261c0d43402c5cd3687d4b208233f028d7cb1ef7fe99a5a25" -t="sha256" -f="C:\ProgramData\chocolatey\lib\resharper-platform\JetBrains.dotUltimate.2024.1.3.exe"]
2024-06-10 01:40:08,840 4952 [WARN ] - Chocolatey timed out waiting for the command to finish. The timeout
 specified (or the default value) was '2700' seconds. Perhaps try a
 higher `--execution-timeout`? See `choco -h` for details.
2024-06-10 01:40:08,888 4952 [DEBUG] - Calling command ['"C:\Windows\System32\shutdown.exe" /a']
2024-06-10 01:40:09,955 4952 [DEBUG] - Command ['"C:\Windows\System32\shutdown.exe" /a'] exited with '1116'
2024-06-10 01:40:10,683 4952 [DEBUG] - Capturing package files in 'C:\ProgramData\chocolatey\lib\resharper-platform'
2024-06-10 01:40:10,745 4952 [WARN ] - Error computing hash for 'C:\ProgramData\chocolatey\lib\resharper-platform\JetBrains.dotUltimate.2024.1.3.exe'
 Hash will be special code for locked file or file too big instead.
 Captured error:
  The file is too long. This operation is currently limited to supporting files less than 2 gigabytes in size.



### Additional Context

_No response_

TheCakeIsNaOH commented 3 months ago

checksum.exe can hash files that are larger than 2gb, see https://github.com/chocolatey/choco/issues

The warning about Error computing hash for is from the file hashing built into Chocolatey CLI itself: https://github.com/chocolatey/choco/issues/747 I have a PR open to fix it: https://github.com/chocolatey/choco/pull/2714

The hashing taking way longer than alternative hashing methods would be an issue (although it probably would need to be benchmarked), which could be related tochecksum.exe (or some aspect of how it is called).

flcdrg commented 3 months ago

Might be worth looking at the Get-FileHash source at https://github.com/PowerShell/PowerShell/blob/master/src/Microsoft.PowerShell.Commands.Utility/commands/utility/GetHash.cs to see if there's tweaks you can adopt here.

chocolatey / checksum