Support streaming hashes

[ruckc]

What You Are Seeing?

Installing large packages fail when file size is larger than 2GB.

When looking at CryptoHashProvider, it appears it reads the entire file into memory and then compute the hash, and then throw the byte[] away.

It would make sense to change the IHashAlgorithm/HashAlgorithm to handle IO.Stream.

var hash = _hashAlgorithm.ComputeHash(_fileSystem.read_file_bytes(filePath));

What is Expected?

The ability to install large packages.

How Did You Get This To Happen? (Steps to Reproduce)

Build a large nupkg and install via chocolatey.

Output Log

Error computing hash for 'C:\ProgramData\chocolatey\lib\BigPackage\BigPackage.nupkg'
 Hash will be special code for locked file or file too big instead.
 Captured error:
  The file is too long. This operation is currently limited to supporting files less than 2 gigabytes in size.

[ferventcoder]

Please submit the gist of the chocolatey log output. Then we can look from there.

[ferventcoder]

Trying to determine where the actual issue occurs - more explanation: We only warn if we can't compute a hash. If you are seeing an error that is something else entirely.

[ruckc]

Roger... looking further in the long and we found the reason the install failed. It would be nice if streaming hashes were supported though...

[ferventcoder]

Was it a reason we could correct or is it something else?

[MichaelGerhart]

@ferventcoder was there ever any progress made on this? I am also having trouble deploying a package that is larger than 2GB with the same hash error, see the attached log file. log.txt

[ferventcoder]

Error or Warning?

[MichaelGerhart]

Its listed as a warning, but causes the package installation to fail.

[ferventcoder]

Looks like warning

 Hash will be special code for locked file or file too big instead.
 Captured error:
  The file is too long. This operation is currently limited to supporting files less than 2 gigabytes in size.
2017-07-13 14:18:27,536 5380 [WARN ] - Error computing hash for 'C:\ProgramData\chocolatey\lib\MSSQL2014STD\tools\Sources\SQLServer2014x64Standard\STDISO.ISO'
 Hash will be special code for locked file or file too big instead.
 Captured error:
  The file is too long. This operation is currently limited to supporting files less than 2 gigabytes in size.sh

And then I see the package fails. I see the package fails afterwards.

Can you open up the chocolatey.log instead of the chocolatey.summary.log and include that output instead? We'd need all of that to determine whether that is really causing the issue or not. Thanks!

[ferventcoder]

Debug output is going to be much more helpful.

[KurtTheBerner]

I have exactly the same problem with a package larger than 2GB. Gist Link with Debug Output: https://gist.github.com/KurtTheBerner/6efdf08cbcac24a1f4ee2d28323383c0

Console Output:

Chocolatey v0.10.7
Installing the following packages:
MyBigPackage
By installing you accept licenses for the packages.

mybigpackage v1.2.3
mybigpackage package files install completed. Performing other installation steps.
Error computing hash for 'C:\ProgramData\chocolatey\lib\mybigpackage\mybigpackage.nupkg
 Hash will be special code for locked file or file too big instead.
 Captured error:
  The file is too long. This operation is currently limited to supporting files less than 2 gigabytes in size.
 The install of mybigpackage was successful.
  Software install location not explicitly set, could be in package or
  default install location if installer.
Chocolatey installed 1/1 packages.
 See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).

Although chocolatey says that The install of mybigpackage was successful, the package was not installed. For us this is an important point for the use of Chocolatey, we would be glad if the problem were solved soon.

[KurtTheBerner]

@ferventcoder any news about this issue?

[ferventcoder]

@KurtTheBerner I just reviewed your log, and the installation was successful for the package, but maybe not the software you were attempting to install. Perhaps there was some issue in the install-cmd for the software, but it did not report the error correctly back to to the powershell script which did not fail.

Looking at https://gist.githubusercontent.com/KurtTheBerner/6efdf08cbcac24a1f4ee2d28323383c0/raw/2522ba1cadfe8f6c11afd1f0e731661d21411ee1/chocolatey.log, you will see it reports the package install as successful, and it does see all of the files when it goes through and runs the checksums for the files (after the powerShell script runs). Finding the files and reporting success means the package installation was successful.

I would spend a closer look at the actual thing the installer is calling to see how it is not installing the software itself.

Here is the script you are running:

<#
==============================================================================
 FILE     : $chocolateyInstall.ps1 $
 PROJECT  : Install Script Library
 COPYRIGHT: Copyright (c) myCompany
 PURPOSE  : 
 COMMENT  : 
------------------------------------------------------------------------------
 HISTORY (latest entry first):
 2012-03-22  Version 1.0
==============================================================================
#>

$ErrorActionPreference = 'Stop'

$packageName = 'My big Package'
$toolsDir    = "$(Split-Path -parent $MyInvocation.MyCommand.Definition)"
$logDir      = $env:ALLUSERSPROFILE + "\MyCompany\InstallLogs\"
$logFileName = $logDir + "$packageName.log"

& $toolsDir\SQL-Install.cmd | Out-Null

Copy-Item -Path "$env:ProgramFiles\Microsoft SQL Server\120\Setup Bootstrap\Log\summary.txt" -Destination "$logDir\$packageName.log"

As you can see from the log, it finishes running that without any reported issues:

2017-08-11 14:36:26,081 7020 [DEBUG] - Built-in PowerShell host called with ['[System.Threading.Thread]::CurrentThread.CurrentCulture = '';[System.Threading.Thread]::CurrentThread.CurrentUICulture = ''; & import-module -name 'C:\ProgramData\chocolatey\helpers\chocolateyInstaller.psm1'; & 'C:\ProgramData\chocolatey\helpers\chocolateyScriptRunner.ps1' -packageScript 'C:\ProgramData\chocolatey\lib\mybigpackage\tools\chocolateyInstall.ps1' -installArguments '' -packageParameters '''] exited with '0'.

Since you are copying the log from the software install, I would take a closer look at that to see what errors it ran into. That's where you are going to likely find the issue. HTH

[ferventcoder]

Calling a .cmd file may not produce an exit code like you want, but you can always check LastExitCode to see if it errored on install.

[KurtTheBerner]

@ferventcoder I found the issue, the installation was cancelled and the ExitCode was not reported correctly. I fixed it and now the Package has been successfully installed, but still the following Warning occurs:

Error computing hash for 'C:\ProgramData\chocolatey\lib\mybigpackage\mybigpackage.nupkg'
 Hash will be special code for locked file or file too big instead.
 Captured error:
  The file is too long. This operation is currently limited to supporting files less than 2 gigabytes in size.

This warning has irritated me and I thought the problem is caused by this Warning and prevents me from looking deeper in my Code. Thanks for your Help.