`System.OutOfMemoryException` when copying large file

[dusek]

Repro:

$ go get github.com/packer-community/winrmcp
$ cd /path/to/winrmcp
$ make
$ bin/winrmcp -user vagrant -pass vagrant -addr localhost:2894 /Applications/VirtualBox.app/Contents/MacOS/VBoxGuestAdditions.iso VBoxGuestAdditions.iso
#< CLIXML
<Objs Version="1.1.0.1" xmlns="http://schemas.microsoft.com/powershell/2004/04"><S S="Error">Exception calling "FromBase64String" with "1" argument(s): "Create excepti_x000D__x000A_</S><S S="Error">on of type System.OutOfMemoryException."_x000D__x000A_</S><S S="Error">At line:15 char:47_x000D__x000A_</S><S S="Error">+             $bytes = [System.Convert]::FromBase64String &lt;&lt;&lt;&lt; ($base64_string)_x000D__x000A_</S><S S="Error"> _x000D__x000A_</S><S S="Error">    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException_x000D__x000A_</S><S S="Error">    + FullyQualifiedErrorId : DotNetMethodException_x000D__x000A_</S><S S="Error"> _x000D__x000A_</S><S S="Error">Exception calling "WriteAllBytes" with "2" argument(s): "Value cannot be NUL_x000D__x000A_</S><S S="Error">L._x000D__x000A_</S><S S="Error">Paremeter name: bytes"_x000D__x000A_</S><S S="Error">At line:16 char:35_x000D__x000A_</S><S S="Error">+             [System.IO.File]::WriteAllBytes &lt;&lt;&lt;&lt; ($dest_file_path, $bytes)_x000D__x000A_</S><S S="Error">    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException_x000D__x000A_</S><S S="Error">    + FullyQualifiedErrorId : DotNetMethodException_x000D__x000A_</S><S S="Error"> _x000D__x000A_</S></Objs>Error restoring file from $env:TEMP\winrmcp-563f367b-b70a-7f21-15a2-cf378c7ce057.tmp to VBoxGuestAdditions.iso: restore operation returned code=1

When I raise the Memory limit of PowerShell, the problem goes away (i.e., I can successfully copy the same big file to the VM) - inspiration from here:

powershell -Command "Set-Item WSMan:\localhost\Shell\MaxMemoryPerShellMB 1024" <NUL
%SystemRoot%\SysWOW64\cmd.exe /c powershell -Command "Set-Item WSMan:\localhost\Shell\MaxMemoryPerShellMB 1024" <NUL

Size of the uploaded file: 56M (from ls -lh)

Found when using boxcutter/windows to build a base box.

I think this is caused by cp.go:128 which in this case tries to decode a ~78MB base64 string (if encoded in ASCII) into ~56MB in-memory bytes (and might still be holding another ~78MB string array in memory). I think it would be better if the decoding of the base64 file was done in a "streaming way" - read n lines, decode them from base64, append the bytes to a temp binary file, repeat, and at the end (when end of base64 file reached), move the temp binary file to the destination.

OS X: 10.11.1 (host) Win: 7 Pro x64 Czech (guest) - without SP1 VirtualBox: 5.0.8

Notes: the error message was translated from Czech, because Czech version of Windows (which I am using) does log certain parts of error output in the Windows language (and some parts in English). Specifically:

• "Create exception of type System.OutOfMemoryException."
• "Value cannot be NUL"
• "Parameter names"

So just saying in case you would like to Google them (but I think the problem is obvious even without Googling them).

[dusek]

Some possible tips for writing a memory efficient version.

[dusek]

I found that the machine where this was failing was configured with:

winrm set winrm/config/winrs @{MaxMemoryPerShellMB="800"}

[samsonkolge]

I am facing the exact same issue for transferring a file of 22MB in size using cookbook_file over winrm. Below is the error dump:

kolge@kolge2 MINGW64 ~/repos/chef-repo/cookbooks/m1-2.0 (master) $ kitchen converge -----> Starting Kitchen (v1.8.0) -----> Converging ... Preparing files for transfer Preparing dna.json Resolving cookbook dependencies with Berkshelf 4.3.3... Removing non-cookbook files before transfer Preparing validation.pem Preparing client.rb -----> Installing Chef Omnibus (install only if missing)

   Downloading package from https://packages.chef.io/stable/windows/2012r2/chef-client-12.13.37-1-x64.msi

   Download complete.

   Successfully verified C:\Users\vagrant\AppData\Local\Temp\chef-true.msi

   Installing Chef Omnibus package C:\Users\vagrant\AppData\Local\Temp\chef-true.msi

   Installation complete

   Transferring files to <default-windows-2008-r2>

------Exception------- Class: Kitchen::ActionFailed Message: Failed to complete #converge action: [[WinRM::FS::Core::FileTransporter] Upload failed (exitcode: 1) Decode-Base64File : Exception of type 'System.OutOfMemoryException' was thrown. At line:20 char:24

• Decode-Base64File <<<< $tmp $decoded
  • CategoryInfo : NotSpecified: (:) [Decode-Base64File], OutOfMemo ryException
  • FullyQualifiedErrorId : System.OutOfMemoryException,Decode-Base64File

]

---

Please see .kitchen/logs/kitchen.log for more details Also try running kitchen diagnose --all for configuration

Has anybody got this error yet? Please advise.