rubyzip-1.2.0 failed to allocate memory (NoMemoryError) while syncing big file

[tal5ab]

Hi i am using your plugin to sync our folders while using vagrant :

vagrant plugin list winrm-fs (0.2.2)

• Version Constraint: 0.2.2 vagrant-aws (0.6.0)
• Version Constraint: 0.6.0 vagrant-awsinfo (0.0.8) vagrant-serverspec (1.1.0)
• Version Constraint: 1.1.0 vagrant-share (1.1.5, system) vagrant-winrm-syncedfolders (1.0.0)
• Version Constraint: 1.0.0

vagrant --version Vagrant 1.7.4

We have a few step process, so after starting a vm on AWS with up , we wish to place a big file and run provisioning to uplda the file and run new recepie

But when we try that we get this failure:

==> win: Uploading with WinRM: C:/DevOps_31.03.16x/kitchen/output => /vagrant/output C:/Users/tabramso/.vagrant.d/gems/gems/rubyzip-1.2.0/lib/zip/deflater.rb:6:ininitialize': failed to allocate memory (NoMemoryError) from C:/Users/tabramso/.vagrant.d/gems/gems/rubyzip-1.2.0/lib/zip/deflater.rb:6:in new' from C:/Users/tabramso/.vagrant.d/gems/gems/rubyzip-1.2.0/lib/zip/deflater.rb:6:ininitialize' from C:/Users/tabramso/.vagrant.d/gems/gems/rubyzip-1.2.0/lib/zip/output_stream.rb:148:in new' from C:/Users/tabramso/.vagrant.d/gems/gems/rubyzip-1.2.0/lib/zip/output_stream.rb:148:inget_compressor' from C:/Users/tabramso/.vagrant.d/gems/gems/rubyzip-1.2.0/lib/zip/output_stream.rb:142:in init_next_entry' from C:/Users/tabramso/.vagrant.d/gems/gems/rubyzip-1.2.0/lib/zip/output_stream.rb:100:input_next_entry' from C:/Users/tabramso/.vagrant.d/gems/gems/rubyzip-1.2.0/lib/zip/entry.rb:537:in write_to_zip_output_stream' from C:/Users/tabramso/.vagrant.d/gems/gems/rubyzip-1.2.0/lib/zip/file.rb:313:inblock (3 levels) in commit' from C:/Users/tabramso/.vagrant.d/gems/gems/rubyzip-1.2.0/lib/zip/entry_set.rb:38:in call' from C:/Users/tabramso/.vagrant.d/gems/gems/rubyzip-1.2.0/lib/zip/entry_set.rb:38:inblock in each' from C:/Users/tabramso/.vagrant.d/gems/gems/rubyzip-1.2.0/lib/zip/entry_set.rb:37:in each' from C:/Users/tabramso/.vagrant.d/gems/gems/rubyzip-1.2.0/lib/zip/entry_set.rb:37:ineach' from C:/Users/tabramso/.vagrant.d/gems/gems/rubyzip-1.2.0/lib/zip/file.rb:312:in block (2 levels) in commit' from C:/Users/tabramso/.vagrant.d/gems/gems/rubyzip-1.2.0/lib/zip/output_stream.rb:53:inopen' from C:/Users/tabramso/.vagrant.d/gems/gems/rubyzip-1.2.0/lib/zip/file.rb:311:in block in commit' from C:/Users/tabramso/.vagrant.d/gems/gems/rubyzip-1.2.0/lib/zip/file.rb:407:inon_success_replace' from C:/Users/tabramso/.vagrant.d/gems/gems/rubyzip-1.2.0/lib/zip/file.rb:310:in commit' from C:/Users/tabramso/.vagrant.d/gems/gems/rubyzip-1.2.0/lib/zip/file.rb:334:inclose' from C:/Users/tabramso/.vagrant.d/gems/gems/winrm-fs-0.2.2/lib/winrm-fs/core/temp_zip_file.rb:144:in close' from C:/Users/tabramso/.vagrant.d/gems/gems/winrm-fs-0.2.2/lib/winrm-fs/core/temp_zip_file.rb:140:inbuild' from C:/Users/tabramso/.vagrant.d/gems/gems/winrm-fs-0.2.2/lib/winrm-fs/core/temp_zip_file.rb:59:in build' from C:/Users/tabramso/.vagrant.d/gems/gems/winrm-fs-0.2.2/lib/winrm-fs/core/upload_orchestrator.rb:113:increate_temp_zip_file' from C:/Users/tabramso/.vagrant.d/gems/gems/winrm-fs-0.2.2/lib/winrm-fs/core/upload_orchestrator.rb:78:in with_local_zip' from C:/Users/tabramso/.vagrant.d/gems/gems/winrm-fs-0.2.2/lib/winrm-fs/core/upload_orchestrator.rb:46:inupload_directory' from C:/Users/tabramso/.vagrant.d/gems/gems/winrm-fs-0.2.2/lib/winrm-fs/file_manager.rb:113:in upload' from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/plugins/communicators/winrm/shell.rb:70:inupload' from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/plugins/communicators/winrm/communicator.rb:163:in upload' from C:/Users/tabramso/.vagrant.d/gems/gems/vagrant-winrm-syncedfolders-1.0.0/lib/vagrant-winrm-syncedfolders/synced_folder.rb:44:inblock (2 levels) in sync' from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/lib/vagrant/util/retryable.rb:17:in retryable' from C:/Users/tabramso/.vagrant.d/gems/gems/vagrant-winrm-syncedfolders-1.0.0/lib/vagrant-winrm-syncedfolders/synced_folder.rb:42:inblock in sync' from C:/Users/tabramso/.vagrant.d/gems/gems/vagrant-winrm-syncedfolders-1.0.0/lib/vagrant-winrm-syncedfolders/synced_folder.rb:37:in tap' from C:/Users/tabramso/.vagrant.d/gems/gems/vagrant-winrm-syncedfolders-1.0.0/lib/vagrant-winrm-syncedfolders/synced_folder.rb:37:insync' from C:/Users/tabramso/.vagrant.d/gems/gems/vagrant-winrm-syncedfolders-1.0.0/lib/vagrant-winrm-syncedfolders/synced_folder.rb:26:in block in enable' from C:/Users/tabramso/.vagrant.d/gems/gems/vagrant-winrm-syncedfolders-1.0.0/lib/vagrant-winrm-syncedfolders/synced_folder.rb:23:ineach' from C:/Users/tabramso/.vagrant.d/gems/gems/vagrant-winrm-syncedfolders-1.0.0/lib/vagrant-winrm-syncedfolders/synced_folder.rb:23:in enable' from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/lib/vagrant/action/builtin/synced_folders.rb:92:inblock in call' from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/lib/vagrant/action/builtin/synced_folders.rb:89:in each' from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/lib/vagrant/action/builtin/synced_folders.rb:89:incall' from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/lib/vagrant/action/warden.rb:34:in call' from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/lib/vagrant/action/builtin/provision.rb:80:incall' from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/lib/vagrant/action/warden.rb:34:in call' from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/lib/vagrant/action/warden.rb:95:inblock in finalize_action' from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/lib/vagrant/action/warden.rb:34:in call' from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/lib/vagrant/action/warden.rb:34:incall' from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/lib/vagrant/action/builder.rb:116:in call' from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/lib/vagrant/action/runner.rb:66:inblock in run' from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/lib/vagrant/util/busy.rb:19:in busy' from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/lib/vagrant/action/runner.rb:66:inrun' from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/lib/vagrant/action/builtin/call.rb:53:in call' from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/lib/vagrant/action/warden.rb:34:incall' from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/lib/vagrant/action/builtin/config_validate.rb:25:in call' from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/lib/vagrant/action/warden.rb:34:incall' from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/lib/vagrant/action/builder.rb:116:in call' from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/lib/vagrant/action/runner.rb:66:inblock in run' from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/lib/vagrant/util/busy.rb:19:in busy' from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/lib/vagrant/action/runner.rb:66:inrun' from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/lib/vagrant/machine.rb:214:in action_raw' from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/lib/vagrant/machine.rb:191:inblock in action' from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/lib/vagrant/environment.rb:516:in lock' from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/lib/vagrant/machine.rb:178:incall' from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/lib/vagrant/machine.rb:178:in action' from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/plugins/commands/provision/command.rb:30:inblock in execute' from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/lib/vagrant/plugin/v2/command.rb:226:in block in with_target_vms' from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/lib/vagrant/plugin/v2/command.rb:220:ineach' from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/lib/vagrant/plugin/v2/command.rb:220:in with_target_vms' from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/plugins/commands/provision/command.rb:29:inexecute' from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/lib/vagrant/cli.rb:42:in execute' from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/lib/vagrant/environment.rb:301:incli' from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/bin/vagrant:174:in <main>' C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/bin/vagrant: unexpected return C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/bin/vagrant: unexpected return C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/bin/vagrant: unexpected return C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/bin/vagrant: unexpected return C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/bin/vagrant: unexpected return C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/bin/vagrant: unexpected return C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/bin/vagrant: unexpected return C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/bin/vagrant: unexpected return C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/bin/vagrant: unexpected return C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/bin/vagrant: unexpected return C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.7.4/bin/vagrant: unexpected return .... many more of these and then .... [FATAL] failed to allocate memory

i cant find any related error Really need your help on this hope you are still active and have time for help me out

Regards Tal

[tal5ab]

looking into the zip module i found this : https://github.com/rubyzip/rubyzip/pull/250 but i think its already in the version you use

SO i tried to replace line 6 in deflater.rb
from @zlib_deflater = ::Zlib::Deflate.new(level, -::Zlib::MAX_WBITS)

to @zlib_deflater = ::Zlib::Deflate.new(level, 15, Zlib::MAX_MEM_LEVEL, Zlib::HUFFMAN_ONLY)

and now vagrant fails on a different plugin: C:/Users/tabramso/.vagrant.d/gems/gems/winrm-fs-0.2.2/lib/winrm-fs/core/file_uploader.rb:53:in `chars': failed to allocate memory (NoMemoryError)

[maclennann]

Hi @tal5ab,

Thanks for the report and the investigation you've already performed. We don't really have any Windows servers left (:tada:), so we aren't particularly active on the this project anymore, but I'd be glad to help you out if I can.

It looks like what's happening after you made the change to deflator.rb is that rubyzip is able to zip up the directory, but then winrm-fs loads the whole zip into memory in order to do the transfer, which is where it's choking.

Luckily, it looks like winrm-fs-0.3.0+ was rewritten to buffer and stream large files from disk on transfer, so it might not be a problem in newer versions.

We had locked to a specific version of winrm-fs a while ago when vagrant 1.7.4 first came out, but that probably isn't a good restriction to have anymore.

It's possible that a combination of upgrading to Vagrant 1.8.3 (to get the new winrm-fs in your bundle) and removing the line from our gemspec would get things working for you (assuming winrm-fs kept the same interface).

Please feel free to try that and report back if you've got the time. Otherwise, I'll try to get to it soon, but I can't make any guarantees.

[tal5ab]

So after a long period , i am back on this So first thing i tried was upgrading to vagrant 1.8.5 Switched to winrm-fs 0.4.3 , and vagrant-aws 0.7.2 Then changed your plugin gemspec file and created the gem file (running rake) ,and installed it manually But now i am getting this failure:

vagrant up win-yyy
Bringing machine 'win-yyy' up with 'aws' provider...
==> win-yyy: Auto-generating node name for Chef...
C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/builtin/mixin_synced_folders.rb:137:in `block in synced_folders': Internal error. Report this as a bug. Invalid: winrm (RuntimeError)
    from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/builtin/mixin_synced_folders.rb:125:in `each'
    from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/builtin/mixin_synced_folders.rb:125:in `synced_folders'
    from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/builtin/synced_folders.rb:27:in `call'
    from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/warden.rb:34:in `call'
    from C:/Users/tabramso/.vagrant.d/gems/gems/vagrant-omnibus-1.4.1/lib/vagrant-omnibus/action/install_chef.rb:38:in `call'
    from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/warden.rb:34:in `call'
    from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/builtin/provision.rb:80:in `call'
    from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/warden.rb:34:in `call'
    from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/warden.rb:95:in `block in finalize_action'
    from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/warden.rb:34:in `call'
    from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/warden.rb:34:in `call'
    from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/builder.rb:116:in `call'
    from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/runner.rb:66:in `block in run'
    from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/util/busy.rb:19:in `busy'
    from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/runner.rb:66:in `run'
    from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/builtin/call.rb:53:in `call'
    from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/warden.rb:34:in `call'
    from C:/Users/tabramso/.vagrant.d/gems/gems/vagrant-aws-0.7.2/lib/vagrant-aws/action/connect_aws.rb:43:in `call'
    from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/warden.rb:34:in `call'
    from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/builtin/box_check_outdated.rb:36:in `call'
    from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/warden.rb:34:in `call'
    from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/builtin/config_validate.rb:25:in `call'
    from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/warden.rb:34:in `call'
    from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/builtin/handle_box.rb:56:in `call'
    from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/warden.rb:34:in `call'
    from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/builder.rb:116:in `call'
    from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/runner.rb:66:in `block in run'
    from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/util/busy.rb:19:in `busy'
    from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/action/runner.rb:66:in `run'
    from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/machine.rb:225:in `action_raw'
    from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/machine.rb:200:in `block in action'
    from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/environment.rb:561:in `lock'
    from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/machine.rb:186:in `call'
    from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/machine.rb:186:in `action'
    from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.8.5/lib/vagrant/batch_action.rb:82:in `block (2 levels) in run'
    from C:/HashiCorp/Vagrant/embedded/gems/gems/logging-2.1.0/lib/logging/diagnostic_context.rb:450:in `call'
    from C:/HashiCorp/Vagrant/embedded/gems/gems/logging-2.1.0/lib/logging/diagnostic_context.rb:450:in `block in create_with_logging_context'
Deploy step failed to run.

Which make me belive vagrant is not picking up the type=winrm that is handled by the plugin

Can you write down the proper steps i need to do to recreate your plugin with the change i did ? i might messed this part

[potashj]

Hey @tal5ab,

I think you're looking for the steps to build and install the plugin, which can be found in the vagrant documentation on plugins. In particular, the usage and plugin development basics sections should be helpful. Please let me know if that doesn't cover what you're looking for.

[tal5ab]

i did tried to follow these instructions i ran bundle gem , copied over your plugin files , edit the rackfile , but there are no instructions what to do next I tried the bundle exec vagrant , but my whole setup is in a different folder structure with lots of dependencies So i wanted to pack the gem , and tried the bundle package with lots of errors same with trying to follow the readme

I somehow got a gem file and tried the plugin install from local , but the new version never showed up in the plugin list

Anyway , we are moving to push our big files by FTP , so i will spend no more time on this

Thank you for your time Tal