ryandoyle / vagrancy

Your private Vagrant cloud
GNU General Public License v3.0
89 stars 29 forks source link

Uploading large boxes results in 500 error #2

Closed syntacticvexation closed 8 years ago

syntacticvexation commented 8 years ago

Hi,

Really great tool, thanks for making this available to the community. I have a question. I'm having problems uploading a large Vagrant box (approx. 3.6GB) to vagrancy. Are there parameters I can tweak in Puma to permit these large downloads - ie write to disk rather than try to keep the whole box in memory?

My log is below, thanks heaps.

2016-08-12_06:47:30.76917 192.168.1.6 - - [12/Aug/2016:16:47:30 +1000] "POST /api/v1/artifacts/username/boxname/vagrant.box HTTP/1.1" 200 97 0.0022
2016-08-12_06:53:58.00812 NoMemoryError - failed to allocate memory:
2016-08-12_06:53:58.01254       /srv/vagrancy-0.0.1-linux-x86_64/lib/app/lib/vagrancy/app.rb:27:in `block in <class:App>'
2016-08-12_06:53:58.01255       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1603:in `call'
2016-08-12_06:53:58.01256       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1603:in `block in compile!'
2016-08-12_06:53:58.01256       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:966:in `[]'
2016-08-12_06:53:58.01256       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:966:in `block (3 levels) in route!'
2016-08-12_06:53:58.01256       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:985:in `route_eval'
2016-08-12_06:53:58.01257       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:966:in `block (2 levels) in route!'
2016-08-12_06:53:58.01257       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1006:in `block in process_route'
2016-08-12_06:53:58.01257       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1004:in `catch'
2016-08-12_06:53:58.01257       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1004:in `process_route'
2016-08-12_06:53:58.01259       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:964:in `block in route!'
2016-08-12_06:53:58.01259       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:963:in `each'
2016-08-12_06:53:58.01259       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:963:in `route!'
2016-08-12_06:53:58.01259       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1076:in `block in dispatch!'
2016-08-12_06:53:58.01260       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `block in invoke'
2016-08-12_06:53:58.01260       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `catch'
2016-08-12_06:53:58.01260       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `invoke'
2016-08-12_06:53:58.01260       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1073:in `dispatch!'
2016-08-12_06:53:58.01261       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:898:in `block in call!'
2016-08-12_06:53:58.01262       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `block in invoke'
2016-08-12_06:53:58.01263       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `catch'
2016-08-12_06:53:58.01263       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `invoke'
2016-08-12_06:53:58.01263       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:898:in `call!'
2016-08-12_06:53:58.01264       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:886:in `call'
2016-08-12_06:53:58.01264       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18:in `call'
2016-08-12_06:53:58.01264       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb:16:in `call'
2016-08-12_06:53:58.01264       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb:18:in `call'
2016-08-12_06:53:58.01264       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
2016-08-12_06:53:58.01265       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
2016-08-12_06:53:58.01265       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'
2016-08-12_06:53:58.01265       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/rack-1.6.0/lib/rack/logger.rb:15:in `call'
2016-08-12_06:53:58.01266       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/puma-2.11.2/lib/puma/rack_patch.rb:13:in `call'
2016-08-12_06:53:58.01266       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:217:in `call'
2016-08-12_06:53:58.01266       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:210:in `call'
2016-08-12_06:53:58.01266       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/rack-1.6.0/lib/rack/head.rb:13:in `call'
2016-08-12_06:53:58.01266       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:180:in `call'
2016-08-12_06:53:58.01267       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:2014:in `call'
2016-08-12_06:53:58.01267       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1478:in `block in call'
2016-08-12_06:53:58.01267       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1788:in `synchronize'
2016-08-12_06:53:58.01267       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1478:in `call'
2016-08-12_06:53:58.01268       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/puma-2.11.2/lib/puma/configuration.rb:51:in `call'
2016-08-12_06:53:58.01268       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/puma-2.11.2/lib/puma/server.rb:507:in `handle_request'
2016-08-12_06:53:58.01268       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/puma-2.11.2/lib/puma/server.rb:375:in `process_client'
2016-08-12_06:53:58.01269       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/puma-2.11.2/lib/puma/server.rb:262:in `block in run'
2016-08-12_06:53:58.01269       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/puma-2.11.2/lib/puma/thread_pool.rb:104:in `call'
2016-08-12_06:53:58.01269       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/puma-2.11.2/lib/puma/thread_pool.rb:104:in `block in spawn_thread'
ryandoyle commented 8 years ago

Hey @syntacticvexation, thanks for trying out Vagrancy. Are you using Packer to upload this box or are you manually doing it via curl (or something similar)? I'll have to double check but I /think/ the POST /api/v1/artifacts/username/boxname/vagrant.box doesn't actually upload the box itself, but rather creates the information about the box only. The PUT request it where you actually upload the box.

syntacticvexation commented 8 years ago

I'm using Packer with the snippet shown in the README. I should have provided more of the log, I didn't think that the error would be interleaved in the log. Updated below:

2016-08-12_05:40:36.14891 192.168.1.6 - - [12/Aug/2016:15:40:36 +1000] "PUT /username/boxname/1.0.0/virtualbox HTTP/1.1" 500 30 0.2112
2016-08-12_06:02:41.06790 192.168.1.6 - - [12/Aug/2016:16:02:41 +1000] "GET /api/v1/authenticate HTTP/1.1" 200 - 0.0006
2016-08-12_06:45:13.74688 192.168.1.6 - - [12/Aug/2016:16:45:13 +1000] "GET /api/v1/artifacts/username/boxname HTTP/1.1" 200 67 0.0037
2016-08-12_06:47:30.76917 192.168.1.6 - - [12/Aug/2016:16:47:30 +1000] "POST /api/v1/artifacts/username/boxname/vagrant.box HTTP/1.1" 200 97 0.0022
2016-08-12_06:53:58.00812 NoMemoryError - failed to allocate memory:
2016-08-12_06:53:58.01254       /srv/vagrancy-0.0.1-linux-x86_64/lib/app/lib/vagrancy/app.rb:27:in `block in <class:App>'
2016-08-12_06:53:58.01255       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1603:in `call'
2016-08-12_06:53:58.01256       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1603:in `block in compile!'
2016-08-12_06:53:58.01256       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:966:in `[]'
2016-08-12_06:53:58.01256       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:966:in `block (3 levels) in route!'
2016-08-12_06:53:58.01256       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:985:in `route_eval'
2016-08-12_06:53:58.01257       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:966:in `block (2 levels) in route!'
2016-08-12_06:53:58.01257       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1006:in `block in process_route'
2016-08-12_06:53:58.01257       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1004:in `catch'
2016-08-12_06:53:58.01257       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1004:in `process_route'
2016-08-12_06:53:58.01259       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:964:in `block in route!'
2016-08-12_06:53:58.01259       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:963:in `each'
2016-08-12_06:53:58.01259       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:963:in `route!'
2016-08-12_06:53:58.01259       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1076:in `block in dispatch!'
2016-08-12_06:53:58.01260       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `block in invoke'
2016-08-12_06:53:58.01260       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `catch'
2016-08-12_06:53:58.01260       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `invoke'
2016-08-12_06:53:58.01260       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1073:in `dispatch!'
2016-08-12_06:53:58.01261       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:898:in `block in call!'
2016-08-12_06:53:58.01262       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `block in invoke'
2016-08-12_06:53:58.01263       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `catch'
2016-08-12_06:53:58.01263       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in `invoke'
2016-08-12_06:53:58.01263       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:898:in `call!'
2016-08-12_06:53:58.01264       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:886:in `call'
2016-08-12_06:53:58.01264       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18:in `call'
2016-08-12_06:53:58.01264       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb:16:in `call'
2016-08-12_06:53:58.01264       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb:18:in `call'
2016-08-12_06:53:58.01264       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
2016-08-12_06:53:58.01265       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
2016-08-12_06:53:58.01265       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'
2016-08-12_06:53:58.01265       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/rack-1.6.0/lib/rack/logger.rb:15:in `call'
2016-08-12_06:53:58.01266       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/puma-2.11.2/lib/puma/rack_patch.rb:13:in `call'
2016-08-12_06:53:58.01266       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:217:in `call'
2016-08-12_06:53:58.01266       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:210:in `call'
2016-08-12_06:53:58.01266       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/rack-1.6.0/lib/rack/head.rb:13:in `call'
2016-08-12_06:53:58.01266       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:180:in `call'
2016-08-12_06:53:58.01267       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:2014:in `call'
2016-08-12_06:53:58.01267       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1478:in `block in call'
2016-08-12_06:53:58.01267       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1788:in `synchronize'
2016-08-12_06:53:58.01267       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1478:in `call'
2016-08-12_06:53:58.01268       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/puma-2.11.2/lib/puma/configuration.rb:51:in `call'
2016-08-12_06:53:58.01268       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/puma-2.11.2/lib/puma/server.rb:507:in `handle_request'
2016-08-12_06:53:58.01268       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/puma-2.11.2/lib/puma/server.rb:375:in `process_client'
2016-08-12_06:53:58.01269       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/puma-2.11.2/lib/puma/server.rb:262:in `block in run'
2016-08-12_06:53:58.01269       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/puma-2.11.2/lib/puma/thread_pool.rb:104:in `call'
2016-08-12_06:53:58.01269       /srv/vagrancy/lib/vendor/ruby/2.2.0/gems/puma-2.11.2/lib/puma/thread_pool.rb:104:in `block in spawn_thread'
ryandoyle commented 8 years ago

Ah yeah I get it now, the POST did complete but the PUT failed (and hence failed to print a log line). I'll see how I can get Puma/Rack/Sinatra do deal with this as an IO stream so it doesn't end up in memory.

syntacticvexation commented 8 years ago

Awesome thanks.

syntacticvexation commented 8 years ago

I've created a pull request for this one, see #3