Error when uploading a stemcell

[omazhary]

We're trying to update to the 3196 stemcell. So, by running bosh prepare deployment the stemcell is downloaded, uploaded, verified, and then the following issue occurs: Somehow the director is unable to interpret the tarball, and returns an error.

The stack-trace is as follows:

$ bosh prepare deployment
RSA 1024 bit CA certificates are loaded due to old openssl compatibility

Fetching release 'cf' to satisfy template references
Ref 'd35c355f03f00cc7188e1c14c6f6348a46c44957' has been checkout into:
- /home/vcap/workspace/landscape/cf-boshworkspace/.releases/cf

Release 'cf/230.1' exists
Skipping upload

Uploading 'bosh-openstack-kvm-ubuntu-trusty-go_agent/3196'
Uploading stemcell 'bosh-stemcell-3196-openstack-kvm-ubuntu-trusty-go_agent.tgz'

Acting as user 'admin' on 'bosh'

Verifying stemcell...
File exists and readable                                     OK
Verifying tarball...
Read tarball                                                 OK
/home/vcap/.rvm/gems/ruby-2.1.7/gems/minitar-0.5.4/lib/archive/tar/minitar.rb:630:in `read': unexpected end of file (Zlib::GzipFile::Error)
        from /home/vcap/.rvm/gems/ruby-2.1.7/gems/minitar-0.5.4/lib/archive/tar/minitar.rb:630:in `block in each_entry'
        from /home/vcap/.rvm/gems/ruby-2.1.7/gems/minitar-0.5.4/lib/archive/tar/minitar.rb:611:in `loop'
        from /home/vcap/.rvm/gems/ruby-2.1.7/gems/minitar-0.5.4/lib/archive/tar/minitar.rb:611:in `each_entry'
        from /home/vcap/.rvm/gems/ruby-2.1.7/gems/minitar-0.5.4/lib/archive/tar/minitar.rb:593:in `each'
        from /home/vcap/.rvm/gems/ruby-2.1.7/gems/minitar-0.5.4/lib/archive/tar/minitar.rb:685:in `each'
        from /home/vcap/.rvm/gems/ruby-2.1.7/gems/bosh_cli-1.3192.0/lib/cli/stemcell.rb:34:in `perform_validation'
        from /home/vcap/.rvm/gems/ruby-2.1.7/gems/bosh_cli-1.3192.0/lib/cli/validation.rb:18:in `validate'
        from /home/vcap/.rvm/gems/ruby-2.1.7/gems/bosh_cli-1.3192.0/lib/cli/commands/stemcell.rb:63:in `upload'
        from /home/vcap/.rvm/gems/ruby-2.1.7/gems/bosh-workspace-0.9.11/lib/bosh/workspace/helpers/stemcell_helper.rb:19:in `stemcell_upload'
        from /home/vcap/.rvm/gems/ruby-2.1.7/gems/bosh-workspace-0.9.11/lib/bosh/cli/commands/prepare.rb:90:in `cached_stemcell_upload'
        from /home/vcap/.rvm/gems/ruby-2.1.7/gems/bosh-workspace-0.9.11/lib/bosh/cli/commands/prepare.rb:79:in `prepare_stemcell'
        from /home/vcap/.rvm/gems/ruby-2.1.7/gems/bosh-workspace-0.9.11/lib/bosh/cli/commands/prepare.rb:70:in `block in prepare_stemcells'
        from /home/vcap/.rvm/gems/ruby-2.1.7/gems/bosh-workspace-0.9.11/lib/bosh/cli/commands/prepare.rb:69:in `each'
        from /home/vcap/.rvm/gems/ruby-2.1.7/gems/bosh-workspace-0.9.11/lib/bosh/cli/commands/prepare.rb:69:in `prepare_stemcells'
        from /home/vcap/.rvm/gems/ruby-2.1.7/gems/bosh-workspace-0.9.11/lib/bosh/cli/commands/prepare.rb:23:in `prepare'
        from /home/vcap/.rvm/gems/ruby-2.1.7/gems/bosh_cli-1.3192.0/lib/cli/command_handler.rb:57:in `run'
        from /home/vcap/.rvm/gems/ruby-2.1.7/gems/bosh_cli-1.3192.0/lib/cli/runner.rb:59:in `run'
        from /home/vcap/.rvm/gems/ruby-2.1.7/gems/bosh_cli-1.3192.0/bin/bosh:19:in `<top (required)>'
        from /home/vcap/.rvm/gems/ruby-2.1.7/bin/bosh:23:in `load'
        from /home/vcap/.rvm/gems/ruby-2.1.7/bin/bosh:23:in `<main>'
        from /home/vcap/.rvm/gems/ruby-2.1.7/bin/ruby_executable_hooks:15:in `eval'
        from /home/vcap/.rvm/gems/ruby-2.1.7/bin/ruby_executable_hooks:15:in `<main>'

[rkoster]

@omazhary could you be running out of disk space? It looks like your stemcell is somehow corrupted. The corrupted file is located in ./.stemcells/bosh-stemcell-3196-openstack-kvm-ubuntu-trusty-go_agent.tgz.

[omazhary]

Just to make sure, I just downloaded it, and uploaded it manually using bosh upload stemcell after checking the hash. It worked fine. @rkoster, I don't think memory is the problem.

[rkoster]

I'm glad you got past this issue. I don't think this is an bosh-workspace issue, since the error was raised from bosh_cli-1.3192.0/lib/cli/stemcell.rb:34. Also for the stemcell download bosh-workspace is just using the bosh_cli. I have seen similar issues due to network hiccup during stemcell downloads. I will close this issue now. Please reopen when this issue happens a again in a persistent way.