search

MeltwaterArchive / datasift-connector

A set of components designed to retrieve data from third-party APIs and storage systems, and to pass that data in to a DataSift account.
http://datasift.com/
MIT License
9 stars 5 forks source link

Packer hangs #53

Closed nwolff closed 9 years ago

nwolff commented 9 years ago

For release https://github.com/datasift/datasift-connector/releases/tag/1.0.19-1

I run ./build.sh ....

I get:

+ cd ../chef
+ berks vendor vendor/cookbooks
No entry for terminal type "rxvt-unicode-256color";
using dumb terminal settings.
Resolving cookbook dependencies...
Fetching 'datasift-kafka' from source at cookbooks/datasift-kafka
Fetching 'datasift-stats' from source at cookbooks/datasift-stats
Fetching 'datasift-writer' from source at cookbooks/datasift-writer
Fetching 'gnip-reader' from source at cookbooks/gnip-reader
Fetching 'historics-api' from source at cookbooks/historics-api
Fetching 'historics-reader' from source at cookbooks/historics-reader
Fetching 'influxdb' from https://github.com/datasift/chef-influxdb.git (at 232e2af)
Fetching 'kafka' from https://github.com/mthssdrbrg/kafka-cookbook.git (at v0.7.1)
Fetching 'supervisor' from https://github.com/poise/supervisor.git (at v0.4.12)
Fetching 'twitterapi-reader' from source at cookbooks/twitterapi-reader
Fetching 'webapp' from source at cookbooks/webapp
Fetching cookbook index from https://supermarket.chef.io...
Installing 7-zip (1.0.2)
Installing apt (2.7.0)
Installing ark (0.9.0)
E, [2015-08-13T09:36:08.353261 #10417] ERROR -- : Actor crashed!
Errno::ETIMEDOUT: Connection timed out - connect(2) for "app-supermarket-prod-i-6171d6a7.opscode.us" port 443
    /opt/chefdk/embedded/lib/ruby/2.1.0/net/http.rb:879:in `initialize'
    /opt/chefdk/embedded/lib/ruby/2.1.0/net/http.rb:879:in `open'
    /opt/chefdk/embedded/lib/ruby/2.1.0/net/http.rb:879:in `block in connect'
    /opt/chefdk/embedded/lib/ruby/2.1.0/timeout.rb:76:in `timeout'
    /opt/chefdk/embedded/lib/ruby/2.1.0/net/http.rb:878:in `connect'
    /opt/chefdk/embedded/lib/ruby/2.1.0/net/http.rb:863:in `do_start'
    /opt/chefdk/embedded/lib/ruby/2.1.0/net/http.rb:852:in `start'
    /opt/chefdk/embedded/lib/ruby/2.1.0/net/http.rb:1369:in `request'
    /opt/chefdk/embedded/lib/ruby/2.1.0/net/http.rb:1128:in `get'
    /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/faraday-0.9.1/lib/faraday/adapter/net_http.rb:80:in `perform_request'
    /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/faraday-0.9.1/lib/faraday/adapter/net_http.rb:40:in `block in call'
    /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/faraday-0.9.1/lib/faraday/adapter/net_http.rb:87:in `with_net_http_connection'
    /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/faraday-0.9.1/lib/faraday/adapter/net_http.rb:32:in `call'
    /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/faraday-0.9.1/lib/faraday/request/retry.rb:110:in `call'
    /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/ridley-4.2.0/lib/ridley/middleware/follow_redirects.rb:67:in `perform_with_redirection'
    /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/ridley-4.2.0/lib/ridley/middleware/follow_redirects.rb:60:in `call'
    /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/faraday-0.9.1/lib/faraday/response.rb:8:in `call'
    /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/faraday-0.9.1/lib/faraday/response.rb:8:in `call'
    /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/faraday-0.9.1/lib/faraday/rack_builder.rb:139:in `build_response'
    /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/faraday-0.9.1/lib/faraday/connection.rb:377:in `run_request'
    /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/faraday-0.9.1/lib/faraday/connection.rb:140:in `get'
    /opt/chefdk/embedded/apps/berkshelf/lib/berkshelf/community_rest.rb:119:in `find'
    /opt/chefdk/embedded/apps/berkshelf/lib/berkshelf/community_rest.rb:103:in `download'
    /opt/chefdk/embedded/apps/berkshelf/lib/berkshelf/downloader.rb:62:in `try_download'
    /opt/chefdk/embedded/apps/berkshelf/lib/berkshelf/downloader.rb:36:in `block in download'
    /opt/chefdk/embedded/apps/berkshelf/lib/berkshelf/downloader.rb:35:in `each'
    /opt/chefdk/embedded/apps/berkshelf/lib/berkshelf/downloader.rb:35:in `download'
    /opt/chefdk/embedded/apps/berkshelf/lib/berkshelf/installer.rb:105:in `install'
    /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/celluloid-0.16.0/lib/celluloid/calls.rb:26:in `public_send'
    /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/celluloid-0.16.0/lib/celluloid/calls.rb:26:in `dispatch'
    /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/celluloid-0.16.0/lib/celluloid/calls.rb:63:in `dispatch'
    /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/celluloid-0.16.0/lib/celluloid/cell.rb:60:in `block in invoke'
    /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/celluloid-0.16.0/lib/celluloid/cell.rb:71:in `block in task'
    /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/celluloid-0.16.0/lib/celluloid/actor.rb:357:in `block in task'
    /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/celluloid-0.16.0/lib/celluloid/tasks.rb:57:in `block in initialize'
    /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/celluloid-0.16.0/lib/celluloid/tasks/task_fiber.rb:15:in `block in create'
E, [2015-08-13T09:36:08.353510 #10417] ERROR -- : Actor crashed!

I tried this 4 times over a one hour period and always got the same error.

Maybe related to #12 ?

Do you think you could publish the amazon AMI so that we wouldn't have to run packer at all?

AndyJS commented 9 years ago

The error seems to be occurring prior to Packer being run in build.sh. Berkshelf is updating Cookbook dependencies, and is connecting to an invalid URL for Ark or Build-essential. #12 is unrelated, and would cause issues later on in the deployment process, when Chef attempts to download RPM files from GitHub once it begins to build the AMI.

We have some information on purging the Berkshelf cache to allow the berks vendor command to fetch a fresh set of cookbook indexes from Chef Supermarket: https://github.com/datasift/datasift-connector/blob/97b3a12de7c62cc38255f42aefc7511f06d6f8bc/README.md#packer-builds-and-vagrant-provisioning

If the issue re-appears, attempt the following:

cd packer
rm -rf ~/.berkshelf/*
./build.sh <access-key> <secret-key>
nwolff commented 9 years ago

FWIW I just reran ./build.sh a few hours later and it worked. Maybe berkshelf automatically removed the stale cache entries ?

AndyJS commented 9 years ago

This was caused by a temporary opscode availability issue. Refer to last post by Nathen Harvey in the following thread:

https://forums.aws.amazon.com/thread.jspa?threadID=212224