Closed thewmf closed 10 years ago
The ASG looks working. Errors would happen in the microscaler container or there are problems about communications between the microscaler and the Docker daemon.
Could you show me following things?
ps
the result of ps
$ ps -ef | grep docker
root 788 1 0 09:03 ? 00:00:29 /usr/bin/docker -d -H tcp://0.0.0.0:2375 -H unix://var/run/docker.sock
root 5792 788 0 15:15 ? 00:00:01 /go/bin/skydock -ttl 30 -environment local -s /docker.sock -domain flyacmeair.net -name skydns
user1 10464 1649 0 15:27 pts/0 00:00:00 grep --color=auto docker
the log of health manager in microscaler container
$ ssh -i id_rsa root@`docker inspect --format '{{ .NetworkSettings.IPAddress }}' microscaler` cat /var/log/supervisor/healthmanager-stde*
D, [2014-06-25T19:24:16.748226 #16] DEBUG -- : HM ---> new instance id=e03b2785a2c8 account=user01 asg_name=acmeair_webapp
D, [2014-06-25T19:24:16.756307 #16] DEBUG -- : HM ---> new instance id=40b12efb22d1 account=user01 asg_name=acmeair_auth_service
D, [2014-06-25T19:24:17.669664 #16] DEBUG -- : HM ---> asg=acmeair_webapp account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T19:24:17.671354 #16] DEBUG -- : HM ---> asg=acmeair_auth_service account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T19:24:37.712058 #16] DEBUG -- : HM ---> asg=acmeair_webapp account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T19:24:37.713254 #16] DEBUG -- : HM ---> asg=acmeair_auth_service account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T19:24:57.747672 #16] DEBUG -- : HM ---> asg=acmeair_webapp account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T19:24:57.748981 #16] DEBUG -- : HM ---> asg=acmeair_auth_service account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T19:25:17.772234 #16] DEBUG -- : HM ---> target=1, actual=1, stalled=0 asg=acmeair_webapp account=user01
D, [2014-06-25T19:25:17.775425 #16] DEBUG -- : HM ---> target=1, actual=1, stalled=0 asg=acmeair_auth_service account=user01
D, [2014-06-25T19:25:37.802084 #16] DEBUG -- : HM ---> target=1, actual=1, stalled=0 asg=acmeair_webapp account=user01
D, [2014-06-25T19:25:37.807175 #16] DEBUG -- : HM ---> target=1, actual=1, stalled=0 asg=acmeair_auth_service account=user01
D, [2014-06-25T19:25:57.830541 #16] DEBUG -- : HM ---> target=1, actual=1, stalled=0 asg=acmeair_webapp account=user01
D, [2014-06-25T19:25:57.831781 #16] DEBUG -- : HM ---> target=1, actual=1, stalled=0 asg=acmeair_auth_service account=user01
D, [2014-06-25T19:26:17.880338 #16] DEBUG -- : HM ---> target=1, actual=1, stalled=0 asg=acmeair_webapp account=user01
D, [2014-06-25T19:26:17.881656 #16] DEBUG -- : HM ---> target=1, actual=1, stalled=0 asg=acmeair_auth_service account=user01
D, [2014-06-25T19:26:37.903726 #16] DEBUG -- : HM ---> target=1, actual=1, stalled=0 asg=acmeair_webapp account=user01
D, [2014-06-25T19:26:37.907400 #16] DEBUG -- : HM ---> target=1, actual=1, stalled=0 asg=acmeair_auth_service account=user01
the log of launcher in microscaler container
$ ssh -i id_rsa root@`docker inspect --format '{{ .NetworkSettings.IPAddress }}' microscaler` cat /var/log/supervisor/worker-launch-stde*
WARNING: This way of doing signal handling is now deprecated. Please see http://hone.heroku.com/resque/2012/08/21/resque-signals.html for more info.
D, [2014-06-25T19:24:07.428648 #228] DEBUG -- : deleting lock ba2ddf39-9ee8-4030-8751-939b46db3567 type=start_type asg=acmeair_auth_service
D, [2014-06-25T19:24:07.820279 #228] DEBUG -- : container launched with id=40b12efb22d1, user metadata={"asg_name"=>"acmeair_auth_service", "account"=>"user01", "domain"=>"user01.", "local_http_port"=>"", "metadata"=>{}, "az_name"=>"docker-local-1a"}, FQHN=acmeair-auth-service-3564cfbb3e.auth-service.local.flyacmeair.net, AZ=docker-local-1a
D, [2014-06-25T19:24:07.978242 #248] DEBUG -- : deleting lock 340b9d18-9c05-4294-91e0-6f35676df1c2 type=start_type asg=acmeair_webapp
D, [2014-06-25T19:24:08.285408 #248] DEBUG -- : container launched with id=e03b2785a2c8, user metadata={"asg_name"=>"acmeair_webapp", "account"=>"user01", "domain"=>"user01.", "local_http_port"=>"", "metadata"=>{}, "az_name"=>"docker-local-1a"}, FQHN=acmeair-webapp-ad02b56dd4.webapp.local.flyacmeair.net, AZ=docker-local-1a
$ ps -ef | grep docker
root 4708 1 0 Jun24 ? 00:01:44 /usr/bin/docker -d
root 24755 4708 0 15:57 ? 00:00:00 /go/bin/skydock -ttl 30 -environment local -s /docker.sock -domain flyacmeair.net -name skydns
wmf 26763 20446 0 16:03 pts/4 00:00:00 grep docker
$ ssh -i id_rsa root@`docker inspect --format '{{ .NetworkSettings.IPAddress }}' microscaler` cat /var/log/supervisor/healthmanager-stde*
/var/lib/gems/1.9.1/gems/mongo-1.10.2/lib/mongo/mongo_client.rb:422:in `connect': Failed to connect to a master node at localhost:27017 (Mongo::ConnectionFailure)
from /var/lib/gems/1.9.1/gems/mongo-1.10.2/lib/mongo/mongo_client.rb:661:in `setup'
from /var/lib/gems/1.9.1/gems/mongo-1.10.2/lib/mongo/mongo_client.rb:177:in `initialize'
from /usr/local/microscaler/lib/db_manager.rb:35:in `new'
from /usr/local/microscaler/lib/db_manager.rb:35:in `initialize'
from /usr/local/microscaler/lib/auth_manager.rb:31:in `new'
from /usr/local/microscaler/lib/auth_manager.rb:31:in `initialize'
from /usr/local/microscaler/lib/health_manager.rb:50:in `new'
from /usr/local/microscaler/lib/health_manager.rb:50:in `initialize'
from /usr/local/microscaler/bin/hm.rb:29:in `new'
from /usr/local/microscaler/bin/hm.rb:29:in `<main>'
D, [2014-06-25T21:00:07.679357 #52] DEBUG -- : HM ---> asg=acmeair_webapp account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T21:00:07.679885 #52] DEBUG -- : HM ---> asg=acmeair_auth_service account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T21:00:27.687621 #52] DEBUG -- : HM ---> asg=acmeair_webapp account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T21:00:27.688140 #52] DEBUG -- : HM ---> asg=acmeair_auth_service account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T21:00:47.728228 #52] DEBUG -- : HM ---> asg=acmeair_webapp account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T21:00:47.728726 #52] DEBUG -- : HM ---> asg=acmeair_auth_service account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T21:01:07.774866 #52] DEBUG -- : HM ---> target=1, actual=0, stalled=0 asg=acmeair_webapp account=user01
D, [2014-06-25T21:01:07.775322 #52] DEBUG -- : HM ---> scale-up asg=acmeair_webapp account=user01
D, [2014-06-25T21:01:07.778375 #52] DEBUG -- : HM ---> starting 1 instances for acmeair_webapp account=user01
D, [2014-06-25T21:01:07.780190 #52] DEBUG -- : launching instance for asg=acmeair_webapp and user=user01 with az=docker-local-1a
D, [2014-06-25T21:01:07.781625 #52] DEBUG -- : {"name"=>"acmeair_webapp", "availability_zones"=>["docker-local-1a"], "launch_configuration"=>"acmeair_webapp", "min_size"=>1, "max_size"=>4, "desired_capacity"=>1, "scale_out_cooldown"=>300, "scale_in_cooldown"=>60, "domain"=>"webapp.local.flyacmeair.net", "state"=>"started", "url"=>"N/A", "last_scale_out_ts"=>1403730067}
D, [2014-06-25T21:01:07.784298 #52] DEBUG -- : cannot lease lock for account user01 and asg acmeair_webapp
W, [2014-06-25T21:01:07.784341 #52] WARN -- : could not acquire lock for updating n instances
D, [2014-06-25T21:01:07.785323 #52] DEBUG -- : HM ---> target=1, actual=0, stalled=0 asg=acmeair_auth_service account=user01
D, [2014-06-25T21:01:07.785609 #52] DEBUG -- : HM ---> scale-up asg=acmeair_auth_service account=user01
D, [2014-06-25T21:01:07.786415 #52] DEBUG -- : HM ---> starting 1 instances for acmeair_auth_service account=user01
D, [2014-06-25T21:01:07.787042 #52] DEBUG -- : launching instance for asg=acmeair_auth_service and user=user01 with az=docker-local-1a
D, [2014-06-25T21:01:07.787961 #52] DEBUG -- : {"name"=>"acmeair_auth_service", "availability_zones"=>["docker-local-1a"], "launch_configuration"=>"acmeair_auth_service", "min_size"=>1, "max_size"=>4, "desired_capacity"=>1, "scale_out_cooldown"=>300, "scale_in_cooldown"=>60, "domain"=>"auth-service.local.flyacmeair.net", "state"=>"started", "url"=>"N/A", "last_scale_out_ts"=>1403730067}
D, [2014-06-25T21:01:07.790153 #52] DEBUG -- : cannot lease lock for account user01 and asg acmeair_auth_service
W, [2014-06-25T21:01:07.790196 #52] WARN -- : could not acquire lock for updating n instances
D, [2014-06-25T21:01:27.835326 #52] DEBUG -- : HM ---> asg=acmeair_webapp account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T21:01:27.835863 #52] DEBUG -- : HM ---> asg=acmeair_auth_service account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T21:01:47.878518 #52] DEBUG -- : HM ---> asg=acmeair_webapp account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T21:01:47.879049 #52] DEBUG -- : HM ---> asg=acmeair_auth_service account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T21:02:07.922313 #52] DEBUG -- : HM ---> asg=acmeair_webapp account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T21:02:07.922841 #52] DEBUG -- : HM ---> asg=acmeair_auth_service account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T21:02:27.938794 #52] DEBUG -- : HM ---> target=1, actual=0, stalled=0 asg=acmeair_webapp account=user01
D, [2014-06-25T21:02:27.939323 #52] DEBUG -- : HM ---> scale-up asg=acmeair_webapp account=user01
D, [2014-06-25T21:02:27.940532 #52] DEBUG -- : HM ---> starting 1 instances for acmeair_webapp account=user01
D, [2014-06-25T21:02:27.941316 #52] DEBUG -- : launching instance for asg=acmeair_webapp and user=user01 with az=docker-local-1a
D, [2014-06-25T21:02:27.942438 #52] DEBUG -- : {"name"=>"acmeair_webapp", "availability_zones"=>["docker-local-1a"], "launch_configuration"=>"acmeair_webapp", "min_size"=>1, "max_size"=>4, "desired_capacity"=>1, "scale_out_cooldown"=>300, "scale_in_cooldown"=>60, "domain"=>"webapp.local.flyacmeair.net", "state"=>"started", "url"=>"N/A", "last_scale_out_ts"=>1403730147}
D, [2014-06-25T21:02:27.944870 #52] DEBUG -- : cannot lease lock for account user01 and asg acmeair_webapp
W, [2014-06-25T21:02:27.944912 #52] WARN -- : could not acquire lock for updating n instances
D, [2014-06-25T21:02:27.945867 #52] DEBUG -- : HM ---> target=1, actual=0, stalled=0 asg=acmeair_auth_service account=user01
D, [2014-06-25T21:02:27.946183 #52] DEBUG -- : HM ---> scale-up asg=acmeair_auth_service account=user01
D, [2014-06-25T21:02:27.947042 #52] DEBUG -- : HM ---> starting 1 instances for acmeair_auth_service account=user01
D, [2014-06-25T21:02:27.947767 #52] DEBUG -- : launching instance for asg=acmeair_auth_service and user=user01 with az=docker-local-1a
D, [2014-06-25T21:02:27.948704 #52] DEBUG -- : {"name"=>"acmeair_auth_service", "availability_zones"=>["docker-local-1a"], "launch_configuration"=>"acmeair_auth_service", "min_size"=>1, "max_size"=>4, "desired_capacity"=>1, "scale_out_cooldown"=>300, "scale_in_cooldown"=>60, "domain"=>"auth-service.local.flyacmeair.net", "state"=>"started", "url"=>"N/A", "last_scale_out_ts"=>1403730147}
D, [2014-06-25T21:02:27.954911 #52] DEBUG -- : cannot lease lock for account user01 and asg acmeair_auth_service
W, [2014-06-25T21:02:27.954970 #52] WARN -- : could not acquire lock for updating n instances
D, [2014-06-25T21:02:47.996062 #52] DEBUG -- : HM ---> asg=acmeair_webapp account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T21:02:47.996581 #52] DEBUG -- : HM ---> asg=acmeair_auth_service account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T21:03:08.036771 #52] DEBUG -- : HM ---> asg=acmeair_webapp account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T21:03:08.037253 #52] DEBUG -- : HM ---> asg=acmeair_auth_service account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T21:03:28.078341 #52] DEBUG -- : HM ---> target=1, actual=0, stalled=0 asg=acmeair_webapp account=user01
D, [2014-06-25T21:03:28.078810 #52] DEBUG -- : HM ---> scale-up asg=acmeair_webapp account=user01
D, [2014-06-25T21:03:28.079960 #52] DEBUG -- : HM ---> starting 1 instances for acmeair_webapp account=user01
D, [2014-06-25T21:03:28.080873 #52] DEBUG -- : launching instance for asg=acmeair_webapp and user=user01 with az=docker-local-1a
D, [2014-06-25T21:03:28.082218 #52] DEBUG -- : {"name"=>"acmeair_webapp", "availability_zones"=>["docker-local-1a"], "launch_configuration"=>"acmeair_webapp", "min_size"=>1, "max_size"=>4, "desired_capacity"=>1, "scale_out_cooldown"=>300, "scale_in_cooldown"=>60, "domain"=>"webapp.local.flyacmeair.net", "state"=>"started", "url"=>"N/A", "last_scale_out_ts"=>1403730208}
D, [2014-06-25T21:03:28.084715 #52] DEBUG -- : cannot lease lock for account user01 and asg acmeair_webapp
W, [2014-06-25T21:03:28.084777 #52] WARN -- : could not acquire lock for updating n instances
D, [2014-06-25T21:03:28.085617 #52] DEBUG -- : HM ---> target=1, actual=0, stalled=0 asg=acmeair_auth_service account=user01
D, [2014-06-25T21:03:28.085880 #52] DEBUG -- : HM ---> scale-up asg=acmeair_auth_service account=user01
D, [2014-06-25T21:03:28.086694 #52] DEBUG -- : HM ---> starting 1 instances for acmeair_auth_service account=user01
D, [2014-06-25T21:03:28.087417 #52] DEBUG -- : launching instance for asg=acmeair_auth_service and user=user01 with az=docker-local-1a
D, [2014-06-25T21:03:28.088401 #52] DEBUG -- : {"name"=>"acmeair_auth_service", "availability_zones"=>["docker-local-1a"], "launch_configuration"=>"acmeair_auth_service", "min_size"=>1, "max_size"=>4, "desired_capacity"=>1, "scale_out_cooldown"=>300, "scale_in_cooldown"=>60, "domain"=>"auth-service.local.flyacmeair.net", "state"=>"started", "url"=>"N/A", "last_scale_out_ts"=>1403730208}
D, [2014-06-25T21:03:28.090751 #52] DEBUG -- : cannot lease lock for account user01 and asg acmeair_auth_service
W, [2014-06-25T21:03:28.090792 #52] WARN -- : could not acquire lock for updating n instances
D, [2014-06-25T21:03:48.132052 #52] DEBUG -- : HM ---> asg=acmeair_webapp account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T21:03:48.132576 #52] DEBUG -- : HM ---> asg=acmeair_auth_service account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T21:04:08.173472 #52] DEBUG -- : HM ---> asg=acmeair_webapp account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T21:04:08.173971 #52] DEBUG -- : HM ---> asg=acmeair_auth_service account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T21:04:28.189179 #52] DEBUG -- : HM ---> asg=acmeair_webapp account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T21:04:28.189751 #52] DEBUG -- : HM ---> asg=acmeair_auth_service account=user01 is in COOLDOWN state! no action taken until cooldown expires.
Another Mongo connection problem?
$ ssh -i id_rsa root@`docker inspect --format '{{ .NetworkSettings.IPAddress }}' microscaler` cat /var/log/supervisor/worker-launch-stde*
WARNING: This way of doing signal handling is now deprecated. Please see http://hone.heroku.com/resque/2012/08/21/resque-signals.html for more info.
D, [2014-06-25T21:00:06.177348 #244] DEBUG -- : deleting lock 4fafa58e-973e-4917-80e2-5fef742790d5 type=start_type asg=acmeair_auth_service
D, [2014-06-25T21:00:06.178579 #244] DEBUG -- : container launched with id=, user metadata={"asg_name"=>"acmeair_auth_service", "account"=>"user01", "domain"=>"user01.", "local_http_port"=>"", "metadata"=>{}, "az_name"=>"docker-local-1a"}, FQHN=acmeair-auth-service-837aa3bbec.auth-service.local.flyacmeair.net, AZ=docker-local-1a
E, [2014-06-25T21:00:06.178712 #244] ERROR -- : Could not create & launch Docker container: Connection refused - connect(2) - ["/usr/local/microscaler/lib/im_worker.rb:87:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/job.rb:227:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:250:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:189:in `block in work'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `loop'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `work'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/tasks.rb:41:in `block (2 levels) in <top (required)>'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `call'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `block in execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `each'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:184:in `block in invoke_with_call_chain'", "/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:177:in `invoke_with_call_chain'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:170:in `invoke'", "/usr/local/microscaler/bin/rakefile:26:in `block (2 levels) in <top (required)>'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `call'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `block in execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `each'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:184:in `block in invoke_with_call_chain'", "/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:177:in `invoke_with_call_chain'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:170:in `invoke'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:143:in `invoke_task'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `block (2 levels) in top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `each'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `block in top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:110:in `run_with_threads'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:95:in `top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:73:in `block in run'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:160:in `standard_exception_handling'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:70:in `run'", "/usr/bin/rake:27:in `<main>'"]
D, [2014-06-25T21:00:06.196839 #252] DEBUG -- : deleting lock 165cbce4-9e79-471b-b0ab-c64e5765bcc8 type=start_type asg=acmeair_webapp
D, [2014-06-25T21:00:06.197867 #252] DEBUG -- : container launched with id=, user metadata={"asg_name"=>"acmeair_webapp", "account"=>"user01", "domain"=>"user01.", "local_http_port"=>"", "metadata"=>{}, "az_name"=>"docker-local-1a"}, FQHN=acmeair-webapp-3983cf6c71.webapp.local.flyacmeair.net, AZ=docker-local-1a
E, [2014-06-25T21:00:06.198015 #252] ERROR -- : Could not create & launch Docker container: Connection refused - connect(2) - ["/usr/local/microscaler/lib/im_worker.rb:87:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/job.rb:227:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:250:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:189:in `block in work'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `loop'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `work'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/tasks.rb:41:in `block (2 levels) in <top (required)>'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `call'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `block in execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `each'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:184:in `block in invoke_with_call_chain'", "/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:177:in `invoke_with_call_chain'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:170:in `invoke'", "/usr/local/microscaler/bin/rakefile:26:in `block (2 levels) in <top (required)>'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `call'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `block in execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `each'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:184:in `block in invoke_with_call_chain'", "/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:177:in `invoke_with_call_chain'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:170:in `invoke'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:143:in `invoke_task'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `block (2 levels) in top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `each'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `block in top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:110:in `run_with_threads'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:95:in `top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:73:in `block in run'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:160:in `standard_exception_handling'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:70:in `run'", "/usr/bin/rake:27:in `<main>'"]
D, [2014-06-25T21:01:11.222858 #260] DEBUG -- : deleting lock 647939fa-0673-467d-bb49-69317ac4e79a type=start_type asg=acmeair_webapp
D, [2014-06-25T21:01:11.224113 #260] DEBUG -- : container launched with id=, user metadata={"asg_name"=>"acmeair_webapp", "account"=>"user01", "domain"=>"user01.", "local_http_port"=>"", "metadata"=>{}, "az_name"=>"docker-local-1a"}, FQHN=acmeair-webapp-806b1e023b.webapp.local.flyacmeair.net, AZ=docker-local-1a
E, [2014-06-25T21:01:11.224270 #260] ERROR -- : Could not create & launch Docker container: Connection refused - connect(2) - ["/usr/local/microscaler/lib/im_worker.rb:87:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/job.rb:227:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:250:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:189:in `block in work'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `loop'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `work'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/tasks.rb:41:in `block (2 levels) in <top (required)>'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `call'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `block in execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `each'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:184:in `block in invoke_with_call_chain'", "/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:177:in `invoke_with_call_chain'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:170:in `invoke'", "/usr/local/microscaler/bin/rakefile:26:in `block (2 levels) in <top (required)>'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `call'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `block in execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `each'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:184:in `block in invoke_with_call_chain'", "/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:177:in `invoke_with_call_chain'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:170:in `invoke'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:143:in `invoke_task'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `block (2 levels) in top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `each'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `block in top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:110:in `run_with_threads'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:95:in `top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:73:in `block in run'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:160:in `standard_exception_handling'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:70:in `run'", "/usr/bin/rake:27:in `<main>'"]
D, [2014-06-25T21:01:11.242347 #268] DEBUG -- : deleting lock e790ae3a-53bd-4241-b38f-5edcdee6c3dd type=start_type asg=acmeair_auth_service
D, [2014-06-25T21:01:11.243521 #268] DEBUG -- : container launched with id=, user metadata={"asg_name"=>"acmeair_auth_service", "account"=>"user01", "domain"=>"user01.", "local_http_port"=>"", "metadata"=>{}, "az_name"=>"docker-local-1a"}, FQHN=acmeair-auth-service-d98c30c315.auth-service.local.flyacmeair.net, AZ=docker-local-1a
E, [2014-06-25T21:01:11.243658 #268] ERROR -- : Could not create & launch Docker container: Connection refused - connect(2) - ["/usr/local/microscaler/lib/im_worker.rb:87:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/job.rb:227:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:250:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:189:in `block in work'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `loop'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `work'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/tasks.rb:41:in `block (2 levels) in <top (required)>'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `call'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `block in execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `each'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:184:in `block in invoke_with_call_chain'", "/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:177:in `invoke_with_call_chain'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:170:in `invoke'", "/usr/local/microscaler/bin/rakefile:26:in `block (2 levels) in <top (required)>'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `call'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `block in execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `each'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:184:in `block in invoke_with_call_chain'", "/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:177:in `invoke_with_call_chain'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:170:in `invoke'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:143:in `invoke_task'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `block (2 levels) in top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `each'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `block in top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:110:in `run_with_threads'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:95:in `top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:73:in `block in run'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:160:in `standard_exception_handling'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:70:in `run'", "/usr/bin/rake:27:in `<main>'"]
D, [2014-06-25T21:02:31.269966 #276] DEBUG -- : deleting lock 58a78a97-a7ee-4ba8-9610-f5f5c50ff83d type=start_type asg=acmeair_webapp
D, [2014-06-25T21:02:31.271264 #276] DEBUG -- : container launched with id=, user metadata={"asg_name"=>"acmeair_webapp", "account"=>"user01", "domain"=>"user01.", "local_http_port"=>"", "metadata"=>{}, "az_name"=>"docker-local-1a"}, FQHN=acmeair-webapp-2ba9252ffe.webapp.local.flyacmeair.net, AZ=docker-local-1a
E, [2014-06-25T21:02:31.271392 #276] ERROR -- : Could not create & launch Docker container: Connection refused - connect(2) - ["/usr/local/microscaler/lib/im_worker.rb:87:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/job.rb:227:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:250:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:189:in `block in work'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `loop'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `work'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/tasks.rb:41:in `block (2 levels) in <top (required)>'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `call'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `block in execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `each'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:184:in `block in invoke_with_call_chain'", "/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:177:in `invoke_with_call_chain'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:170:in `invoke'", "/usr/local/microscaler/bin/rakefile:26:in `block (2 levels) in <top (required)>'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `call'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `block in execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `each'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:184:in `block in invoke_with_call_chain'", "/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:177:in `invoke_with_call_chain'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:170:in `invoke'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:143:in `invoke_task'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `block (2 levels) in top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `each'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `block in top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:110:in `run_with_threads'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:95:in `top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:73:in `block in run'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:160:in `standard_exception_handling'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:70:in `run'", "/usr/bin/rake:27:in `<main>'"]
D, [2014-06-25T21:02:31.289467 #284] DEBUG -- : deleting lock ee31162f-9370-471f-83de-4f9fc69ff9f9 type=start_type asg=acmeair_auth_service
D, [2014-06-25T21:02:31.290683 #284] DEBUG -- : container launched with id=, user metadata={"asg_name"=>"acmeair_auth_service", "account"=>"user01", "domain"=>"user01.", "local_http_port"=>"", "metadata"=>{}, "az_name"=>"docker-local-1a"}, FQHN=acmeair-auth-service-be0ef7a52a.auth-service.local.flyacmeair.net, AZ=docker-local-1a
E, [2014-06-25T21:02:31.290829 #284] ERROR -- : Could not create & launch Docker container: Connection refused - connect(2) - ["/usr/local/microscaler/lib/im_worker.rb:87:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/job.rb:227:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:250:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:189:in `block in work'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `loop'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `work'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/tasks.rb:41:in `block (2 levels) in <top (required)>'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `call'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `block in execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `each'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:184:in `block in invoke_with_call_chain'", "/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:177:in `invoke_with_call_chain'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:170:in `invoke'", "/usr/local/microscaler/bin/rakefile:26:in `block (2 levels) in <top (required)>'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `call'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `block in execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `each'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:184:in `block in invoke_with_call_chain'", "/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:177:in `invoke_with_call_chain'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:170:in `invoke'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:143:in `invoke_task'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `block (2 levels) in top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `each'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `block in top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:110:in `run_with_threads'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:95:in `top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:73:in `block in run'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:160:in `standard_exception_handling'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:70:in `run'", "/usr/bin/rake:27:in `<main>'"]
D, [2014-06-25T21:03:31.315464 #292] DEBUG -- : deleting lock f85944ca-cebd-46f6-a1fd-6dde9af11966 type=start_type asg=acmeair_webapp
D, [2014-06-25T21:03:31.316893 #292] DEBUG -- : container launched with id=, user metadata={"asg_name"=>"acmeair_webapp", "account"=>"user01", "domain"=>"user01.", "local_http_port"=>"", "metadata"=>{}, "az_name"=>"docker-local-1a"}, FQHN=acmeair-webapp-f55aef206c.webapp.local.flyacmeair.net, AZ=docker-local-1a
E, [2014-06-25T21:03:31.317029 #292] ERROR -- : Could not create & launch Docker container: Connection refused - connect(2) - ["/usr/local/microscaler/lib/im_worker.rb:87:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/job.rb:227:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:250:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:189:in `block in work'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `loop'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `work'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/tasks.rb:41:in `block (2 levels) in <top (required)>'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `call'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `block in execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `each'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:184:in `block in invoke_with_call_chain'", "/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:177:in `invoke_with_call_chain'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:170:in `invoke'", "/usr/local/microscaler/bin/rakefile:26:in `block (2 levels) in <top (required)>'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `call'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `block in execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `each'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:184:in `block in invoke_with_call_chain'", "/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:177:in `invoke_with_call_chain'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:170:in `invoke'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:143:in `invoke_task'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `block (2 levels) in top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `each'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `block in top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:110:in `run_with_threads'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:95:in `top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:73:in `block in run'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:160:in `standard_exception_handling'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:70:in `run'", "/usr/bin/rake:27:in `<main>'"]
D, [2014-06-25T21:03:31.334611 #300] DEBUG -- : deleting lock 649394ca-e33d-4bd3-b1f2-f924aef0d8c5 type=start_type asg=acmeair_auth_service
D, [2014-06-25T21:03:31.336610 #300] DEBUG -- : container launched with id=, user metadata={"asg_name"=>"acmeair_auth_service", "account"=>"user01", "domain"=>"user01.", "local_http_port"=>"", "metadata"=>{}, "az_name"=>"docker-local-1a"}, FQHN=acmeair-auth-service-22da5c9e3b.auth-service.local.flyacmeair.net, AZ=docker-local-1a
E, [2014-06-25T21:03:31.336741 #300] ERROR -- : Could not create & launch Docker container: Connection refused - connect(2) - ["/usr/local/microscaler/lib/im_worker.rb:87:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/job.rb:227:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:250:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:189:in `block in work'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `loop'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `work'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/tasks.rb:41:in `block (2 levels) in <top (required)>'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `call'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `block in execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `each'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:184:in `block in invoke_with_call_chain'", "/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:177:in `invoke_with_call_chain'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:170:in `invoke'", "/usr/local/microscaler/bin/rakefile:26:in `block (2 levels) in <top (required)>'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `call'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `block in execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `each'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:184:in `block in invoke_with_call_chain'", "/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:177:in `invoke_with_call_chain'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:170:in `invoke'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:143:in `invoke_task'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `block (2 levels) in top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `each'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `block in top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:110:in `run_with_threads'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:95:in `top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:73:in `block in run'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:160:in `standard_exception_handling'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:70:in `run'", "/usr/bin/rake:27:in `<main>'"]
D, [2014-06-25T21:04:51.364510 #322] DEBUG -- : deleting lock 16a3e1bc-e972-4c91-8c42-0ca51ba96595 type=start_type asg=acmeair_webapp
D, [2014-06-25T21:04:51.365804 #322] DEBUG -- : container launched with id=, user metadata={"asg_name"=>"acmeair_webapp", "account"=>"user01", "domain"=>"user01.", "local_http_port"=>"", "metadata"=>{}, "az_name"=>"docker-local-1a"}, FQHN=acmeair-webapp-f76c5da1df.webapp.local.flyacmeair.net, AZ=docker-local-1a
E, [2014-06-25T21:04:51.365933 #322] ERROR -- : Could not create & launch Docker container: Connection refused - connect(2) - ["/usr/local/microscaler/lib/im_worker.rb:87:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/job.rb:227:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:250:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:189:in `block in work'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `loop'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `work'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/tasks.rb:41:in `block (2 levels) in <top (required)>'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `call'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `block in execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `each'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:184:in `block in invoke_with_call_chain'", "/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:177:in `invoke_with_call_chain'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:170:in `invoke'", "/usr/local/microscaler/bin/rakefile:26:in `block (2 levels) in <top (required)>'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `call'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `block in execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `each'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:184:in `block in invoke_with_call_chain'", "/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:177:in `invoke_with_call_chain'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:170:in `invoke'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:143:in `invoke_task'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `block (2 levels) in top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `each'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `block in top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:110:in `run_with_threads'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:95:in `top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:73:in `block in run'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:160:in `standard_exception_handling'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:70:in `run'", "/usr/bin/rake:27:in `<main>'"]
D, [2014-06-25T21:04:51.383173 #330] DEBUG -- : deleting lock 97d9ced8-155f-4aa3-9b79-f41e085cf18f type=start_type asg=acmeair_auth_service
D, [2014-06-25T21:04:51.384408 #330] DEBUG -- : container launched with id=, user metadata={"asg_name"=>"acmeair_auth_service", "account"=>"user01", "domain"=>"user01.", "local_http_port"=>"", "metadata"=>{}, "az_name"=>"docker-local-1a"}, FQHN=acmeair-auth-service-d9953ae32f.auth-service.local.flyacmeair.net, AZ=docker-local-1a
E, [2014-06-25T21:04:51.384537 #330] ERROR -- : Could not create & launch Docker container: Connection refused - connect(2) - ["/usr/local/microscaler/lib/im_worker.rb:87:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/job.rb:227:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:250:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:189:in `block in work'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `loop'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `work'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/tasks.rb:41:in `block (2 levels) in <top (required)>'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `call'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `block in execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `each'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:184:in `block in invoke_with_call_chain'", "/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:177:in `invoke_with_call_chain'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:170:in `invoke'", "/usr/local/microscaler/bin/rakefile:26:in `block (2 levels) in <top (required)>'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `call'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `block in execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `each'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:184:in `block in invoke_with_call_chain'", "/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:177:in `invoke_with_call_chain'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:170:in `invoke'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:143:in `invoke_task'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `block (2 levels) in top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `each'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `block in top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:110:in `run_with_threads'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:95:in `top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:73:in `block in run'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:160:in `standard_exception_handling'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:70:in `run'", "/usr/bin/rake:27:in `<main>'"]
Is microscaler having trouble connecting to dockerd? Is this a permission problem? It may be relevant that my host system is Ubuntu 13.10 not boot2docker (for performance I am running on a real server not a VM).
The cause would be your Docker daemon does not listen TCP socket. The launcher reports connection refused by Docker daemon and your docker
process is not given -H
option. Please see Enable remote API access of Docker daemon via TCP socket.
Easy check Docker daemon is available via TCP
$ curl localhost:2375/_ping
OK
I didn't realize that enabling remote access was necessary for autoscaling. This is obvious in hindsight.
It starts properly now.
That's good.
Then I wait a few minutes, but I never see the webapp-liberty container get started
I don't know where to start debugging this.