search

EmergingTechnologyInstitute / acmeair-netflixoss-dockerlocal

acmeair-netflixoss-dockerlocal
Apache License 2.0
33 stars 11 forks source link

startminimum.sh never starts acmeair/webapp? #5

Closed thewmf closed 10 years ago

thewmf commented 10 years ago 
$ ./startminimum.sh 
Starting Skydns and Skydock
58d6826237bd9f43c6f835235cf31bb7caa2d2fe28ab00ca25b23f9cd5594e7c
c1fe4b74a622989d94a177ea17f1a56f9224df326dc70240e11f46a2d96b40c2
Starting Cassandra
3240bdc3487b8c8e33edf56739cbe62883401e7b238deff972d24d1faf4d7e53
Waiting Cassandra
Start data loader
log4j:WARN No appenders could be found for logger (org.springframework.core.env.StandardEnvironment).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
closing c* cluster
Starting Eureka
9852907e47198ebe3d00b031c57c65cd34f2005383467f1fe126671d84547ce4
eureka 172.17.0.163
Starting Zuul
c5cc408390bafb1acc6744992b37e11a93fd64da7b97d81977eb90e71a949e6a
zuul 172.17.0.164
Starting Microscaler
47ef16eb6eae68230a1bf52294fe4e9c8787a9dc55f6dd0f3fc1e4e91360f7d5
Waiting Microscaler
Starting Microscaler Agent
aa7e31bfc909cc0f6b763d448e93ac40518628477654dbb7e97c15156ae95444
Starting Asgard
8d05f1c066f66b7f34649f69493fcfb9c59d531b5cafac43b68f3f3833aa5ca8
asgard 172.17.0.167
Configure ASGs
Pseudo-terminal will not be allocated because stdin is not a terminal.
The authenticity of host '172.17.0.165 (172.17.0.165)' can't be established.
ECDSA key fingerprint is 0d:47:e1:70:4a:d7:13:4e:ba:12:1c:ee:58:6d:c6:32.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.17.0.165' (ECDSA) to the list of known hosts.
/usr/bin/xauth:  file /root/.Xauthority does not exist
Welcome to Ubuntu 14.04 LTS (GNU/Linux 3.2.0-58-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

stdin: is not a tty
"logging in @http://localhost:56785/asgcc/ user=user01 key=***** ..."
"OK"
"adding launch configuration {\"name\"=>\"acmeair_auth_service\", \"image_id\"=>\"acmeair/auth-service-liberty\", \"instances_type\"=>\"m1.small\", \"key\"=>\"key1\"} ..."
"OK"
"adding launch configuration {\"name\"=>\"acmeair_webapp\", \"image_id\"=>\"acmeair/webapp-liberty\", \"instances_type\"=>\"m1.small\", \"key\"=>\"key1\"} ..."
"OK"
"Some options not provided, loading missing options from defaults..."
"acmeair_auth_service"
{"name"=>"acmeair_auth_service", "availability_zones"=>["docker-local-1a"], "launch_configuration"=>"acmeair_auth_service", "min_size"=>1, "max_size"=>4, "desired_capacity"=>1, "scale_out_cooldown"=>300, "scale_in_cooldown"=>60, "domain"=>"auth-service.local.flyacmeair.net"}
"adding autoscaling group {\"name\"=>\"acmeair_auth_service\", \"availability_zones\"=>[\"docker-local-1a\"], \"launch_configuration\"=>\"acmeair_auth_service\", \"min_size\"=>1, \"max_size\"=>4, \"desired_capacity\"=>1, \"scale_out_cooldown\"=>300, \"scale_in_cooldown\"=>60, \"domain\"=>\"auth-service.local.flyacmeair.net\"} ..."
"OK"
"Some options not provided, loading missing options from defaults..."
"acmeair_webapp"
{"name"=>"acmeair_webapp", "availability_zones"=>["docker-local-1a"], "launch_configuration"=>"acmeair_webapp", "min_size"=>1, "max_size"=>4, "desired_capacity"=>1, "scale_out_cooldown"=>300, "scale_in_cooldown"=>60, "domain"=>"webapp.local.flyacmeair.net"}
"adding autoscaling group {\"name\"=>\"acmeair_webapp\", \"availability_zones\"=>[\"docker-local-1a\"], \"launch_configuration\"=>\"acmeair_webapp\", \"min_size\"=>1, \"max_size\"=>4, \"desired_capacity\"=>1, \"scale_out_cooldown\"=>300, \"scale_in_cooldown\"=>60, \"domain\"=>\"webapp.local.flyacmeair.net\"} ..."
"OK"
Starting ASGs
Pseudo-terminal will not be allocated because stdin is not a terminal.
Welcome to Ubuntu 14.04 LTS (GNU/Linux 3.11.0-12-generic x86_64)

 * Documentation:  https://help.ubuntu.com/
stdin: is not a tty
"logging in @http://localhost:56785/asgcc/ user=user01 key=***** ..."
"OK"
"running start autoscaling group for acmeair_auth_service ..."
"OK"
"running start autoscaling group for acmeair_webapp ..."
"OK"

Then I wait a few minutes, but I never see the webapp-liberty container get started

$ docker ps
CONTAINER ID        IMAGE                              COMMAND                CREATED             STATUS              PORTS                                                NAMES
aa7e31bfc909        acmeair/microscaler-agent:latest   /usr/bin/supervisord   9 minutes ago       Up 9 minutes        22/tcp                                               microscaler-agent   
8d05f1c066f6        acmeair/asgard:latest              /usr/bin/supervisord   9 minutes ago       Up 9 minutes        22/tcp, 80/tcp, 8009/tcp                             asgard              
c5cc408390ba        acmeair/zuul:latest                /usr/bin/supervisord   10 minutes ago      Up 10 minutes       22/tcp, 80/tcp, 8009/tcp                             zuul                
9852907e4719        acmeair/eureka:latest              /usr/bin/supervisord   10 minutes ago      Up 10 minutes       22/tcp, 80/tcp, 8009/tcp                             eureka              
47ef16eb6eae        acmeair/microscaler:latest         /usr/bin/supervisord   10 minutes ago      Up 10 minutes       22/tcp                                               microscaler         
3240bdc3487b        acmeair/cassandra:latest           /usr/bin/supervisord   10 minutes ago      Up 10 minutes       22/tcp                                               cassandra1          
c1fe4b74a622        crosbymichael/skydock:latest       /go/bin/skydock -ttl   10 minutes ago      Up 10 minutes                                                            skydock             
58d6826237bd        crosbymichael/skydns:latest        skydns -http 0.0.0.0   10 minutes ago      Up 10 minutes       172.17.42.1:53->53/udp, 172.17.42.1:8080->8080/tcp   skydns

I don't know where to start debugging this.

tkhinaba commented 10 years ago

The ASG looks working. Errors would happen in the microscaler container or there are problems about communications between the microscaler and the Docker daemon.

Could you show me following things?

  1. the result of ps
  2. the log of health manager in microscaler container
  3. the log of launcher in microscaler container

the result of ps

$ ps -ef | grep docker
root       788     1  0 09:03 ?        00:00:29 /usr/bin/docker -d -H tcp://0.0.0.0:2375 -H unix://var/run/docker.sock
root      5792   788  0 15:15 ?        00:00:01 /go/bin/skydock -ttl 30 -environment local -s /docker.sock -domain flyacmeair.net -name skydns
user1    10464  1649  0 15:27 pts/0    00:00:00 grep --color=auto docker

the log of health manager in microscaler container

$ ssh -i id_rsa root@`docker inspect --format '{{ .NetworkSettings.IPAddress }}' microscaler` cat /var/log/supervisor/healthmanager-stde*
D, [2014-06-25T19:24:16.748226 #16] DEBUG -- : HM ---> new instance id=e03b2785a2c8 account=user01 asg_name=acmeair_webapp
D, [2014-06-25T19:24:16.756307 #16] DEBUG -- : HM ---> new instance id=40b12efb22d1 account=user01 asg_name=acmeair_auth_service
D, [2014-06-25T19:24:17.669664 #16] DEBUG -- : HM ---> asg=acmeair_webapp account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T19:24:17.671354 #16] DEBUG -- : HM ---> asg=acmeair_auth_service account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T19:24:37.712058 #16] DEBUG -- : HM ---> asg=acmeair_webapp account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T19:24:37.713254 #16] DEBUG -- : HM ---> asg=acmeair_auth_service account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T19:24:57.747672 #16] DEBUG -- : HM ---> asg=acmeair_webapp account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T19:24:57.748981 #16] DEBUG -- : HM ---> asg=acmeair_auth_service account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T19:25:17.772234 #16] DEBUG -- : HM ---> target=1, actual=1, stalled=0 asg=acmeair_webapp account=user01
D, [2014-06-25T19:25:17.775425 #16] DEBUG -- : HM ---> target=1, actual=1, stalled=0 asg=acmeair_auth_service account=user01
D, [2014-06-25T19:25:37.802084 #16] DEBUG -- : HM ---> target=1, actual=1, stalled=0 asg=acmeair_webapp account=user01
D, [2014-06-25T19:25:37.807175 #16] DEBUG -- : HM ---> target=1, actual=1, stalled=0 asg=acmeair_auth_service account=user01
D, [2014-06-25T19:25:57.830541 #16] DEBUG -- : HM ---> target=1, actual=1, stalled=0 asg=acmeair_webapp account=user01
D, [2014-06-25T19:25:57.831781 #16] DEBUG -- : HM ---> target=1, actual=1, stalled=0 asg=acmeair_auth_service account=user01
D, [2014-06-25T19:26:17.880338 #16] DEBUG -- : HM ---> target=1, actual=1, stalled=0 asg=acmeair_webapp account=user01
D, [2014-06-25T19:26:17.881656 #16] DEBUG -- : HM ---> target=1, actual=1, stalled=0 asg=acmeair_auth_service account=user01
D, [2014-06-25T19:26:37.903726 #16] DEBUG -- : HM ---> target=1, actual=1, stalled=0 asg=acmeair_webapp account=user01
D, [2014-06-25T19:26:37.907400 #16] DEBUG -- : HM ---> target=1, actual=1, stalled=0 asg=acmeair_auth_service account=user01

the log of launcher in microscaler container

$ ssh -i id_rsa root@`docker inspect --format '{{ .NetworkSettings.IPAddress }}' microscaler` cat /var/log/supervisor/worker-launch-stde*
WARNING: This way of doing signal handling is now deprecated. Please see http://hone.heroku.com/resque/2012/08/21/resque-signals.html for more info.
D, [2014-06-25T19:24:07.428648 #228] DEBUG -- : deleting lock ba2ddf39-9ee8-4030-8751-939b46db3567 type=start_type asg=acmeair_auth_service
D, [2014-06-25T19:24:07.820279 #228] DEBUG -- : container launched with id=40b12efb22d1, user metadata={"asg_name"=>"acmeair_auth_service", "account"=>"user01", "domain"=>"user01.", "local_http_port"=>"", "metadata"=>{}, "az_name"=>"docker-local-1a"}, FQHN=acmeair-auth-service-3564cfbb3e.auth-service.local.flyacmeair.net, AZ=docker-local-1a
D, [2014-06-25T19:24:07.978242 #248] DEBUG -- : deleting lock 340b9d18-9c05-4294-91e0-6f35676df1c2 type=start_type asg=acmeair_webapp
D, [2014-06-25T19:24:08.285408 #248] DEBUG -- : container launched with id=e03b2785a2c8, user metadata={"asg_name"=>"acmeair_webapp", "account"=>"user01", "domain"=>"user01.", "local_http_port"=>"", "metadata"=>{}, "az_name"=>"docker-local-1a"}, FQHN=acmeair-webapp-ad02b56dd4.webapp.local.flyacmeair.net, AZ=docker-local-1a
thewmf commented 10 years ago 
$ ps -ef | grep docker
root      4708     1  0 Jun24 ?        00:01:44 /usr/bin/docker -d
root     24755  4708  0 15:57 ?        00:00:00 /go/bin/skydock -ttl 30 -environment local -s /docker.sock -domain flyacmeair.net -name skydns
wmf      26763 20446  0 16:03 pts/4    00:00:00 grep docker
$ ssh -i id_rsa root@`docker inspect --format '{{ .NetworkSettings.IPAddress }}' microscaler` cat /var/log/supervisor/healthmanager-stde*
/var/lib/gems/1.9.1/gems/mongo-1.10.2/lib/mongo/mongo_client.rb:422:in `connect': Failed to connect to a master node at localhost:27017 (Mongo::ConnectionFailure)
    from /var/lib/gems/1.9.1/gems/mongo-1.10.2/lib/mongo/mongo_client.rb:661:in `setup'
    from /var/lib/gems/1.9.1/gems/mongo-1.10.2/lib/mongo/mongo_client.rb:177:in `initialize'
    from /usr/local/microscaler/lib/db_manager.rb:35:in `new'
    from /usr/local/microscaler/lib/db_manager.rb:35:in `initialize'
    from /usr/local/microscaler/lib/auth_manager.rb:31:in `new'
    from /usr/local/microscaler/lib/auth_manager.rb:31:in `initialize'
    from /usr/local/microscaler/lib/health_manager.rb:50:in `new'
    from /usr/local/microscaler/lib/health_manager.rb:50:in `initialize'
    from /usr/local/microscaler/bin/hm.rb:29:in `new'
    from /usr/local/microscaler/bin/hm.rb:29:in `<main>'
D, [2014-06-25T21:00:07.679357 #52] DEBUG -- : HM ---> asg=acmeair_webapp account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T21:00:07.679885 #52] DEBUG -- : HM ---> asg=acmeair_auth_service account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T21:00:27.687621 #52] DEBUG -- : HM ---> asg=acmeair_webapp account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T21:00:27.688140 #52] DEBUG -- : HM ---> asg=acmeair_auth_service account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T21:00:47.728228 #52] DEBUG -- : HM ---> asg=acmeair_webapp account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T21:00:47.728726 #52] DEBUG -- : HM ---> asg=acmeair_auth_service account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T21:01:07.774866 #52] DEBUG -- : HM ---> target=1, actual=0, stalled=0 asg=acmeair_webapp account=user01
D, [2014-06-25T21:01:07.775322 #52] DEBUG -- : HM ---> scale-up asg=acmeair_webapp account=user01
D, [2014-06-25T21:01:07.778375 #52] DEBUG -- : HM ---> starting 1 instances for acmeair_webapp account=user01
D, [2014-06-25T21:01:07.780190 #52] DEBUG -- : launching instance for asg=acmeair_webapp and user=user01 with az=docker-local-1a
D, [2014-06-25T21:01:07.781625 #52] DEBUG -- : {"name"=>"acmeair_webapp", "availability_zones"=>["docker-local-1a"], "launch_configuration"=>"acmeair_webapp", "min_size"=>1, "max_size"=>4, "desired_capacity"=>1, "scale_out_cooldown"=>300, "scale_in_cooldown"=>60, "domain"=>"webapp.local.flyacmeair.net", "state"=>"started", "url"=>"N/A", "last_scale_out_ts"=>1403730067}
D, [2014-06-25T21:01:07.784298 #52] DEBUG -- : cannot lease lock for account user01 and asg acmeair_webapp
W, [2014-06-25T21:01:07.784341 #52]  WARN -- : could not acquire lock for updating n instances
D, [2014-06-25T21:01:07.785323 #52] DEBUG -- : HM ---> target=1, actual=0, stalled=0 asg=acmeair_auth_service account=user01
D, [2014-06-25T21:01:07.785609 #52] DEBUG -- : HM ---> scale-up asg=acmeair_auth_service account=user01
D, [2014-06-25T21:01:07.786415 #52] DEBUG -- : HM ---> starting 1 instances for acmeair_auth_service account=user01
D, [2014-06-25T21:01:07.787042 #52] DEBUG -- : launching instance for asg=acmeair_auth_service and user=user01 with az=docker-local-1a
D, [2014-06-25T21:01:07.787961 #52] DEBUG -- : {"name"=>"acmeair_auth_service", "availability_zones"=>["docker-local-1a"], "launch_configuration"=>"acmeair_auth_service", "min_size"=>1, "max_size"=>4, "desired_capacity"=>1, "scale_out_cooldown"=>300, "scale_in_cooldown"=>60, "domain"=>"auth-service.local.flyacmeair.net", "state"=>"started", "url"=>"N/A", "last_scale_out_ts"=>1403730067}
D, [2014-06-25T21:01:07.790153 #52] DEBUG -- : cannot lease lock for account user01 and asg acmeair_auth_service
W, [2014-06-25T21:01:07.790196 #52]  WARN -- : could not acquire lock for updating n instances
D, [2014-06-25T21:01:27.835326 #52] DEBUG -- : HM ---> asg=acmeair_webapp account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T21:01:27.835863 #52] DEBUG -- : HM ---> asg=acmeair_auth_service account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T21:01:47.878518 #52] DEBUG -- : HM ---> asg=acmeair_webapp account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T21:01:47.879049 #52] DEBUG -- : HM ---> asg=acmeair_auth_service account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T21:02:07.922313 #52] DEBUG -- : HM ---> asg=acmeair_webapp account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T21:02:07.922841 #52] DEBUG -- : HM ---> asg=acmeair_auth_service account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T21:02:27.938794 #52] DEBUG -- : HM ---> target=1, actual=0, stalled=0 asg=acmeair_webapp account=user01
D, [2014-06-25T21:02:27.939323 #52] DEBUG -- : HM ---> scale-up asg=acmeair_webapp account=user01
D, [2014-06-25T21:02:27.940532 #52] DEBUG -- : HM ---> starting 1 instances for acmeair_webapp account=user01
D, [2014-06-25T21:02:27.941316 #52] DEBUG -- : launching instance for asg=acmeair_webapp and user=user01 with az=docker-local-1a
D, [2014-06-25T21:02:27.942438 #52] DEBUG -- : {"name"=>"acmeair_webapp", "availability_zones"=>["docker-local-1a"], "launch_configuration"=>"acmeair_webapp", "min_size"=>1, "max_size"=>4, "desired_capacity"=>1, "scale_out_cooldown"=>300, "scale_in_cooldown"=>60, "domain"=>"webapp.local.flyacmeair.net", "state"=>"started", "url"=>"N/A", "last_scale_out_ts"=>1403730147}
D, [2014-06-25T21:02:27.944870 #52] DEBUG -- : cannot lease lock for account user01 and asg acmeair_webapp
W, [2014-06-25T21:02:27.944912 #52]  WARN -- : could not acquire lock for updating n instances
D, [2014-06-25T21:02:27.945867 #52] DEBUG -- : HM ---> target=1, actual=0, stalled=0 asg=acmeair_auth_service account=user01
D, [2014-06-25T21:02:27.946183 #52] DEBUG -- : HM ---> scale-up asg=acmeair_auth_service account=user01
D, [2014-06-25T21:02:27.947042 #52] DEBUG -- : HM ---> starting 1 instances for acmeair_auth_service account=user01
D, [2014-06-25T21:02:27.947767 #52] DEBUG -- : launching instance for asg=acmeair_auth_service and user=user01 with az=docker-local-1a
D, [2014-06-25T21:02:27.948704 #52] DEBUG -- : {"name"=>"acmeair_auth_service", "availability_zones"=>["docker-local-1a"], "launch_configuration"=>"acmeair_auth_service", "min_size"=>1, "max_size"=>4, "desired_capacity"=>1, "scale_out_cooldown"=>300, "scale_in_cooldown"=>60, "domain"=>"auth-service.local.flyacmeair.net", "state"=>"started", "url"=>"N/A", "last_scale_out_ts"=>1403730147}
D, [2014-06-25T21:02:27.954911 #52] DEBUG -- : cannot lease lock for account user01 and asg acmeair_auth_service
W, [2014-06-25T21:02:27.954970 #52]  WARN -- : could not acquire lock for updating n instances
D, [2014-06-25T21:02:47.996062 #52] DEBUG -- : HM ---> asg=acmeair_webapp account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T21:02:47.996581 #52] DEBUG -- : HM ---> asg=acmeair_auth_service account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T21:03:08.036771 #52] DEBUG -- : HM ---> asg=acmeair_webapp account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T21:03:08.037253 #52] DEBUG -- : HM ---> asg=acmeair_auth_service account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T21:03:28.078341 #52] DEBUG -- : HM ---> target=1, actual=0, stalled=0 asg=acmeair_webapp account=user01
D, [2014-06-25T21:03:28.078810 #52] DEBUG -- : HM ---> scale-up asg=acmeair_webapp account=user01
D, [2014-06-25T21:03:28.079960 #52] DEBUG -- : HM ---> starting 1 instances for acmeair_webapp account=user01
D, [2014-06-25T21:03:28.080873 #52] DEBUG -- : launching instance for asg=acmeair_webapp and user=user01 with az=docker-local-1a
D, [2014-06-25T21:03:28.082218 #52] DEBUG -- : {"name"=>"acmeair_webapp", "availability_zones"=>["docker-local-1a"], "launch_configuration"=>"acmeair_webapp", "min_size"=>1, "max_size"=>4, "desired_capacity"=>1, "scale_out_cooldown"=>300, "scale_in_cooldown"=>60, "domain"=>"webapp.local.flyacmeair.net", "state"=>"started", "url"=>"N/A", "last_scale_out_ts"=>1403730208}
D, [2014-06-25T21:03:28.084715 #52] DEBUG -- : cannot lease lock for account user01 and asg acmeair_webapp
W, [2014-06-25T21:03:28.084777 #52]  WARN -- : could not acquire lock for updating n instances
D, [2014-06-25T21:03:28.085617 #52] DEBUG -- : HM ---> target=1, actual=0, stalled=0 asg=acmeair_auth_service account=user01
D, [2014-06-25T21:03:28.085880 #52] DEBUG -- : HM ---> scale-up asg=acmeair_auth_service account=user01
D, [2014-06-25T21:03:28.086694 #52] DEBUG -- : HM ---> starting 1 instances for acmeair_auth_service account=user01
D, [2014-06-25T21:03:28.087417 #52] DEBUG -- : launching instance for asg=acmeair_auth_service and user=user01 with az=docker-local-1a
D, [2014-06-25T21:03:28.088401 #52] DEBUG -- : {"name"=>"acmeair_auth_service", "availability_zones"=>["docker-local-1a"], "launch_configuration"=>"acmeair_auth_service", "min_size"=>1, "max_size"=>4, "desired_capacity"=>1, "scale_out_cooldown"=>300, "scale_in_cooldown"=>60, "domain"=>"auth-service.local.flyacmeair.net", "state"=>"started", "url"=>"N/A", "last_scale_out_ts"=>1403730208}
D, [2014-06-25T21:03:28.090751 #52] DEBUG -- : cannot lease lock for account user01 and asg acmeair_auth_service
W, [2014-06-25T21:03:28.090792 #52]  WARN -- : could not acquire lock for updating n instances
D, [2014-06-25T21:03:48.132052 #52] DEBUG -- : HM ---> asg=acmeair_webapp account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T21:03:48.132576 #52] DEBUG -- : HM ---> asg=acmeair_auth_service account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T21:04:08.173472 #52] DEBUG -- : HM ---> asg=acmeair_webapp account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T21:04:08.173971 #52] DEBUG -- : HM ---> asg=acmeair_auth_service account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T21:04:28.189179 #52] DEBUG -- : HM ---> asg=acmeair_webapp account=user01 is in COOLDOWN state! no action taken until cooldown expires.
D, [2014-06-25T21:04:28.189751 #52] DEBUG -- : HM ---> asg=acmeair_auth_service account=user01 is in COOLDOWN state! no action taken until cooldown expires.

Another Mongo connection problem?

$ ssh -i id_rsa root@`docker inspect --format '{{ .NetworkSettings.IPAddress }}' microscaler` cat /var/log/supervisor/worker-launch-stde*
WARNING: This way of doing signal handling is now deprecated. Please see http://hone.heroku.com/resque/2012/08/21/resque-signals.html for more info.
D, [2014-06-25T21:00:06.177348 #244] DEBUG -- : deleting lock 4fafa58e-973e-4917-80e2-5fef742790d5 type=start_type asg=acmeair_auth_service
D, [2014-06-25T21:00:06.178579 #244] DEBUG -- : container launched with id=, user metadata={"asg_name"=>"acmeair_auth_service", "account"=>"user01", "domain"=>"user01.", "local_http_port"=>"", "metadata"=>{}, "az_name"=>"docker-local-1a"}, FQHN=acmeair-auth-service-837aa3bbec.auth-service.local.flyacmeair.net, AZ=docker-local-1a
E, [2014-06-25T21:00:06.178712 #244] ERROR -- : Could not create & launch Docker container: Connection refused - connect(2) -  ["/usr/local/microscaler/lib/im_worker.rb:87:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/job.rb:227:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:250:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:189:in `block in work'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `loop'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `work'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/tasks.rb:41:in `block (2 levels) in <top (required)>'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `call'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `block in execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `each'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:184:in `block in invoke_with_call_chain'", "/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:177:in `invoke_with_call_chain'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:170:in `invoke'", "/usr/local/microscaler/bin/rakefile:26:in `block (2 levels) in <top (required)>'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `call'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `block in execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `each'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:184:in `block in invoke_with_call_chain'", "/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:177:in `invoke_with_call_chain'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:170:in `invoke'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:143:in `invoke_task'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `block (2 levels) in top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `each'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `block in top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:110:in `run_with_threads'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:95:in `top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:73:in `block in run'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:160:in `standard_exception_handling'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:70:in `run'", "/usr/bin/rake:27:in `<main>'"]
D, [2014-06-25T21:00:06.196839 #252] DEBUG -- : deleting lock 165cbce4-9e79-471b-b0ab-c64e5765bcc8 type=start_type asg=acmeair_webapp
D, [2014-06-25T21:00:06.197867 #252] DEBUG -- : container launched with id=, user metadata={"asg_name"=>"acmeair_webapp", "account"=>"user01", "domain"=>"user01.", "local_http_port"=>"", "metadata"=>{}, "az_name"=>"docker-local-1a"}, FQHN=acmeair-webapp-3983cf6c71.webapp.local.flyacmeair.net, AZ=docker-local-1a
E, [2014-06-25T21:00:06.198015 #252] ERROR -- : Could not create & launch Docker container: Connection refused - connect(2) -  ["/usr/local/microscaler/lib/im_worker.rb:87:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/job.rb:227:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:250:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:189:in `block in work'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `loop'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `work'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/tasks.rb:41:in `block (2 levels) in <top (required)>'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `call'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `block in execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `each'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:184:in `block in invoke_with_call_chain'", "/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:177:in `invoke_with_call_chain'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:170:in `invoke'", "/usr/local/microscaler/bin/rakefile:26:in `block (2 levels) in <top (required)>'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `call'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `block in execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `each'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:184:in `block in invoke_with_call_chain'", "/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:177:in `invoke_with_call_chain'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:170:in `invoke'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:143:in `invoke_task'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `block (2 levels) in top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `each'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `block in top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:110:in `run_with_threads'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:95:in `top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:73:in `block in run'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:160:in `standard_exception_handling'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:70:in `run'", "/usr/bin/rake:27:in `<main>'"]
D, [2014-06-25T21:01:11.222858 #260] DEBUG -- : deleting lock 647939fa-0673-467d-bb49-69317ac4e79a type=start_type asg=acmeair_webapp
D, [2014-06-25T21:01:11.224113 #260] DEBUG -- : container launched with id=, user metadata={"asg_name"=>"acmeair_webapp", "account"=>"user01", "domain"=>"user01.", "local_http_port"=>"", "metadata"=>{}, "az_name"=>"docker-local-1a"}, FQHN=acmeair-webapp-806b1e023b.webapp.local.flyacmeair.net, AZ=docker-local-1a
E, [2014-06-25T21:01:11.224270 #260] ERROR -- : Could not create & launch Docker container: Connection refused - connect(2) -  ["/usr/local/microscaler/lib/im_worker.rb:87:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/job.rb:227:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:250:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:189:in `block in work'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `loop'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `work'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/tasks.rb:41:in `block (2 levels) in <top (required)>'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `call'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `block in execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `each'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:184:in `block in invoke_with_call_chain'", "/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:177:in `invoke_with_call_chain'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:170:in `invoke'", "/usr/local/microscaler/bin/rakefile:26:in `block (2 levels) in <top (required)>'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `call'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `block in execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `each'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:184:in `block in invoke_with_call_chain'", "/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:177:in `invoke_with_call_chain'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:170:in `invoke'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:143:in `invoke_task'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `block (2 levels) in top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `each'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `block in top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:110:in `run_with_threads'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:95:in `top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:73:in `block in run'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:160:in `standard_exception_handling'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:70:in `run'", "/usr/bin/rake:27:in `<main>'"]
D, [2014-06-25T21:01:11.242347 #268] DEBUG -- : deleting lock e790ae3a-53bd-4241-b38f-5edcdee6c3dd type=start_type asg=acmeair_auth_service
D, [2014-06-25T21:01:11.243521 #268] DEBUG -- : container launched with id=, user metadata={"asg_name"=>"acmeair_auth_service", "account"=>"user01", "domain"=>"user01.", "local_http_port"=>"", "metadata"=>{}, "az_name"=>"docker-local-1a"}, FQHN=acmeair-auth-service-d98c30c315.auth-service.local.flyacmeair.net, AZ=docker-local-1a
E, [2014-06-25T21:01:11.243658 #268] ERROR -- : Could not create & launch Docker container: Connection refused - connect(2) -  ["/usr/local/microscaler/lib/im_worker.rb:87:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/job.rb:227:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:250:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:189:in `block in work'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `loop'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `work'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/tasks.rb:41:in `block (2 levels) in <top (required)>'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `call'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `block in execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `each'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:184:in `block in invoke_with_call_chain'", "/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:177:in `invoke_with_call_chain'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:170:in `invoke'", "/usr/local/microscaler/bin/rakefile:26:in `block (2 levels) in <top (required)>'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `call'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `block in execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `each'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:184:in `block in invoke_with_call_chain'", "/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:177:in `invoke_with_call_chain'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:170:in `invoke'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:143:in `invoke_task'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `block (2 levels) in top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `each'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `block in top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:110:in `run_with_threads'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:95:in `top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:73:in `block in run'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:160:in `standard_exception_handling'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:70:in `run'", "/usr/bin/rake:27:in `<main>'"]
D, [2014-06-25T21:02:31.269966 #276] DEBUG -- : deleting lock 58a78a97-a7ee-4ba8-9610-f5f5c50ff83d type=start_type asg=acmeair_webapp
D, [2014-06-25T21:02:31.271264 #276] DEBUG -- : container launched with id=, user metadata={"asg_name"=>"acmeair_webapp", "account"=>"user01", "domain"=>"user01.", "local_http_port"=>"", "metadata"=>{}, "az_name"=>"docker-local-1a"}, FQHN=acmeair-webapp-2ba9252ffe.webapp.local.flyacmeair.net, AZ=docker-local-1a
E, [2014-06-25T21:02:31.271392 #276] ERROR -- : Could not create & launch Docker container: Connection refused - connect(2) -  ["/usr/local/microscaler/lib/im_worker.rb:87:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/job.rb:227:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:250:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:189:in `block in work'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `loop'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `work'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/tasks.rb:41:in `block (2 levels) in <top (required)>'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `call'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `block in execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `each'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:184:in `block in invoke_with_call_chain'", "/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:177:in `invoke_with_call_chain'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:170:in `invoke'", "/usr/local/microscaler/bin/rakefile:26:in `block (2 levels) in <top (required)>'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `call'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `block in execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `each'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:184:in `block in invoke_with_call_chain'", "/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:177:in `invoke_with_call_chain'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:170:in `invoke'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:143:in `invoke_task'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `block (2 levels) in top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `each'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `block in top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:110:in `run_with_threads'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:95:in `top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:73:in `block in run'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:160:in `standard_exception_handling'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:70:in `run'", "/usr/bin/rake:27:in `<main>'"]
D, [2014-06-25T21:02:31.289467 #284] DEBUG -- : deleting lock ee31162f-9370-471f-83de-4f9fc69ff9f9 type=start_type asg=acmeair_auth_service
D, [2014-06-25T21:02:31.290683 #284] DEBUG -- : container launched with id=, user metadata={"asg_name"=>"acmeair_auth_service", "account"=>"user01", "domain"=>"user01.", "local_http_port"=>"", "metadata"=>{}, "az_name"=>"docker-local-1a"}, FQHN=acmeair-auth-service-be0ef7a52a.auth-service.local.flyacmeair.net, AZ=docker-local-1a
E, [2014-06-25T21:02:31.290829 #284] ERROR -- : Could not create & launch Docker container: Connection refused - connect(2) -  ["/usr/local/microscaler/lib/im_worker.rb:87:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/job.rb:227:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:250:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:189:in `block in work'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `loop'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `work'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/tasks.rb:41:in `block (2 levels) in <top (required)>'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `call'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `block in execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `each'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:184:in `block in invoke_with_call_chain'", "/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:177:in `invoke_with_call_chain'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:170:in `invoke'", "/usr/local/microscaler/bin/rakefile:26:in `block (2 levels) in <top (required)>'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `call'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `block in execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `each'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:184:in `block in invoke_with_call_chain'", "/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:177:in `invoke_with_call_chain'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:170:in `invoke'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:143:in `invoke_task'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `block (2 levels) in top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `each'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `block in top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:110:in `run_with_threads'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:95:in `top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:73:in `block in run'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:160:in `standard_exception_handling'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:70:in `run'", "/usr/bin/rake:27:in `<main>'"]
D, [2014-06-25T21:03:31.315464 #292] DEBUG -- : deleting lock f85944ca-cebd-46f6-a1fd-6dde9af11966 type=start_type asg=acmeair_webapp
D, [2014-06-25T21:03:31.316893 #292] DEBUG -- : container launched with id=, user metadata={"asg_name"=>"acmeair_webapp", "account"=>"user01", "domain"=>"user01.", "local_http_port"=>"", "metadata"=>{}, "az_name"=>"docker-local-1a"}, FQHN=acmeair-webapp-f55aef206c.webapp.local.flyacmeair.net, AZ=docker-local-1a
E, [2014-06-25T21:03:31.317029 #292] ERROR -- : Could not create & launch Docker container: Connection refused - connect(2) -  ["/usr/local/microscaler/lib/im_worker.rb:87:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/job.rb:227:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:250:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:189:in `block in work'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `loop'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `work'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/tasks.rb:41:in `block (2 levels) in <top (required)>'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `call'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `block in execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `each'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:184:in `block in invoke_with_call_chain'", "/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:177:in `invoke_with_call_chain'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:170:in `invoke'", "/usr/local/microscaler/bin/rakefile:26:in `block (2 levels) in <top (required)>'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `call'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `block in execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `each'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:184:in `block in invoke_with_call_chain'", "/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:177:in `invoke_with_call_chain'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:170:in `invoke'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:143:in `invoke_task'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `block (2 levels) in top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `each'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `block in top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:110:in `run_with_threads'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:95:in `top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:73:in `block in run'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:160:in `standard_exception_handling'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:70:in `run'", "/usr/bin/rake:27:in `<main>'"]
D, [2014-06-25T21:03:31.334611 #300] DEBUG -- : deleting lock 649394ca-e33d-4bd3-b1f2-f924aef0d8c5 type=start_type asg=acmeair_auth_service
D, [2014-06-25T21:03:31.336610 #300] DEBUG -- : container launched with id=, user metadata={"asg_name"=>"acmeair_auth_service", "account"=>"user01", "domain"=>"user01.", "local_http_port"=>"", "metadata"=>{}, "az_name"=>"docker-local-1a"}, FQHN=acmeair-auth-service-22da5c9e3b.auth-service.local.flyacmeair.net, AZ=docker-local-1a
E, [2014-06-25T21:03:31.336741 #300] ERROR -- : Could not create & launch Docker container: Connection refused - connect(2) -  ["/usr/local/microscaler/lib/im_worker.rb:87:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/job.rb:227:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:250:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:189:in `block in work'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `loop'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `work'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/tasks.rb:41:in `block (2 levels) in <top (required)>'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `call'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `block in execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `each'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:184:in `block in invoke_with_call_chain'", "/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:177:in `invoke_with_call_chain'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:170:in `invoke'", "/usr/local/microscaler/bin/rakefile:26:in `block (2 levels) in <top (required)>'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `call'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `block in execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `each'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:184:in `block in invoke_with_call_chain'", "/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:177:in `invoke_with_call_chain'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:170:in `invoke'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:143:in `invoke_task'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `block (2 levels) in top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `each'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `block in top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:110:in `run_with_threads'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:95:in `top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:73:in `block in run'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:160:in `standard_exception_handling'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:70:in `run'", "/usr/bin/rake:27:in `<main>'"]
D, [2014-06-25T21:04:51.364510 #322] DEBUG -- : deleting lock 16a3e1bc-e972-4c91-8c42-0ca51ba96595 type=start_type asg=acmeair_webapp
D, [2014-06-25T21:04:51.365804 #322] DEBUG -- : container launched with id=, user metadata={"asg_name"=>"acmeair_webapp", "account"=>"user01", "domain"=>"user01.", "local_http_port"=>"", "metadata"=>{}, "az_name"=>"docker-local-1a"}, FQHN=acmeair-webapp-f76c5da1df.webapp.local.flyacmeair.net, AZ=docker-local-1a
E, [2014-06-25T21:04:51.365933 #322] ERROR -- : Could not create & launch Docker container: Connection refused - connect(2) -  ["/usr/local/microscaler/lib/im_worker.rb:87:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/job.rb:227:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:250:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:189:in `block in work'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `loop'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `work'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/tasks.rb:41:in `block (2 levels) in <top (required)>'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `call'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `block in execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `each'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:184:in `block in invoke_with_call_chain'", "/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:177:in `invoke_with_call_chain'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:170:in `invoke'", "/usr/local/microscaler/bin/rakefile:26:in `block (2 levels) in <top (required)>'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `call'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `block in execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `each'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:184:in `block in invoke_with_call_chain'", "/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:177:in `invoke_with_call_chain'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:170:in `invoke'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:143:in `invoke_task'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `block (2 levels) in top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `each'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `block in top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:110:in `run_with_threads'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:95:in `top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:73:in `block in run'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:160:in `standard_exception_handling'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:70:in `run'", "/usr/bin/rake:27:in `<main>'"]
D, [2014-06-25T21:04:51.383173 #330] DEBUG -- : deleting lock 97d9ced8-155f-4aa3-9b79-f41e085cf18f type=start_type asg=acmeair_auth_service
D, [2014-06-25T21:04:51.384408 #330] DEBUG -- : container launched with id=, user metadata={"asg_name"=>"acmeair_auth_service", "account"=>"user01", "domain"=>"user01.", "local_http_port"=>"", "metadata"=>{}, "az_name"=>"docker-local-1a"}, FQHN=acmeair-auth-service-d9953ae32f.auth-service.local.flyacmeair.net, AZ=docker-local-1a
E, [2014-06-25T21:04:51.384537 #330] ERROR -- : Could not create & launch Docker container: Connection refused - connect(2) -  ["/usr/local/microscaler/lib/im_worker.rb:87:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/job.rb:227:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:250:in `perform'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:189:in `block in work'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `loop'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/worker.rb:166:in `work'", "/var/lib/gems/1.9.1/gems/resque-1.25.2/lib/resque/tasks.rb:41:in `block (2 levels) in <top (required)>'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `call'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `block in execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `each'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:184:in `block in invoke_with_call_chain'", "/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:177:in `invoke_with_call_chain'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:170:in `invoke'", "/usr/local/microscaler/bin/rakefile:26:in `block (2 levels) in <top (required)>'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `call'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:246:in `block in execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `each'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:241:in `execute'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:184:in `block in invoke_with_call_chain'", "/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:177:in `invoke_with_call_chain'", "/usr/lib/ruby/vendor_ruby/rake/task.rb:170:in `invoke'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:143:in `invoke_task'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `block (2 levels) in top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `each'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:101:in `block in top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:110:in `run_with_threads'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:95:in `top_level'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:73:in `block in run'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:160:in `standard_exception_handling'", "/usr/lib/ruby/vendor_ruby/rake/application.rb:70:in `run'", "/usr/bin/rake:27:in `<main>'"]

Is microscaler having trouble connecting to dockerd? Is this a permission problem? It may be relevant that my host system is Ubuntu 13.10 not boot2docker (for performance I am running on a real server not a VM).

tkhinaba commented 10 years ago

The cause would be your Docker daemon does not listen TCP socket. The launcher reports connection refused by Docker daemon and your docker process is not given -H option. Please see Enable remote API access of Docker daemon via TCP socket.

tkhinaba commented 10 years ago

Easy check Docker daemon is available via TCP

$ curl localhost:2375/_ping
OK
thewmf commented 10 years ago

I didn't realize that enabling remote access was necessary for autoscaling. This is obvious in hindsight.

It starts properly now.

tkhinaba commented 10 years ago

That's good.