New host instances can't start tasks using ECS ami-3fc1b928

[dextervip]

I have been using weave with three hosts however two of these instances are spot. They go down and back up few times but lately, all tasks in these instances are started but they are not fully initialized, simply there is no logs for these tasks.

Checking weave container logs I could see, It is trying to connect a dead instance. Any ideias what could be wrong or how to solve it?

ca420ecc9f70 weaveworks/weave:1.6.2 "/home/weave/weaver -" 10 minutes ago Up 10 minutes weave [ec2-user@ip-10-0-102-15 ~]$ docker logs ca420ecc9f70 INFO: 2016/10/26 20:42:23.109116 Command line options: map[dns-effective-listen-address:172.17.0.1 dns-listen-address:172.17.0.1:53 http-addr:127.0.0.1:6784 ipalloc-ran ge:10.32.0.0/12 port:6783 resolv-conf:/var/run/weave/etc/resolv.conf datapath:datapath name:4a:f3:9e:4b:61:a2 nickname:ip-10-0-102-15] INFO: 2016/10/26 20:42:23.114180 Communication between peers is unencrypted. INFO: 2016/10/26 20:42:23.125634 Our name is 4a:f3:9e:4b:61:a2(ip-10-0-102-15) INFO: 2016/10/26 20:42:23.125689 Launch detected - using supplied peer list: [10.0.101.5 10.0.102.7] INFO: 2016/10/26 20:42:23.128557 Docker API on unix:///var/run/docker.sock: &[GoVersion=go1.5.3 Os=linux Arch=amd64 KernelVersion=4.4.19-29.55.amzn1.x86_64 Version=1.11 .2 ApiVersion=1.23 GitCommit=b9f10c9/1.11.2] INFO: 2016/10/26 20:42:23.129696 [allocator 4a:f3:9e:4b:61:a2] No valid persisted data INFO: 2016/10/26 20:42:23.152445 [allocator 4a:f3:9e:4b:61:a2] Initialising via deferred consensus INFO: 2016/10/26 20:42:23.164245 Listening for DNS queries on 172.17.0.1 INFO: 2016/10/26 20:42:23.164352 Sniffing traffic on datapath (via ODP) INFO: 2016/10/26 20:42:23.169327 Listening for HTTP control messages on 127.0.0.1:6784 INFO: 2016/10/26 20:42:23.200582 ->[10.0.101.5:6783] attempting connection INFO: 2016/10/26 20:42:23.200740 ->[10.0.102.7:6783] attempting connection INFO: 2016/10/26 20:42:23.205764 ->[10.0.102.7:6783] error during connection attempt: dial tcp4 :0->10.0.102.7:6783: getsockopt: connection refused INFO: 2016/10/26 20:42:23.207641 ->[10.0.101.5:6783|86:7b:00:0c:f2:8e(ip-10-0-101-5)]: connection ready; using protocol version 2 INFO: 2016/10/26 20:42:23.207812 overlay_switch ->[86:7b:00:0c:f2:8e(ip-10-0-101-5)] using fastdp INFO: 2016/10/26 20:42:23.207968 ->[10.0.101.5:6783|86:7b:00:0c:f2:8e(ip-10-0-101-5)]: connection added (new peer) INFO: 2016/10/26 20:42:23.221856 EMSGSIZE on send, expecting PMTU update (IP packet was 60028 bytes, payload was 60020 bytes) INFO: 2016/10/26 20:42:23.222017 overlay_switch ->[86:7b:00:0c:f2:8e(ip-10-0-101-5)] using sleeve INFO: 2016/10/26 20:42:23.222091 ->[10.0.101.5:6783|86:7b:00:0c:f2:8e(ip-10-0-101-5)]: connection fully established INFO: 2016/10/26 20:42:23.222286 overlay_switch ->[86:7b:00:0c:f2:8e(ip-10-0-101-5)] using fastdp INFO: 2016/10/26 20:42:23.223776 sleeve ->[10.0.101.5:6783|86:7b:00:0c:f2:8e(ip-10-0-101-5)]: Effective MTU verified at 8939 INFO: 2016/10/26 20:42:23.834597 Weave version 1.7.2 is available; please update at https://github.com/weaveworks/weave/releases/download/v1.7.2/weave INFO: 2016/10/26 20:42:24.823201 ->[10.0.102.7:6783] attempting connection INFO: 2016/10/26 20:42:24.823539 ->[10.0.102.7:6783] error during connection attempt: dial tcp4 :0->10.0.102.7:6783: getsockopt: connection refused INFO: 2016/10/26 20:42:29.256167 ->[10.0.102.7:6783] attempting connection INFO: 2016/10/26 20:42:29.256576 ->[10.0.102.7:6783] error during connection attempt: dial tcp4 :0->10.0.102.7:6783: getsockopt: connection refused INFO: 2016/10/26 20:42:35.863440 ->[10.0.102.7:6783] attempting connection INFO: 2016/10/26 20:42:35.863976 ->[10.0.102.7:6783] error during connection attempt: dial tcp4 :0->10.0.102.7:6783: getsockopt: connection refused INFO: 2016/10/26 20:42:42.595814 ->[10.0.102.7:6783] attempting connection INFO: 2016/10/26 20:42:42.596230 ->[10.0.102.7:6783] error during connection attempt: dial tcp4 :0->10.0.102.7:6783: getsockopt: connection refused INFO: 2016/10/26 20:42:49.290667 ->[10.0.102.7:6783] attempting connection INFO: 2016/10/26 20:42:49.291078 ->[10.0.102.7:6783] error during connection attempt: dial tcp4 :0->10.0.102.7:6783: getsockopt: connection refused INFO: 2016/10/26 20:43:08.272476 ->[10.0.102.7:6783] attempting connection INFO: 2016/10/26 20:43:08.272786 ->[10.0.102.7:6783] error during connection attempt: dial tcp4 :0->10.0.102.7:6783: getsockopt: connection refused INFO: 2016/10/26 20:43:33.728925 ->[10.0.102.7:6783] attempting connection INFO: 2016/10/26 20:43:33.729304 ->[10.0.102.7:6783] error during connection attempt: dial tcp4 :0->10.0.102.7:6783: getsockopt: connection refused INFO: 2016/10/26 20:44:17.556602 ->[10.0.102.7:6783] attempting connection INFO: 2016/10/26 20:44:17.569108 ->[10.0.102.7:6783] error during connection attempt: dial tcp4 :0->10.0.102.7:6783: getsockopt: connection refused INFO: 2016/10/26 20:45:28.611499 ->[10.0.102.7:6783] attempting connection INFO: 2016/10/26 20:45:28.612065 ->[10.0.102.7:6783] error during connection attempt: dial tcp4 :0->10.0.102.7:6783: getsockopt: connection refused INFO: 2016/10/26 20:47:03.677807 Discovered remote MAC c2:02:40:94:6c:05 at 86:7b:00:0c:f2:8e(ip-10-0-101-5) INFO: 2016/10/26 20:47:06.797489 ->[10.0.102.7:6783] attempting connection INFO: 2016/10/26 20:47:06.798023 ->[10.0.102.7:6783] error during connection attempt: dial tcp4 :0->10.0.102.7:6783: getsockopt: connection refused INFO: 2016/10/26 20:48:17.435153 ->[10.0.102.7:6783] attempting connection INFO: 2016/10/26 20:48:17.435563 ->[10.0.102.7:6783] error during connection attempt: dial tcp4 :0->10.0.102.7:6783: getsockopt: connection refused INFO: 2016/10/26 20:50:12.123765 ->[10.0.102.7:6783] attempting connection INFO: 2016/10/26 20:50:12.124367 ->[10.0.102.7:6783] error during connection attempt: dial tcp4 :0->10.0.102.7:6783: getsockopt: connection refused INFO: 2016/10/26 20:51:00.419966 [allocator]: Allocate request for 2f52bd30d5c49116a8f62bb789f08e5742a9d46cf1f79b1b474c91ccbc480322 cancelled

[bboreham]

Hi @dextervip!

The peer list 10.0.101.5 10.0.102.7: were those, at the time of launch, running Weave Net?

The logs indicate that they refused connection.

[bboreham]

Also, can you run weave status on the node where you have tasks that won't start.