Listener crashing on state change

[sivaramsk]

I see listener crashing when the updater propagates state change to the client

devel@devel:~$ docker run --net=host bobrik/marathoner-listener:1.8 -u 16.2.1.1:7676 -b 127.0.0.1 2015/05/28 06:31:11 dial succeeded 16.2.1.1:7676 2015/05/28 06:31:11 received update request panic: runtime error: index out of range

goroutine 7 [running]: github.com/bobrik/marathoner.stateToApps(0xc20806fb00, 0x1) /go/src/github.com/bobrik/marathoner/haproxy.go:242 +0x59c github.com/bobrik/marathoner.(_HaproxyConfigurator).update(0xc20800c1e0, 0xc20806fb00, 0xc20802bb00, 0x0, 0x0) /go/src/github.com/bobrik/marathoner/haproxy.go:82 +0x168 github.com/bobrik/marathoner.(_HaproxyConfigurator).Update(0xc20800c1e0, 0xc20806fb00, 0xc20802bb00, 0x0, 0x0) /go/src/github.com/bobrik/marathoner/haproxy.go:67 +0x65 github.com/bobrik/marathoner.(_Configurator).Update(0xc20802b300, 0xc20806fb00, 0xc20802bb00, 0x0, 0x0) /go/src/github.com/bobrik/marathoner/configurator.go:16 +0x66 reflect.Value.call(0x6f95a0, 0x70bc80, 0x13, 0x78e210, 0x4, 0xc208030f28, 0x3, 0x3, 0x0, 0x0, ...) /usr/src/go/src/reflect/value.go:419 +0x10e5 reflect.Value.Call(0x6f95a0, 0x70bc80, 0x13, 0xc208030f28, 0x3, 0x3, 0x0, 0x0, 0x0) /usr/src/go/src/reflect/value.go:296 +0xbc net/rpc.(_service).call(0xc208057b80, 0xc208057b40, 0xc20802b330, 0xc20808c400, 0xc20801f0c0, 0x6f2ca0, 0xc2080383a0, 0xd5, 0x679340, 0xc20802bb00, ...) /usr/src/go/src/net/rpc/server.go:382 +0x1f7 created by net/rpc.(*Server).ServeCodec /usr/src/go/src/net/rpc/server.go:476 +0x44a

goroutine 1 [IO wait]: net.(_pollDesc).Wait(0xc208010bc0, 0x72, 0x0, 0x0) /usr/src/go/src/net/fd_poll_runtime.go:84 +0x47 net.(_pollDesc).WaitRead(0xc208010bc0, 0x0, 0x0) /usr/src/go/src/net/fd_poll_runtime.go:89 +0x43 net.(_netFD).Read(0xc208010b60, 0xc20807c000, 0x1000, 0x1000, 0x0, 0x7fbcb1ea3cf0, 0xc20802bb08) /usr/src/go/src/net/fd_unix.go:242 +0x40f net.(_conn).Read(0xc208038110, 0xc20807c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/src/go/src/net/net.go:121 +0xdc bufio.(_Reader).fill(0xc20800c2a0) /usr/src/go/src/bufio/bufio.go:97 +0x1ce bufio.(_Reader).Read(0xc20800c2a0, 0xc20802b310, 0x1, 0x9, 0xc20803af50, 0x0, 0x0) /usr/src/go/src/bufio/bufio.go:174 +0x26c io.ReadAtLeast(0x7fbcb1ea5580, 0xc20800c2a0, 0xc20802b310, 0x1, 0x9, 0x1, 0x0, 0x0, 0x0) /usr/src/go/src/io/io.go:298 +0xf1 io.ReadFull(0x7fbcb1ea5580, 0xc20800c2a0, 0xc20802b310, 0x1, 0x9, 0x749620, 0x0, 0x0) /usr/src/go/src/io/io.go:316 +0x6d encoding/gob.decodeUintReader(0x7fbcb1ea5580, 0xc20800c2a0, 0xc20802b310, 0x9, 0x9, 0x0, 0x1, 0x0, 0x0) /usr/src/go/src/encoding/gob/decode.go:121 +0x99 encoding/gob.(_Decoder).recvMessage(0xc20808c480, 0xc208000120) /usr/src/go/src/encoding/gob/decoder.go:76 +0x55 encoding/gob.(_Decoder).decodeTypeSequence(0xc20808c480, 0xc2080b8200, 0x16) /usr/src/go/src/encoding/gob/decoder.go:140 +0x47 encoding/gob.(_Decoder).DecodeValue(0xc20808c480, 0x680720, 0xc2080b8220, 0x16, 0x0, 0x0) /usr/src/go/src/encoding/gob/decoder.go:208 +0x192 encoding/gob.(_Decoder).Decode(0xc20808c480, 0x680720, 0xc2080b8220, 0x0, 0x0) /usr/src/go/src/encoding/gob/decoder.go:185 +0x297 net/rpc.(_gobServerCodec).ReadRequestHeader(0xc20806ea80, 0xc2080b8220, 0x0, 0x0) /usr/src/go/src/net/rpc/server.go:402 +0x5e net/rpc.(_Server).readRequestHeader(0xc208057b40, 0x7fbcb1ea55d0, 0xc20806ea80, 0x0, 0x0, 0xc2080b8220, 0xc20802b300, 0x0, 0x0) /usr/src/go/src/net/rpc/server.go:575 +0x97 net/rpc.(_Server).readRequest(0xc208057b40, 0x7fbcb1ea55d0, 0xc20806ea80, 0xc20808c400, 0xc20801f0c0, 0x6f2ca0, 0x0, 0x0, 0x0, 0x0, ...) /usr/src/go/src/net/rpc/server.go:542 +0xa2 net/rpc.(_Server).ServeCodec(0xc208057b40, 0x7fbcb1ea55d0, 0xc20806ea80) /usr/src/go/src/net/rpc/server.go:461 +0x6d net/rpc.(_Server).ServeConn(0xc208057b40, 0x7fbcb1ea54d0, 0xc208038110) /usr/src/go/src/net/rpc/server.go:453 +0x208 github.com/bobrik/marathoner.(_Listener).Start(0xc20806e8a0) /go/src/github.com/bobrik/marathoner/listener.go:42 +0x2e8 main.main() /go/src/github.com/bobrik/marathoner/cmd/listener/main.go:39 +0x64d

[bobrik]

Can you show me what state logger sees at the same moment?

[sivaramsk]

devel@devel:~$ docker run --rm bobrik/marathoner-logger:1.7 -u 16.2.1.1:7676 2015/05/28 06:23:19 dial succeeded 16.2.1.1:7676 2015-05-28T06:23:19.638523004Z: {"/busybox-port-8082-1":{"Name":"/busybox-port-8082-1","Labels":{"marathoner_haproxy_enabled":"true"},"Ports":[6379,11001],"Tasks":[{"ID":"busybox-port-8082-1.70df86bc-0475-11e5-9872-6229af2a64b1","Host":"16.1.2.1","Ports":[31000],"StagedAt":"2015-05-27T13:36:55.607Z","StartedAt":"2015-05-27T13:36:56.169Z"}]},"/busybox-port-8082-2":{"Name":"/busybox-port-8082-2","Labels":{"marathoner_haproxy_enabled":"true"},"Ports":[6379,11002],"Tasks":[{"ID":"busybox-port-8082-2.6fb389db-0475-11e5-9872-6229af2a64b1","Host":"16.2.2.1","Ports":[31000],"StagedAt":"2015-05-27T13:36:53.694Z","StartedAt":"2015-05-27T13:36:54.256Z"}]},"/busybox-port-8082-3":{"Name":"/busybox-port-8082-3","Labels":{"marathoner_haproxy_enabled":"true"},"Ports":[6379,11003],"Tasks":[{"ID":"busybox-port-8082-3.7342a40d-0475-11e5-9872-6229af2a64b1","Host":"16.2.2.1","Ports":[31001],"StagedAt":"2015-05-27T13:36:59.611Z","StartedAt":"2015-05-27T13:37:00.145Z"}]}}

[bobrik]

Your apps have two ports, but tasks only have one. Marathoner doesn't support that.

[sivaramsk]

Sorry. I misunderstood the ports tag. Listener comes up and the ports are reachable.

[sivaramsk]

Just to clarify, the ports defined in ports:[0] is what updater picks for the listerner to create the haproxy binding. Is it right? I defined the ports in my json based on this link https://mesosphere.github.io/marathon/docs/rest-api.html#post-/v2/apps

In the above link, the containerport and serviceport combination is defined under ports tag. Isn't that the way to define the ports?

[bobrik]

In container.docker.portMappings:

            portMappings:
              - containerPort: 80
                servicePort: 17002

Plus ports:

        ports:
          - 17002

That would create haproxy on 127.0.0.1:17002 (or any other interface you specify).

With ports: [0], random port would be assigned on haproxy, this is probably not what you want.

[sivaramsk]

That clarifies it. Thanks a lot.