Closed wegel closed 7 years ago
heyitsanthony
commented
7 years ago @wegel The bridge mode will only proxy requests from the client to an actual zk server (this is useful for making sure that zetcd's protocol serialization code works correctly, but not much more). The -oracle flag is needed for cross-checking between the emulated (candidate) responses and responses from an actual (oracle) zk server.
Will investigate. Thanks for your patience with this mesos bootstrapping.
wegel
commented
7 years ago My pleasure; please tell me if there's anything I can do to facilitate your investigation or help in general.
heyitsanthony
commented
7 years ago Hi @wegel, sorry for the delay. I tried mesos 1.1.0 and was able to get a slave to register, resume, and survive a master reboot using the procfile:
I1202 06:41:35.333648 17273 slave.cpp:533] Agent resources: cpus(*):8; mem(*):14999; disk(*):942058; ports(*):[31000-32000]
I1202 06:41:35.333703 17273 slave.cpp:541] Agent attributes: [ ]
I1202 06:41:35.333716 17273 slave.cpp:546] Agent hostname: localhost
I1202 06:41:35.337188 17275 state.cpp:57] Recovering state from '/var/tmp/mesos/meta'
I1202 06:41:35.337268 17275 state.cpp:698] No committed checkpointed resources found at '/var/tmp/mesos/meta/resources/resources.info'
I1202 06:41:35.337889 17276 status_update_manager.cpp:203] Recovering status update manager
I1202 06:41:35.338038 17273 docker.cpp:764] Recovering Docker containers
I1202 06:41:35.338146 17272 containerizer.cpp:555] Recovering containerizer
I1202 06:41:35.340489 17273 provisioner.cpp:253] Provisioner recovery complete
2016-12-02 06:41:35,344:17196(0x7f04cbfff700):ZOO_INFO@check_events@1775: session establishment complete on server [127.0.0.1:2181], sessionId=0x158be3d664a0003, negotiated timeout=10000
I1202 06:41:35.344843 17273 group.cpp:340] Group process (zookeeper-group(1)@127.0.0.1:5051) connected to ZooKeeper
I1202 06:41:35.344879 17273 group.cpp:828] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
I1202 06:41:35.344894 17273 group.cpp:418] Trying to create path '/mesos' in ZooKeeper
I1202 06:41:35.349282 17273 detector.cpp:152] Detected a new leader: (id='0')
I1202 06:41:35.349453 17273 group.cpp:697] Trying to get '/mesos/json.info_0000000000' in ZooKeeper
I1202 06:41:35.352407 17276 zookeeper.cpp:259] A new leading master (UPID=master@127.0.0.1:5050) is detected
I1202 06:41:35.430510 17274 slave.cpp:5281] Finished recovery
I1202 06:41:35.430892 17274 slave.cpp:915] New master detected at master@127.0.0.1:5050
I1202 06:41:35.430891 17272 status_update_manager.cpp:177] Pausing sending status updates
I1202 06:41:35.430912 17274 slave.cpp:936] No credentials provided. Attempting to register without authentication
I1202 06:41:35.430979 17274 slave.cpp:947] Detecting new master
I1202 06:41:35.495085 17274 slave.cpp:1217] Re-registered with master master@127.0.0.1:5050
I1202 06:41:35.495152 17274 slave.cpp:1253] Forwarding total oversubscribed resources {}
I1202 06:41:35.495160 17276 status_update_manager.cpp:184] Resuming sending status updates
I1202 06:42:35.334143 17274 slave.cpp:5044] Current disk usage 78.89%. Max allowed age: 18.657411120295556hrs
I1202 06:43:12.884860 17276 slave.cpp:4179] Got exited event for master@127.0.0.1:5050
W1202 06:43:12.884918 17276 slave.cpp:4184] Master disconnected! Waiting for a new master to be elected
I1202 06:43:22.313520 17277 detector.cpp:152] Detected a new leader: None
I1202 06:43:22.313650 17277 slave.cpp:908] Lost leading master
I1202 06:43:22.313666 17277 slave.cpp:947] Detecting new master
I1202 06:43:22.313693 17277 status_update_manager.cpp:177] Pausing sending status updates
I1202 06:43:29.725848 17274 detector.cpp:152] Detected a new leader: (id='1')
I1202 06:43:29.726013 17275 group.cpp:697] Trying to get '/mesos/json.info_0000000001' in ZooKeeper
I1202 06:43:29.734158 17276 zookeeper.cpp:259] A new leading master (UPID=master@127.0.0.1:5050) is detected
I1202 06:43:29.734261 17276 status_update_manager.cpp:177] Pausing sending status updates
I1202 06:43:29.734302 17275 slave.cpp:915] New master detected at master@127.0.0.1:5050
I1202 06:43:29.734338 17275 slave.cpp:936] No credentials provided. Attempting to register without authentication
I1202 06:43:29.734359 17275 slave.cpp:947] Detecting new master
I1202 06:43:30.174283 17276 slave.cpp:1217] Re-registered with master master@127.0.0.1:5050
I1202 06:43:30.174335 17276 slave.cpp:1253] Forwarding total oversubscribed resources {}
I1202 06:43:30.174350 17273 status_update_manager.cpp:184] Resuming sending status updates
I1202 06:43:35.335189 17277 slave.cpp:5044] Current disk usage 78.89%. Max allowed age: 18.657397263350557hrsI'm not a mesos expert so maybe this doesn't matter, but some things stand out from your logs.
The master appears to contact two separate zk servers:
2016-11-18 21:29:46,493:76(0x7f73b6c24700):ZOO_INFO@check_events@1775: session establishment complete on server [172.16.88.196:2181], sessionId=0x4cfa587954761aab, negotiated timeout=10000
2016-11-18 21:29:46,493:76(0x7f73b6423700):ZOO_INFO@check_events@1775: session establishment complete on server [172.16.88.196:2181], sessionId=0x4cfa587954761aad, negotiated timeout=10000
I1118 21:29:46.493445 90 group.cpp:349] Group process (group(3)@172.16.102.133:5050) connected to ZooKeeper
I1118 21:29:46.493461 90 group.cpp:837] Syncing group operations: queue size (joins, cancels, datas) = (1, 0, 0)
2016-11-18 21:29:46,493:76(0x7f73affff700):ZOO_INFO@check_events@1775: session establishment complete on server [172.16.52.4:2181], sessionId=0x6bf85879547643ef, negotiated timeout=10000
2016-11-18 21:29:46,493:76(0x7f73b4c20700):ZOO_INFO@check_events@1775: session establishment complete on server [172.16.52.4:2181], sessionId=0x6bf85879547643f1, negotiated timeout=10000The slave only tries to connect to one of the zk instances:
2016-11-18 21:45:16,711:49(0x7f049fdf9700):ZOO_INFO@check_events@1728: initiated connection to server [172.16.52.4:2181]
I1118 21:45:16.711513 60 provisioner.cpp:253] Provisioner recovery complete
2016-11-18 21:45:16,717:49(0x7f049fdf9700):ZOO_INFO@check_events@1775: session establishment complete on server [172.16.52.4:2181], sessionId=0x6bf85879547673af, negotiated timeout=10000There's more than one master ID:
I1118 21:29:46.497112 92 group.cpp:706] Trying to get '/mesos/json.info_0000000001' in ZooKeeper
I1118 21:29:46.499739 86 zookeeper.cpp:259] A new leading master (UPID=master@172.16.102.133:5050) is detected
I1118 21:29:46.499851 86 master.cpp:1847] The newly elected leader is master@172.16.102.133:5050 with id 01d0998e-8d9b-4791-af19-227693b85d0c
I1118 21:29:46.504482 93 contender.cpp:268] New candidate (id='4') has entered the contest for leadership
E1118 21:29:48.699439 94 process.cpp:2022] Failed to shutdown socket with fd 23: Transport endpoint is not connected
I1118 21:29:55.822985 90 detector.cpp:152] Detected a new leader: (id='4')
I1118 21:29:55.823264 89 group.cpp:706] Trying to get '/mesos/json.info_0000000004' in ZooKeeper
I1118 21:29:55.830425 91 zookeeper.cpp:259] A new leading master (UPID=master@172.16.102.133:5050) is detected
I1118 21:29:55.830564 89 master.cpp:1847] The newly elected leader is master@172.16.102.133:5050 with id 9f8b1b5c-c8d4-4e4b-bbd0-c03213ab4385Are you starting with clean mesos directories? It looks like the configuration is really confused.
wegel
commented
7 years ago Thanks for your work @heyitsanthony. Indeed, my test setup is a lot more "real-life" than what is currently in the xchk file; the purpose of zookeeper is to coordinate a cluster of masters and slaves, and thus testing zetcd with a single master and a single slave kind of misses the point.
My test setup typically consists of a 'master' cluster of 3 machines on which etcd, zetcd and mesos-master runs, and another 'worker' cluster of 3 machines on which mesos-slave runs.
I'll see if I can build a xchk that reproduces the problem. Thanks!
heyitsanthony
commented
7 years ago OK, but what about it trying to connect to multiple ZK servers? That seems like a misconfiguration if not an intentional redundancy setup.
xiang90
commented
7 years ago multiple ZK servers
To be more clear, I believe @heyitsanthony means that the mesos setup tried to contact two completely independent ZooKeeper clusters.
wegel
commented
7 years ago OK, but what about it trying to connect to multiple ZK servers? That seems like a misconfiguration if not an intentional redundancy setup.
It is indeed an intentional redundancy setup; as I said, one zookeeper per 'master' machine. This is how zookeeper is used in the field pretty much everywhere, since it's a distributed (and thus, multi-machine) key-value store. zetcd has to support this correctly in order to replace zookeeper 1:1. Since zetcd is backed by etcd, this shouldn't pose any problem, in any case.
I'm not totally excluding that there might be a problem with my setup, though it works correctly when I use zookeeper instead of zetcd.
I'll work on a xchk file which replicates my setup more faithfully and hopefully it'll show the problem if there's one. Thanks! I can't wait to ditch zookeeper and use zetcd in production instead ;)
wegel
commented
7 years ago So I've been able to replicate the problem in a, err, replicable manner :p
I've tested using docker-compose, since I wanted the containers to be able to address themselves by names and I didn't want to manage a docker network by myself. The environment that's started by docker-compose is pretty similar to what I'm running in production.
$ cat docker-compose.yml
version: '2'
networks:
cluster-net:
services:
etcd:
image: quay.io/coreos/etcd
container_name: mesos-test-etcd
command: >
etcd --name etcd0
--advertise-client-urls http://etcd:2379,http://etcd:4001
--listen-client-urls http://0.0.0.0:2379,http://0.0.0.0:4001
--initial-advertise-peer-urls http://etcd:2380
--listen-peer-urls http://0.0.0.0:2380
--initial-cluster-token etcd-cluster-1
--initial-cluster etcd0=http://etcd:2380
--initial-cluster-state new
networks:
- cluster-net
zookeeper:
image: zookeeper:3.4.9
networks:
- cluster-net
zetcd-1:
build: .
command: /zetcd -endpoint http://etcd:2379 -zkaddr zetcd:2181 ${ZETCD_OPTS} -logtostderr -v 10
networks:
cluster-net:
aliases:
- zetcd
zetcd-2:
build: .
command: /zetcd -endpoint http://etcd:2379 -zkaddr zetcd:2181 ${ZETCD_OPTS} -logtostderr -v 10
networks:
cluster-net:
aliases:
- zetcd
zetcd-3:
build: .
command: /zetcd -endpoint http://etcd:2379 -zkaddr zetcd:2181 ${ZETCD_OPTS} -logtostderr -v 10
networks:
cluster-net:
aliases:
- zetcd
mesos-master-1:
image: mesosphere/mesos-master:1.0.1-2.0.93.ubuntu1404
environment:
- MESOS_PORT=5050
- MESOS_ZK=zk://${ZK}:2181/mesos
restart: unless-stopped
networks:
cluster-net:
aliases:
- mesos-master
mesos-master-2:
image: mesosphere/mesos-master:1.0.1-2.0.93.ubuntu1404
environment:
- MESOS_PORT=5050
- MESOS_ZK=zk://${ZK}:2181/mesos
restart: unless-stopped
networks:
cluster-net:
aliases:
- mesos-master
mesos-master-3:
image: mesosphere/mesos-master:1.0.1-2.0.93.ubuntu1404
environment:
- MESOS_PORT=5050
- MESOS_ZK=zk://${ZK}:2181/mesos
restart: unless-stopped
networks:
cluster-net:
aliases:
- mesos-master
mesos-slave-1:
image: mesosphere/mesos-slave-dind:0.2.4_mesos-1.0.1_docker-1.12.1_ubuntu-14.04.5
privileged: true
environment:
- MESOS_CONTAINERIZERS=docker,mesos
- MESOS_WORK_DIR=/data
- MESOS_MASTER=zk://${ZK}:2181/mesos
- MESOS_DOCKER_STOP_TIMEOUT=2mins
- MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins
restart: unless-stopped
networks:
cluster-net:
aliases:
- mesos-slave
mesos-slave-2:
image: mesosphere/mesos-slave-dind:0.2.4_mesos-1.0.1_docker-1.12.1_ubuntu-14.04.5
privileged: true
environment:
- MESOS_CONTAINERIZERS=docker,mesos
- MESOS_WORK_DIR=/data
- MESOS_MASTER=zk://${ZK}:2181/mesos
- MESOS_DOCKER_STOP_TIMEOUT=2mins
- MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins
restart: unless-stopped
networks:
cluster-net:
aliases:
- mesos-slave
mesos-slave-3:
image: mesosphere/mesos-slave-dind:0.2.4_mesos-1.0.1_docker-1.12.1_ubuntu-14.04.5
privileged: true
environment:
- MESOS_CONTAINERIZERS=docker,mesos
- MESOS_WORK_DIR=/data
- MESOS_MASTER=zk://${ZK}:2181/mesos
- MESOS_DOCKER_STOP_TIMEOUT=2mins
- MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins
restart: unless-stopped
networks:
cluster-net:
aliases:
- mesos-slaveYou then need to call docker-compose as such:
ZK=zetcd docker-compose up
ZK can be either 'zetcd' or 'zookeeper'. 'zookeeper' will use the native zookeeper container directly, showing that everything works with it. If you want to add a zkbridge to the zetcd containers, you can do:
ZK=zetcd ZETCD_OPTS="-zkbridge zookeeper:2181 -oracle zk" docker-compose up
When I run the above, I get a bunch of:
I1207 19:44:24.447043 1 server.go:110] zkreq={xid:-2 req:*zetcd.PingRequest:&{}}
I1207 19:44:24.447176 1 zklog.go:73] Ping(-2,{})
I1207 19:44:24.447719 1 session.go:116] waitFutureSendResp=&{}
I1207 19:44:24.447529 1 zklog.go:73] Ping(-2,{})
I1207 19:44:24.448823 1 session.go:136] zk/zkresp=({Xid:-2 Zxid:14 Err:0},&{})
W1207 19:44:24.448912 1 zk.go:324] xchk failed (zxid mismatch)
candidate: &{Xid:-2 Zxid:22 Err:0}
oracle: &{Xid:-2 Zxid:14 Err:0}
I1207 19:44:24.448985 1 conn.go:131] sendXchk Xid:-2 ZXid:14 Resp:&{}
I1207 19:44:24.449456 1 conn.go:133] conn.Send(xid=-2, zxid=14, &{})I built a very simplisitic Dockerfile for zetcd (which needs to be in the same directory as docker-compose.yml because of the "build: ." line) which consits of:
FROM scratch
ADD zetcd /And I'm building zetcd statically this way:
CGO_ENABLED=0 GOOS=linux go build -ldflags "-s" -a -installsuffix cgo cmd/zetcd/zetcd.goHope this is useful. Please tell me if I can do anything to help.
heyitsanthony
commented
7 years ago @wegel thank you for the docker compose file; it made this much easier to test. Apologies again for letting this sit around; there is only so much time in a day...
Would you mind giving this another spin, if you get the chance? I tried it against the latest patches which fix some stat issues related to #11 (much easier to debug than a full mesos cluster) and it seems to work.
I'm still getting xchk errors on boot but I believe they're benign; zk lists paths in an unusual order on GetChildren (zetcd sorts), and there's some raciness in the xchker w/r/t zxids and session establishment/revocation which won't give a deterministic ordering (although the ordering will still be a safe/feasible ordering) causing ping mismatches. Aside from that, the logs appear to be fine. Is there a command I can issue to confirm it's working OK?
I see (zetcd stuff grep -v'd out):
mesos-slave-2_1 | I0123 04:51:07.466876 170 containerizer.cpp:522] Recovering containerizer
mesos-slave-3_1 | I0123 04:51:05.287315 163 docker.cpp:775] Recovering Docker containers
mesos-slave-3_1 | I0123 04:51:05.287329 166 containerizer.cpp:522] Recovering containerizer
mesos-slave-2_1 | I0123 04:51:07.468099 168 provisioner.cpp:253] Provisioner recovery complete
mesos-slave-3_1 | I0123 04:51:05.288717 169 provisioner.cpp:253] Provisioner recovery complete
mesos-slave-2_1 | 2017-01-23 04:51:07,519:1(0x7f8aebbea700):ZOO_INFO@check_events@1728: initiated connection to server [172.18.0.2:2181]
mesos-slave-3_1 | 2017-01-23 04:51:05,302:1(0x7f1bf30fa700):ZOO_INFO@check_events@1728: initiated connection to server [172.18.0.2:2181]
mesos-slave-3_1 | I0123 04:51:05.379009 163 slave.cpp:4782] Finished recovery
mesos-slave-2_1 | I0123 04:51:07.558528 169 slave.cpp:4782] Finished recovery
mesos-slave-3_1 | 2017-01-23 04:51:06,885:1(0x7f1bf30fa700):ZOO_INFO@check_events@1775: session establishment complete on server [172.18.0.2:2181], sessionId=0x159c9a9a1720006, negotiated timeout=10000
mesos-slave-3_1 | I0123 04:51:06.885176 166 group.cpp:349] Group process (group(1)@172.18.0.3:5051) connected to ZooKeeper
mesos-slave-3_1 | I0123 04:51:06.885206 166 group.cpp:837] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
mesos-slave-3_1 | I0123 04:51:06.885215 166 group.cpp:427] Trying to create path '/mesos' in ZooKeeper
mesos-slave-3_1 | I0123 04:51:07.333108 169 detector.cpp:152] Detected a new leader: (id='0')
mesos-slave-3_1 | I0123 04:51:07.333237 164 group.cpp:706] Trying to get '/mesos/json.info_0000000000' in ZooKeeper
mesos-slave-3_1 | I0123 04:51:07.361497 164 zookeeper.cpp:259] A new leading master (UPID=master@172.18.0.7:5050) is detected
mesos-slave-3_1 | I0123 04:51:07.361596 168 status_update_manager.cpp:174] Pausing sending status updates
mesos-slave-3_1 | I0123 04:51:07.361621 165 slave.cpp:895] New master detected at master@172.18.0.7:5050
mesos-slave-3_1 | I0123 04:51:07.361656 165 slave.cpp:916] No credentials provided. Attempting to register without authentication
mesos-slave-3_1 | I0123 04:51:07.361668 165 slave.cpp:927] Detecting new master
mesos-slave-1_1 | Writing inode tables: done
mesos-slave-1_1 | Creating journal (32768 blocks): done
mesos-master-3_1 | I0123 04:51:07.942277 16 master.cpp:4550] Registering agent at slave(1)@172.18.0.3:5051 (3b259db8e7e1) with id e04535fe-b44a-4b8e-9e58-bcbe1e42920b-S0
mesos-master-3_1 | I0123 04:51:07.942420 18 registrar.cpp:464] Applied 1 operations in 25565ns; attempting to update the 'registry'
mesos-master-3_1 | I0123 04:51:07.943120 13 registrar.cpp:509] Successfully updated the 'registry' in 669952ns
mesos-master-3_1 | I0123 04:51:07.943575 19 hierarchical.cpp:478] Added agent e04535fe-b44a-4b8e-9e58-bcbe1e42920b-S0 (3b259db8e7e1) with cpus(*):8; mem(*):14999; disk(*):1.00268e+06; ports(*):[31000-32000] (allocated: )
mesos-master-3_1 | I0123 04:51:07.943562 14 master.cpp:4619] Registered agent e04535fe-b44a-4b8e-9e58-bcbe1e42920b-S0 at slave(1)@172.18.0.3:5051 (3b259db8e7e1) with cpus(*):8; mem(*):14999; disk(*):1.00268e+06; ports(*):[31000-32000]
mesos-slave-3_1 | I0123 04:51:07.943737 166 slave.cpp:1095] Registered with master master@172.18.0.7:5050; given agent ID e04535fe-b44a-4b8e-9e58-bcbe1e42920b-S0
mesos-slave-3_1 | I0123 04:51:07.943814 165 status_update_manager.cpp:181] Resuming sending status updates
mesos-slave-3_1 | I0123 04:51:07.944067 166 slave.cpp:1155] Forwarding total oversubscribed resources
mesos-master-3_1 | I0123 04:51:07.944192 18 master.cpp:5002] Received update of agent e04535fe-b44a-4b8e-9e58-bcbe1e42920b-S0 at slave(1)@172.18.0.3:5051 (3b259db8e7e1) with total oversubscribed resources
mesos-master-3_1 | I0123 04:51:07.944242 18 hierarchical.cpp:542] Agent e04535fe-b44a-4b8e-9e58-bcbe1e42920b-S0 (3b259db8e7e1) updated with oversubscribed resources (total: cpus(*):8; mem(*):14999; disk(*):1.00268e+06; ports(*):[31000-32000], allocated: )
mesos-test-etcd | 2017-01-23 04:51:09.326822 W | wal: sync duration of 1.096507817s, expected less than 1s
mesos-slave-2_1 | 2017-01-23 04:51:10,294:1(0x7f8aebbea700):ZOO_INFO@check_events@1775: session establishment complete on server [172.18.0.2:2181], sessionId=0x159c9a9a1720007, negotiated timeout=10000
mesos-slave-2_1 | I0123 04:51:10.294569 168 group.cpp:349] Group process (group(1)@172.18.0.6:5051) connected to ZooKeeper
mesos-slave-2_1 | I0123 04:51:10.294600 168 group.cpp:837] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
mesos-slave-2_1 | I0123 04:51:10.294608 168 group.cpp:427] Trying to create path '/mesos' in ZooKeeper
mesos-slave-2_1 | I0123 04:51:10.317852 170 detector.cpp:152] Detected a new leader: (id='0')
mesos-slave-2_1 | I0123 04:51:10.317966 170 group.cpp:706] Trying to get '/mesos/json.info_0000000000' in ZooKeeper
mesos-slave-2_1 | I0123 04:51:10.321454 167 zookeeper.cpp:259] A new leading master (UPID=master@172.18.0.7:5050) is detected
mesos-slave-2_1 | I0123 04:51:10.321568 169 status_update_manager.cpp:174] Pausing sending status updates
mesos-slave-2_1 | I0123 04:51:10.321607 170 slave.cpp:895] New master detected at master@172.18.0.7:5050
mesos-slave-2_1 | I0123 04:51:10.321653 170 slave.cpp:916] No credentials provided. Attempting to register without authentication
mesos-slave-2_1 | I0123 04:51:10.321671 170 slave.cpp:927] Detecting new master
mesos-slave-1_1 | Writing superblocks and filesystem accounting information: done
mesos-slave-1_1 |
mesos-slave-1_1 | time="2017-01-23T04:51:10.913746911Z" level=info msg="libcontainerd: new containerd process, pid: 90"
mesos-master-3_1 | I0123 04:51:10.963834 15 master.cpp:4550] Registering agent at slave(1)@172.18.0.6:5051 (8f8f6a0921a7) with id e04535fe-b44a-4b8e-9e58-bcbe1e42920b-S1
mesos-master-3_1 | I0123 04:51:10.963953 13 registrar.cpp:464] Applied 1 operations in 20932ns; attempting to update the 'registry'
mesos-master-3_1 | I0123 04:51:10.964715 18 registrar.cpp:509] Successfully updated the 'registry' in 736us
mesos-master-3_1 | I0123 04:51:10.965061 14 hierarchical.cpp:478] Added agent e04535fe-b44a-4b8e-9e58-bcbe1e42920b-S1 (8f8f6a0921a7) with cpus(*):8; mem(*):14999; disk(*):1.00268e+06; ports(*):[31000-32000] (allocated: )
mesos-master-3_1 | I0123 04:51:10.965059 13 master.cpp:4619] Registered agent e04535fe-b44a-4b8e-9e58-bcbe1e42920b-S1 at slave(1)@172.18.0.6:5051 (8f8f6a0921a7) with cpus(*):8; mem(*):14999; disk(*):1.00268e+06; ports(*):[31000-32000]
mesos-slave-2_1 | I0123 04:51:10.965263 167 slave.cpp:1095] Registered with master master@172.18.0.7:5050; given agent ID e04535fe-b44a-4b8e-9e58-bcbe1e42920b-S1
mesos-slave-2_1 | I0123 04:51:10.965349 163 status_update_manager.cpp:181] Resuming sending status updates
mesos-slave-2_1 | I0123 04:51:10.965581 167 slave.cpp:1155] Forwarding total oversubscribed resources
mesos-master-3_1 | I0123 04:51:10.965726 16 master.cpp:5002] Received update of agent e04535fe-b44a-4b8e-9e58-bcbe1e42920b-S1 at slave(1)@172.18.0.6:5051 (8f8f6a0921a7) with total oversubscribed resources
mesos-master-3_1 | I0123 04:51:10.965775 16 hierarchical.cpp:542] Agent e04535fe-b44a-4b8e-9e58-bcbe1e42920b-S1 (8f8f6a0921a7) updated with oversubscribed resources (total: cpus(*):8; mem(*):14999; disk(*):1.00268e+06; ports(*):[31000-32000], allocated: )
mesos-slave-1_1 | time="2017-01-23T04:51:12.482841972Z" level=info msg="Graph migration to content-addressability took 0.00 seconds"
mesos-slave-1_1 | time="2017-01-23T04:51:12.483064388Z" level=warning msg="Your kernel does not support cgroup cfs period"
mesos-slave-1_1 | time="2017-01-23T04:51:12.483081767Z" level=warning msg="Your kernel does not support cgroup cfs quotas"
mesos-slave-1_1 | time="2017-01-23T04:51:12.483100137Z" level=warning msg="Your kernel does not support cgroup blkio weight"
mesos-slave-1_1 | time="2017-01-23T04:51:12.483115870Z" level=warning msg="Your kernel does not support cgroup blkio weight_device"
mesos-slave-1_1 | time="2017-01-23T04:51:12.483135847Z" level=warning msg="Your kernel does not support cgroup blkio throttle.read_bps_device"
mesos-slave-1_1 | time="2017-01-23T04:51:12.483149845Z" level=warning msg="Your kernel does not support cgroup blkio throttle.write_bps_device"
mesos-slave-1_1 | time="2017-01-23T04:51:12.483165784Z" level=warning msg="Your kernel does not support cgroup blkio throttle.read_iops_device"
mesos-slave-1_1 | time="2017-01-23T04:51:12.483179590Z" level=warning msg="Your kernel does not support cgroup blkio throttle.write_iops_device"
mesos-slave-1_1 | time="2017-01-23T04:51:12.483468754Z" level=info msg="Loading containers: start."
mesos-slave-1_1 | time="2017-01-23T04:51:12.485036029Z" level=warning msg="Running modprobe bridge br_netfilter failed with message: modprobe: ERROR: ../libkmod/libkmod.c:556 kmod_search_moddep() could not open moddep file '/lib/modules/4.7.3/modules.dep.bin'\nmodprobe: ERROR: ../libkmod/libkmod.c:556 kmod_search_moddep() could not open moddep file '/lib/modules/4.7.3/modules.dep.bin'\n, error: exit status 1"
mesos-slave-1_1 | time="2017-01-23T04:51:12.486503405Z" level=warning msg="Running modprobe nf_nat failed with message: `modprobe: ERROR: ../libkmod/libkmod.c:556 kmod_search_moddep() could not open moddep file '/lib/modules/4.7.3/modules.dep.bin'`, error: exit status 1"
mesos-slave-1_1 | time="2017-01-23T04:51:12.487825841Z" level=warning msg="Running modprobe xt_conntrack failed with message: `modprobe: ERROR: ../libkmod/libkmod.c:556 kmod_search_moddep() could not open moddep file '/lib/modules/4.7.3/modules.dep.bin'`, error: exit status 1"
mesos-slave-1_1 | time="2017-01-23T04:51:13.263590676Z" level=warning msg="Could not load necessary modules for IPSEC rules: Running modprobe xfrm_user failed with message: `modprobe: ERROR: ../libkmod/libkmod.c:556 kmod_search_moddep() could not open moddep file '/lib/modules/4.7.3/modules.dep.bin'`, error: exit status 1"
mesos-slave-1_1 | time="2017-01-23T04:51:13.263792633Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
mesos-slave-1_1 |
mesos-slave-1_1 | time="2017-01-23T04:51:13.685085498Z" level=info msg="Loading containers: done."
mesos-slave-1_1 | time="2017-01-23T04:51:13.685269237Z" level=info msg="Daemon has completed initialization"
mesos-slave-1_1 | time="2017-01-23T04:51:13.685287627Z" level=info msg="Docker daemon" commit=23cf638 graphdriver=overlay version=1.12.1
mesos-slave-1_1 | time="2017-01-23T04:51:13.691057244Z" level=info msg="API listen on /var/run/docker.sock"
mesos-slave-1_1 | WARNING: Logging before InitGoogleLogging() is written to STDERR
mesos-slave-1_1 | I0123 04:51:13.723639 1 main.cpp:243] Build: 2016-08-26 23:00:07 by ubuntu
mesos-slave-1_1 | I0123 04:51:13.723704 1 main.cpp:244] Version: 1.0.1
mesos-slave-1_1 | I0123 04:51:13.723708 1 main.cpp:247] Git tag: 1.0.1
mesos-slave-1_1 | I0123 04:51:13.723711 1 main.cpp:251] Git SHA: 3611eb0b7eea8d144e9b2e840e0ba16f2f659ee3
mesos-slave-1_1 | I0123 04:51:13.829512 1 containerizer.cpp:196] Using isolation: posix/cpu,posix/mem,filesystem/posix,network/cni
mesos-slave-1_1 | I0123 04:51:13.833534 1 linux_launcher.cpp:101] Using /sys/fs/cgroup/freezer as the freezer hierarchy for the Linux launcher
mesos-slave-1_1 | 2017-01-23 04:51:13,834:1(0x7fa332921700):ZOO_INFO@log_env@726: Client environment:zookeeper.version=zookeeper C client 3.4.8
mesos-slave-1_1 | 2017-01-23 04:51:13,834:1(0x7fa332921700):ZOO_INFO@log_env@730: Client environment:host.name=711c723316b6
mesos-slave-1_1 | 2017-01-23 04:51:13,834:1(0x7fa332921700):ZOO_INFO@log_env@737: Client environment:os.name=Linux
mesos-slave-1_1 | 2017-01-23 04:51:13,834:1(0x7fa332921700):ZOO_INFO@log_env@738: Client environment:os.arch=4.7.3
mesos-slave-1_1 | 2017-01-23 04:51:13,834:1(0x7fa332921700):ZOO_INFO@log_env@739: Client environment:os.version=#1 SMP Thu Sep 8 20:01:00 PDT 2016
mesos-slave-1_1 | 2017-01-23 04:51:13,834:1(0x7fa332921700):ZOO_INFO@log_env@747: Client environment:user.name=(null)
mesos-slave-1_1 | I0123 04:51:13.834796 1 main.cpp:434] Starting Mesos agent
mesos-slave-1_1 | 2017-01-23 04:51:13,834:1(0x7fa332921700):ZOO_INFO@log_env@755: Client environment:user.home=/root
mesos-slave-1_1 | 2017-01-23 04:51:13,834:1(0x7fa332921700):ZOO_INFO@log_env@767: Client environment:user.dir=/
mesos-slave-1_1 | I0123 04:51:13.835283 161 slave.cpp:198] Agent started on 1)@172.18.0.10:5051
mesos-slave-1_1 | I0123 04:51:13.835759 161 slave.cpp:519] Agent resources: cpus(*):8; mem(*):14999; disk(*):1.00268e+06; ports(*):[31000-32000]
mesos-slave-1_1 | I0123 04:51:13.835788 161 slave.cpp:527] Agent attributes: [ ]
mesos-slave-1_1 | I0123 04:51:13.835790 161 slave.cpp:532] Agent hostname: 711c723316b6
mesos-slave-1_1 | I0123 04:51:13.838294 160 state.cpp:57] Recovering state from '/data/meta'
mesos-slave-1_1 | I0123 04:51:13.838382 160 status_update_manager.cpp:200] Recovering status update manager
mesos-slave-1_1 | I0123 04:51:13.838471 163 docker.cpp:775] Recovering Docker containers
mesos-slave-1_1 | I0123 04:51:13.838476 162 containerizer.cpp:522] Recovering containerizer
mesos-slave-1_1 | I0123 04:51:13.839789 161 provisioner.cpp:253] Provisioner recovery complete
mesos-slave-1_1 | 2017-01-23 04:51:13,846:1(0x7fa32dec2700):ZOO_INFO@check_events@1728: initiated connection to server [172.18.0.2:2181]
mesos-slave-1_1 | 2017-01-23 04:51:13,873:1(0x7fa32dec2700):ZOO_INFO@check_events@1775: session establishment complete on server [172.18.0.2:2181], sessionId=0x159c9a9a1720008, negotiated timeout=10000
mesos-slave-1_1 | I0123 04:51:13.873672 161 group.cpp:349] Group process (group(1)@172.18.0.10:5051) connected to ZooKeeper
mesos-slave-1_1 | I0123 04:51:13.873708 161 group.cpp:837] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
mesos-slave-1_1 | I0123 04:51:13.873716 161 group.cpp:427] Trying to create path '/mesos' in ZooKeeper
mesos-slave-1_1 | I0123 04:51:13.879220 164 detector.cpp:152] Detected a new leader: (id='0')
mesos-slave-1_1 | I0123 04:51:13.879377 165 group.cpp:706] Trying to get '/mesos/json.info_0000000000' in ZooKeeper
mesos-slave-1_1 | I0123 04:51:13.882776 165 zookeeper.cpp:259] A new leading master (UPID=master@172.18.0.7:5050) is detected
mesos-slave-1_1 | I0123 04:51:13.930491 158 slave.cpp:4782] Finished recovery
mesos-slave-1_1 | I0123 04:51:13.930879 165 status_update_manager.cpp:174] Pausing sending status updates
mesos-slave-1_1 | I0123 04:51:13.930894 163 slave.cpp:895] New master detected at master@172.18.0.7:5050
mesos-slave-1_1 | I0123 04:51:13.930924 163 slave.cpp:916] No credentials provided. Attempting to register without authentication
mesos-slave-1_1 | I0123 04:51:13.930946 163 slave.cpp:927] Detecting new master
mesos-master-3_1 | I0123 04:51:14.024806 12 master.cpp:4550] Registering agent at slave(1)@172.18.0.10:5051 (711c723316b6) with id e04535fe-b44a-4b8e-9e58-bcbe1e42920b-S2
mesos-master-3_1 | I0123 04:51:14.024988 19 registrar.cpp:464] Applied 1 operations in 33013ns; attempting to update the 'registry'
mesos-master-3_1 | I0123 04:51:14.025919 19 registrar.cpp:509] Successfully updated the 'registry' in 889088ns
mesos-master-3_1 | I0123 04:51:14.026201 18 master.cpp:4619] Registered agent e04535fe-b44a-4b8e-9e58-bcbe1e42920b-S2 at slave(1)@172.18.0.10:5051 (711c723316b6) with cpus(*):8; mem(*):14999; disk(*):1.00268e+06; ports(*):[31000-32000]
mesos-master-3_1 | I0123 04:51:14.026226 14 hierarchical.cpp:478] Added agent e04535fe-b44a-4b8e-9e58-bcbe1e42920b-S2 (711c723316b6) with cpus(*):8; mem(*):14999; disk(*):1.00268e+06; ports(*):[31000-32000] (allocated: )
mesos-slave-1_1 | I0123 04:51:14.026566 164 slave.cpp:1095] Registered with master master@172.18.0.7:5050; given agent ID e04535fe-b44a-4b8e-9e58-bcbe1e42920b-S2
mesos-slave-1_1 | I0123 04:51:14.026659 158 status_update_manager.cpp:181] Resuming sending status updates
mesos-slave-1_1 | I0123 04:51:14.026991 164 slave.cpp:1155] Forwarding total oversubscribed resources
mesos-master-3_1 | I0123 04:51:14.027171 19 master.cpp:5002] Received update of agent e04535fe-b44a-4b8e-9e58-bcbe1e42920b-S2 at slave(1)@172.18.0.10:5051 (711c723316b6) with total oversubscribed resources
mesos-master-3_1 | I0123 04:51:14.027236 19 hierarchical.cpp:542] Agent e04535fe-b44a-4b8e-9e58-bcbe1e42920b-S2 (711c723316b6) updated with oversubscribed resources (total: cpus(*):8; mem(*):14999; disk(*):1.00268e+06; ports(*):[31000-32000], allocated: )
mesos-slave-3_1 | I0123 04:52:05.284935 166 slave.cpp:4591] Current disk usage 33.34%. Max allowed age: 3.966212305291516days
mesos-slave-2_1 | I0123 04:52:07.465173 170 slave.cpp:4591] Current disk usage 33.34%. Max allowed age: 3.966212251027535days
mesos-slave-1_1 | I0123 04:52:13.836541 160 slave.cpp:4591] Current disk usage 33.34%. Max allowed age: 3.966212169631563days
mesos-slave-3_1 | I0123 04:53:05.285915 162 slave.cpp:4591] Current disk usage 33.34%. Max allowed age: 3.966211220011852days
mesos-slave-2_1 | I0123 04:53:07.465814 168 slave.cpp:4591] Current disk usage 33.34%. Max allowed age: 3.966211165747870days
mesos-slave-1_1 | I0123 04:53:13.837671 163 slave.cpp:4591] Current disk usage 33.34%. Max allowed age: 3.966211084351898daysfollowed by endless zk pings (with no xchk errors) since it's idling. I don't see any "Failed" messages from mesos.
wegel
commented
7 years ago Happy to report that the same ZK=zetcd docker-compose up seems to work without errors with the latest master. I'll try zetcd in a real mesos-cluster setup and report back. Thanks!
wegel
commented
7 years ago So there's some really nice progress, but we're not there yet.
I modified docker-compose.yml in order to remove as much noise as possible while still making it more production-like (single etcd, single zetcd, 3 mesos-masters, 1 mesos-slave):
version: '2'
networks:
zetcd-test-net:
services:
etcd:
image: quay.io/coreos/etcd
container_name: mesos-test-etcd
command: >
etcd --name etcd0
--advertise-client-urls http://etcd:2379,http://etcd:4001
--listen-client-urls http://0.0.0.0:2379,http://0.0.0.0:4001
--initial-advertise-peer-urls http://etcd:2380
--listen-peer-urls http://0.0.0.0:2380
--initial-cluster-token etcd-cluster-1
--initial-cluster etcd0=http://etcd:2380
--initial-cluster-state new
networks:
- zetcd-test-net
zookeeper:
image: zookeeper:3.4.9
networks:
- zetcd-test-net
zetcd-1:
build: .
command: zetcd -endpoint http://etcd:2379 -zkaddr zetcd:2181 ${ZETCD_OPTS} -logtostderr -v 10
networks:
zetcd-test-net:
aliases:
- zetcd
mesos-master-1:
image: mesosphere/mesos:1.0.1-2.0.93.ubuntu1404
command: mesos-master --registry=replicated_log --work_dir=/data
environment:
- MESOS_QUORUM=${MESOS_QUORUM}
- MESOS_PORT=5050
- MESOS_ZK=zk://${ZK}:2181/mesos
restart: unless-stopped
ports:
- "5051:5050"
networks:
zetcd-test-net:
aliases:
- mesos-master
mesos-master-2:
image: mesosphere/mesos:1.0.1-2.0.93.ubuntu1404
command: mesos-master --registry=replicated_log --work_dir=/data
environment:
- MESOS_QUORUM=${MESOS_QUORUM}
- MESOS_PORT=5050
- MESOS_ZK=zk://${ZK}:2181/mesos
restart: unless-stopped
ports:
- "5052:5050"
networks:
zetcd-test-net:
aliases:
- mesos-master
mesos-master-3:
image: mesosphere/mesos:1.0.1-2.0.93.ubuntu1404
command: mesos-master --registry=replicated_log --work_dir=/data
environment:
- MESOS_QUORUM=${MESOS_QUORUM}
- MESOS_PORT=5050
- MESOS_ZK=zk://${ZK}:2181/mesos
restart: unless-stopped
ports:
- "5053:5050"
networks:
zetcd-test-net:
aliases:
- mesos-master
mesos-slave-1:
image: mesosphere/mesos-slave-dind:0.2.4_mesos-1.0.1_docker-1.12.1_ubuntu-14.04.5
privileged: true
volumes:
- /sys/fs/cgroup:/sys/fs/cgroup
environment:
- MESOS_CONTAINERIZERS=docker,mesos
- MESOS_WORK_DIR=/data
- MESOS_MASTER=zk://${ZK}:2181/mesos
- MESOS_DOCKER_STOP_TIMEOUT=2mins
- MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins
restart: unless-stopped
networks:
zetcd-test-net:
aliases:
- mesos-slaveWhat has also been modified since last time is --registry=replicated_log --work_dir=/data parameter to mesos-master and the relevant MESOS_QUORUM environment variable. Those are necessary to run a HA mesos cluster, as specified in Mesos Operatonial Guide.
I also added a port-publish for each of the mesos-masters: if you visit localhost:5051 that'll correspond to mesos-master-1, localhost:5052 to mesos-master-2 and localhost:5053 to mesos-master-3. Only one of those 3 will be elected master at all times, and only the actual master will allow you to see the current status (mesos-master doesn't proxy to the current master by itself). So you can try localhost:[5051,5052,5053] until you get the current master.
One way to quickly check if everything seems OK from Mesos' point of view is to open the web interface, and check that there's 1 Agent registered in the line:
Activated 1
You can see the problem by running:
ZK=zetcd MESOS_QUORUM=2 ZETCD_OPTS="-zkbridge zookeeper:2181 -oracle zk" docker-compose up --force-recreate
You will get:
...
zetcd-1_1 | I0201 21:41:08.122600 42 session.go:59] starting the session... id=7635107680527841548
zetcd-1_1 | I0201 21:41:08.122623 42 session.go:84] auth resp OK ({ProtocolVersion:0 TimeOut:10000 SessionID:97385530191511564 Passwd:[51 119 1 68 67 148 21 90 91 220 196 80 48 16 182 153]})
zetcd-1_1 | I0201 21:41:08.122639 42 server.go:46] serving global serial session requests on s=&{Conn:0xc42037b8c0 oracle:0xc420194780 candidate:0xc42030a280 req:{ProtocolVersion:0 LastZxidSeen:0 TimeOut:10000 SessionID:0 Passwd:[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]} sp:0xc420137060}
zetcd-1_1 | I0201 21:41:08.125285 42 session.go:136] zk/zkresp=({Xid:1485985262 Zxid:347 Err:0},&{Stat:{Czxid:94 Mzxid:94 Ctime:1485983036537 Mtime:1485983036537 Version:0 Cversion:42 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:6 Pzxid:333}})
zetcd-1_1 | W0201 21:41:08.125341 42 zk.go:331] xchk failed (err mismatch)
zetcd-1_1 | candidate: &{Xid:1485985262 Zxid:13 Err:-101}
zetcd-1_1 | oracle: &{Xid:1485985262 Zxid:347 Err:0}
zetcd-1_1 | I0201 21:41:08.125350 42 conn.go:131] sendXchk Xid:1485985262 ZXid:347 Resp:&{Stat:{Czxid:94 Mzxid:94 Ctime:1485983036537 Mtime:1485983036537 Version:0 Cversion:42 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:6 Pzxid:333}}
zetcd-1_1 | I0201 21:41:08.125959 42 conn.go:133] conn.Send(xid=1485985262, zxid=347, &{Stat:{Czxid:94 Mzxid:94 Ctime:1485983036537 Mtime:1485983036537 Version:0 Cversion:42 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:6 Pzxid:333}})
zetcd-1_1 | I0201 21:41:08.126209 42 server.go:110] zkreq={xid:1485985263 req:*zetcd.ExistsRequest:&{Path:/mesos Watch:false}}
zetcd-1_1 | I0201 21:41:08.126259 42 zklog.go:38] Exists(1485985263,{Path:/mesos Watch:false})
zetcd-1_1 | I0201 21:41:08.126262 42 zklog.go:38] Exists(1485985263,{Path:/mesos Watch:false})
zetcd-1_1 | I0201 21:41:08.126573 42 session.go:116] waitFutureSendResp=&{Path:/mesos Watch:false}
zetcd-1_1 | I0201 21:41:08.126918 42 session.go:136] zk/zkresp=({Xid:1485985263 Zxid:347 Err:0},&{Stat:{Czxid:3 Mzxid:3 Ctime:1485978557611 Mtime:1485978557611 Version:0 Cversion:73 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:7 Pzxid:332}})
zetcd-1_1 | W0201 21:41:08.126949 42 zk.go:331] xchk failed (err mismatch)
zetcd-1_1 | candidate: &{Xid:1485985263 Zxid:13 Err:-101}
zetcd-1_1 | oracle: &{Xid:1485985263 Zxid:347 Err:0}
zetcd-1_1 | I0201 21:41:08.126958 42 conn.go:131] sendXchk Xid:1485985263 ZXid:347 Resp:&{Stat:{Czxid:3 Mzxid:3 Ctime:1485978557611 Mtime:1485978557611 Version:0 Cversion:73 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:7 Pzxid:332}}
zetcd-1_1 | I0201 21:41:08.127779 42 conn.go:133] conn.Send(xid=1485985263, zxid=347, &{Stat:{Czxid:3 Mzxid:3 Ctime:1485978557611 Mtime:1485978557611 Version:0 Cversion:73 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:7 Pzxid:332}})
zetcd-1_1 | I0201 21:41:08.127808 42 server.go:110] zkreq={xid:1485985264 req:*zetcd.ExistsRequest:&{Path:/mesos Watch:false}}
...If you remove the bridge, the cluster will never stabilize a leader (there will be constant mesos-master election) and the mesos-slave will never be able to register to the master:
ZK=zetcd MESOS_QUORUM=2 docker-compose up --force-recreate
You can validate that by checking localhost:[5051,5052,5053] with the line:
Activated 0
A big thank you for your continued work. If I can do anything else to help, please don't hesitate.
wrouesnel
commented
7 years ago I can confirm this problem. In addition I found a situation while playing with this where two Mesos Master's managed to believe each other were leaders (requests got endlessly redirected).
EDIT: To clarify the issue - agent nodes never appear to detect the leading master. The replicated_log in mesos never appears to successfully catch up, leading to a mesos-master crash of the current leader which forces a re-election.
I used a similar file to the above but added a bit more determism so the various web interfaces work "nicely":
version: '2'
networks:
zetcd-test-net:
driver: bridge
ipam:
config:
- subnet: 172.20.0.0/24
gateway: 172.20.0.1
volumes:
mesos-master-1:
mesos-master-2:
mesos-master-3:
services:
etcd0:
image: quay.io/coreos/etcd
command: |
etcd --name etcd0
--advertise-client-urls http://172.20.0.2:2379,http://172.20.0.2:4001
--listen-client-urls http://0.0.0.0:2379,http://0.0.0.0:4001
--initial-advertise-peer-urls http://172.20.0.2:2380
--listen-peer-urls http://0.0.0.0:2380
--initial-cluster-token etcd-cluster
--initial-cluster etcd0=http://172.20.0.2:2380,etcd1=http://172.20.0.3:2380,etcd2=http://172.20.0.4:2380
--initial-cluster-state new
networks:
zetcd-test-net:
ipv4_address: 172.20.0.2
etcd1:
image: quay.io/coreos/etcd
command: |
etcd --name etcd1
--advertise-client-urls http://172.20.0.3:2379,http://172.20.0.3:4001
--listen-client-urls http://0.0.0.0:2379,http://0.0.0.0:4001
--initial-advertise-peer-urls http://172.20.0.3:2380
--listen-peer-urls http://0.0.0.0:2380
--initial-cluster-token etcd-cluster
--initial-cluster etcd0=http://172.20.0.2:2380,etcd1=http://172.20.0.3:2380,etcd2=http://172.20.0.4:2380
--initial-cluster-state new
networks:
zetcd-test-net:
ipv4_address: 172.20.0.3
etcd2:
image: quay.io/coreos/etcd
command: |
etcd --name etcd2
--advertise-client-urls http://172.20.0.4:2379,http://172.20.0.4:4001
--listen-client-urls http://0.0.0.0:2379,http://0.0.0.0:4001
--initial-advertise-peer-urls http://172.20.0.4:2380
--listen-peer-urls http://0.0.0.0:2380
--initial-cluster-token etcd-cluster
--initial-cluster etcd0=http://172.20.0.2:2380,etcd1=http://172.20.0.3:2380,etcd2=http://172.20.0.4:2380
--initial-cluster-state new
networks:
zetcd-test-net:
ipv4_address: 172.20.0.4
# zookeeper:
# image: zookeeper:3.4.9
# networks:
# - zetcd-test-net
zetcd0:
build: zetcd
command: -endpoint=http://172.20.0.2:2379 -zkaddr=:2181 -logtostderr -v 10
networks:
zetcd-test-net:
ipv4_address: 172.20.0.12
depends_on:
- etcd0
- etcd1
- etcd2
zetcd1:
build: zetcd
command: -endpoint=http://172.20.0.3:2379 -zkaddr=:2181 -logtostderr -v 10
networks:
zetcd-test-net:
ipv4_address: 172.20.0.13
depends_on:
- etcd0
- etcd1
- etcd2
zetcd2:
build: zetcd
command: -endpoint=http://172.20.0.4:2379 -zkaddr=:2181 -logtostderr -v 10
networks:
zetcd-test-net:
ipv4_address: 172.20.0.14
depends_on:
- etcd0
- etcd1
- etcd2
mesos-master-1:
image: mesosphere/mesos:1.2.0-rc2
command: mesos-master --registry=replicated_log --work_dir=/data
volumes:
- mesos-master-1:/data
environment:
- MESOS_QUORUM=2
- MESOS_PORT=5050
- MESOS_ZK=zk://zetcd0:2181,zetcd1:2181,zetcd2:2181/mesos
- MESOS_HOSTNAME=172.20.0.22
restart: unless-stopped
ports:
- 5050:5050
networks:
zetcd-test-net:
ipv4_address: 172.20.0.22
depends_on:
- zetcd0
- zetcd1
- zetcd2
mesos-master-2:
image: mesosphere/mesos:1.2.0-rc2
command: mesos-master --registry=replicated_log --work_dir=/data
volumes:
- mesos-master-2:/data
environment:
- MESOS_QUORUM=2
- MESOS_PORT=5051
- MESOS_ZK=zk://172.20.0.12:2181,172.20.0.13:2181,172.20.0.14:2181/mesos
- MESOS_HOSTNAME=172.20.0.23
restart: unless-stopped
ports:
- 5051:5051
networks:
zetcd-test-net:
ipv4_address: 172.20.0.23
depends_on:
- zetcd0
- zetcd1
- zetcd2
mesos-master-3:
image: mesosphere/mesos:1.2.0-rc2
command: mesos-master --registry=replicated_log --work_dir=/data
volumes:
- mesos-master-3:/data
environment:
- MESOS_QUORUM=2
- MESOS_PORT=5052
- MESOS_ZK=zk://172.20.0.12:2181,172.20.0.13:2181,172.20.0.14:2181/mesos
- MESOS_HOSTNAME=172.20.0.24
restart: unless-stopped
ports:
- 5052:5052
networks:
zetcd-test-net:
ipv4_address: 172.20.0.24
depends_on:
- zetcd0
- zetcd1
- zetcd2
mesos-slave-1:
image: mesosphere/mesos:1.2.0-rc2
privileged: true
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /usr/bin/docker:/usr/bin/docker:ro
- /sys/fs/cgroup:/sys/fs/cgroup
command: mesos-agent
--no-systemd_enable_support
--no-hostname_lookup
environment:
- MESOS_CONTAINERIZERS=docker
- MESOS_WORK_DIR=/data
- MESOS_MASTER=zk://172.20.0.12:2181,172.20.0.13:2181,172.20.0.14:2181/mesos
- MESOS_DOCKER_STOP_TIMEOUT=2mins
- MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins
- MESOS_PORT=5060
- MESOS_HOSTNAME=172.20.0.30
restart: unless-stopped
ports:
- 5060:5060
networks:
zetcd-test-net:
ipv4_address: 172.20.0.30
depends_on:
- zetcd0
- zetcd1
- zetcd2
marathon:
image: mesosphere/marathon
command: --master zk://172.20.0.12:2181,172.20.0.13:2181,172.20.0.14:2181/mesos
--zk zk://172.20.0.12:2181,172.20.0.13:2181,172.20.0.14:2181/marathon
ports:
- 8080:8080
networks:
zetcd-test-net:
ipv4_address: 172.20.0.40Chasing this down a bit, the specific problem is that the watch on /mesos/log_replicas never returns when it's backed by zetcd, whereas it does when backed by zookeeper (bridged via zetcd).
Working (ZK):
$ zkctl --zkaddr="172.20.0.12:2181" watch /mesos/log_replicas
watch dir /mesos/log_replicas
2017/03/01 02:59:52 Connected to 172.20.0.12:2181
2017/03/01 02:59:52 Authenticated: id=97537070881243149, timeout=4000
[0000000001 0000000000 0000000003 0000000002 0000000005 0000000004] &{Czxid:4294967304 Mzxid:4294967304 Ctime:1488212417301 Mtime:1488212417301 Version:0 Cversion:6 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:6 Pzxid:4294967353}
{Type:EventNodeChildrenChanged State:Unknown Path:/mesos/log_replicas Err:<nil> Server:}Not working (etcd):
$ zkctl --zkaddr="172.20.0.12:2181" watch /mesos/log_replicas
watch dir /mesos/log_replicas
2017/03/01 02:59:14 Connected to 172.20.0.12:2181
2017/03/01 02:59:14 Authenticated: id=4220253854931057365, timeout=1000
[] &{Czxid:9 Mzxid:9 Ctime:1488297207999137 Mtime:1488297207999137 Version:0 Cversion:8 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:0 Pzxid:91}
^C(exited because I could see Mesos panicking about not succeeding at log recovery).
So it looks like the problem is the num children changed event isn't firing.
wrouesnel
commented
7 years ago Followed this a bit more and found the problem - when a children watch is specified, the path sent to etcd looks wrong - i.e. /mesos/log_replicas becomes /zk/ver/<len>mesos/log_replicas/ (note the trailing slash) - which doesn't correspond to any key that is watchable.
If I stick a watch on /zk/ver though I can see events for /zk/ver/<len>mesos/log_replicas00000001 and others firing just fine.
So that's the problem - this behavior is bugged. I'm looking into fixing it but I need to cross-check my understanding of the backing key-layout for these types of keys first.
EDIT: To be clear - I'm not sure what the intent of the /zk/ver tree structure is - since an obvious solution is that /zk/ver/<len>mesos/log_replicas00000001 becomes /zk/ver/<len>mesos/log_replicas/00000001 i.e. to reflect the sequence keys since etcd doesn't seem to want to watch an arbitrary prefix.
wrouesnel
commented
7 years ago Okay so I've narrowed this down to FlagSequence nodes being created incorrectly I think:
$ zkctl --zkaddr="172.20.0.12:2181" ls /mesos/log_replicas
ls dir /mesos/log_replicas
2017/03/02 02:25:50 Connected to 172.20.0.12:2181
2017/03/02 02:25:50 Authenticated: id=97542596434657299, timeout=4000
Children:
/mesos/log_replicas/0000000009 (0000000009)
/mesos/log_replicas/0000000010 (0000000010)
/mesos/log_replicas/0000000011 (0000000011)Cross confirmed by zkCli.sh
But when using the etcd backend the keys are created as (note: I've patched zetcd to write the child count as a hex number):
/zk/ver/02/mesos/log_replicas
/zk/ver/02/mesos/log_replicas0000000001
/zk/ver/02/mesos/log_replicas0000000002
/zk/ver/02/mesos/log_replicas0000000003But! The watch command applied for etcd calls getListPfx which (in my patched version - equivalent to the original) produces this path as the prefix to watch: /zk/ver/03/mesos/log_replicas/ - which obviously doesn't line up to anything in the key store.
There's 2 obvious ways to fix this - change the FlagSequence code to make the keys as child directories (which looks like it would be consistent with ZK's internal representation near as I can tell) or special-case the watch logic to use the right prefix when watching a sequential key like this.
I'm going to implement the former and see if that brings Mesos too working.
EDIT: For comparison, this is the output of zkctl when using the zetcd backend (modified) currently:
/mesos/log_replicas (log_replicas)
/mesos/log_replicas0000000000 (log_replicas0000000000)
/mesos/log_replicas0000000001 (log_replicas0000000001)
/mesos/log_replicas0000000002 (log_replicas0000000002)i.e. obviously different.
wrouesnel
commented
7 years ago SOLVED! - a version I've used successfully is here: https://github.com/wrouesnel/zetcd/tree/fix-watches (branch needs clean up - I added miles of logging to figure this out).
Ok: so the exact problem, is the call to path.Clean(zkPath) in mkPath (here: https://github.com/coreos/zetcd/blob/master/path.go#L22)
path.Clean is squashing the trailing / off the parent node path, which under regular Zookeeper operation basically lets you nest sequential keys - Mesos nests
/mesos/json_info-0000000x <- counter tracked under /mesos
and
/mesos/log_replicas <- counter tracked under /mesos/log_replicas
This seems to be consistent with the ZK documentation here: https://zookeeper.apache.org/doc/r3.2.1/zookeeperProgrammers.html#Sequence+Nodes+--+Unique+Naming
So path.Clean is a bug because the extra / should be allowed. That part needs to be replaced I think with some ingress filtering for valid Zookeeper paths, so mkPaths can be treated as the "never fails" type function it is that purely does ZK->etcd conversion.
With this I've started and had a mesos cluster successfully spin up in docker.
xiang90
commented
7 years ago @wrouesnel Great analysis. Can you send a PR to fix this?
wrouesnel
commented
7 years ago PR submitted. Literally a 1-line fix (plus removing the path import in the relevant file since it's not used anymore).
haoyixin
commented
7 years ago wegel
commented
7 years ago mesos still seems broken when using a recent version of zetcd:
$ cat docker-compose.yml
version: '2'
networks:
cluster-net:
services:
etcd:
image: quay.io/coreos/etcd
container_name: mesos-test-etcd
command: >
etcd --name etcd0
--advertise-client-urls http://etcd:2379,http://etcd:4001
--listen-client-urls http://0.0.0.0:2379,http://0.0.0.0:4001
--initial-advertise-peer-urls http://etcd:2380
--listen-peer-urls http://0.0.0.0:2380
--initial-cluster-token etcd-cluster-1
--initial-cluster etcd0=http://etcd:2380
--initial-cluster-state new
networks:
- cluster-net
zookeeper:
image: zookeeper:3.4.9
networks:
- cluster-net
zetcd-1:
image: quay.io/coreos/zetcd
command: -endpoints http://etcd:2379 -zkaddr zetcd:2181 ${ZETCD_OPTS} -logtostderr -v 10
networks:
cluster-net:
aliases:
- zetcd
zetcd-2:
image: quay.io/coreos/zetcd
command: -endpoints http://etcd:2379 -zkaddr zetcd:2181 ${ZETCD_OPTS} -logtostderr -v 10
networks:
cluster-net:
aliases:
- zetcd
zetcd-3:
image: quay.io/coreos/zetcd
command: -endpoints http://etcd:2379 -zkaddr zetcd:2181 ${ZETCD_OPTS} -logtostderr -v 10
networks:
cluster-net:
aliases:
- zetcd
mesos-master-1:
image: mesosphere/mesos-master:1.3.1
environment:
- MESOS_PORT=5050
- MESOS_ZK=zk://${ZK}:2181/mesos
restart: unless-stopped
ports:
- 5051:5050
networks:
cluster-net:
aliases:
- mesos-master
mesos-master-2:
image: mesosphere/mesos-master:1.3.1
environment:
- MESOS_PORT=5050
- MESOS_ZK=zk://${ZK}:2181/mesos
restart: unless-stopped
ports:
- 5052:5050
networks:
cluster-net:
aliases:
- mesos-master
mesos-master-3:
image: mesosphere/mesos-master:1.3.1
environment:
- MESOS_PORT=5050
- MESOS_ZK=zk://${ZK}:2181/mesos
restart: unless-stopped
ports:
- 5053:5050
networks:
cluster-net:
aliases:
- mesos-master
mesos-slave-1:
image: mesosphere/mesos-slave-dind:0.3.0_mesos-1.0.1_docker-1.10.3_ubuntu-14.04.5
privileged: true
environment:
- MESOS_CONTAINERIZERS=docker,mesos
- MESOS_WORK_DIR=/data
- MESOS_MASTER=zk://${ZK}:2181/mesos
- MESOS_DOCKER_STOP_TIMEOUT=2mins
- MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins
restart: unless-stopped
networks:
cluster-net:
aliases:
- mesos-slave
mesos-slave-2:
image: mesosphere/mesos-slave-dind:0.3.0_mesos-1.0.1_docker-1.10.3_ubuntu-14.04.5
privileged: true
environment:
- MESOS_CONTAINERIZERS=docker,mesos
- MESOS_WORK_DIR=/data
- MESOS_MASTER=zk://${ZK}:2181/mesos
- MESOS_DOCKER_STOP_TIMEOUT=2mins
- MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins
restart: unless-stopped
networks:
cluster-net:
aliases:
- mesos-slave
mesos-slave-3:
image: mesosphere/mesos-slave-dind:0.3.0_mesos-1.0.1_docker-1.10.3_ubuntu-14.04.5
privileged: true
environment:
- MESOS_CONTAINERIZERS=docker,mesos
- MESOS_WORK_DIR=/data
- MESOS_MASTER=zk://${ZK}:2181/mesos
- MESOS_DOCKER_STOP_TIMEOUT=2mins
- MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins
restart: unless-stopped
networks:
cluster-net:
aliases:
- mesos-slave
marathon:
image: mesosphere/marathon
command: --master zk://${ZK}:2181/mesos
--zk zk://${ZK}:2181/marathon
ports:
- 8080:8080
networks:
cluster-net:
aliases:
- marathon
depends_on:
- mesos-master-1If you run zetcd in bridge mode:
ZK=zetcd ZETCD_OPTS="-debug-zkbridge zookeeper:2181 -debug-oracle zk" docker-compose up --force-recreate
...you will get a lot of:
...
zetcd-1_1 | I0918 19:42:34.301804 1 conn.go:131] sendXchk Xid:-2 ZXid:214 Resp:&{}
zetcd-1_1 | I0918 19:42:34.302208 1 conn.go:139] conn.Send(xid=-2, zxid=214, &{})
zetcd-1_1 | I0918 19:42:34.302229 1 server.go:110] zkreq={xid:-2 req:*zetcd.PingRequest:&{}}
zetcd-1_1 | I0918 19:42:34.302360 1 zklog.go:73] Ping(-2,{})
zetcd-1_1 | I0918 19:42:34.302392 1 zklog.go:73] Ping(-2,{})
zetcd-1_1 | I0918 19:42:34.303321 1 session.go:122] waitFutureSendResp=&{}
zetcd-1_1 | I0918 19:42:34.305209 1 session.go:142] zk/zkresp=({Xid:-2 Zxid:214 Err:0},&{})
zetcd-1_1 | W0918 19:42:34.305431 1 zk.go:318] xchk failed (zxid mismatch)
zetcd-1_1 | candidate: &{Xid:-2 Zxid:15 Err:0}
zetcd-1_1 | oracle: &{Xid:-2 Zxid:214 Err:0}
zetcd-1_1 | I0918 19:42:34.305489 1 conn.go:131] sendXchk Xid:-2 ZXid:214 Resp:&{}
zetcd-1_1 | I0918 19:42:34.305951 1 conn.go:139] conn.Send(xid=-2, zxid=214, &{})
zetcd-1_1 | I0918 19:42:34.634550 1 server.go:110] zkreq={xid:-2 req:*zetcd.PingRequest:&{}}
zetcd-1_1 | I0918 19:42:34.634727 1 zklog.go:73] Ping(-2,{})
zetcd-1_1 | I0918 19:42:34.634801 1 zklog.go:73] Ping(-2,{})
zetcd-1_1 | I0918 19:42:34.635296 1 session.go:122] waitFutureSendResp=&{}
zetcd-1_1 | I0918 19:42:34.636227 1 session.go:142] zk/zkresp=({Xid:-2 Zxid:214 Err:0},&{})
zetcd-1_1 | W0918 19:42:34.636286 1 zk.go:318] xchk failed (zxid mismatch)
zetcd-1_1 | candidate: &{Xid:-2 Zxid:15 Err:0}
zetcd-1_1 | oracle: &{Xid:-2 Zxid:214 Err:0}
zetcd-1_1 | I0918 19:42:34.636325 1 conn.go:131] sendXchk Xid:-2 ZXid:214 Resp:&{}
zetcd-1_1 | I0918 19:42:34.636365 1 conn.go:139] conn.Send(xid=-2, zxid=214, &{})
zetcd-1_1 | I0918 19:42:34.636413 1 server.go:110] zkreq={xid:-2 req:*zetcd.PingRequest:&{}}
zetcd-1_1 | I0918 19:42:34.636634 1 zklog.go:73] Ping(-2,{})
zetcd-1_1 | I0918 19:42:34.636761 1 zklog.go:73] Ping(-2,{})
zetcd-1_1 | I0918 19:42:34.637247 1 session.go:122] waitFutureSendResp=&{}
zetcd-1_1 | I0918 19:42:34.638166 1 session.go:142] zk/zkresp=({Xid:-2 Zxid:214 Err:0},&{})
zetcd-1_1 | W0918 19:42:34.638233 1 zk.go:318] xchk failed (zxid mismatch)
zetcd-1_1 | candidate: &{Xid:-2 Zxid:15 Err:0}
zetcd-1_1 | oracle: &{Xid:-2 Zxid:214 Err:0}
zetcd-1_1 | I0918 19:42:34.638278 1 conn.go:131] sendXchk Xid:-2 ZXid:214 Resp:&{}
zetcd-1_1 | I0918 19:42:34.638729 1 conn.go:139] conn.Send(xid=-2, zxid=214, &{})
zetcd-1_1 | I0918 19:42:34.638766 1 server.go:110] zkreq={xid:-2 req:*zetcd.PingRequest:&{}}
zetcd-1_1 | I0918 19:42:34.638899 1 zklog.go:73] Ping(-2,{})
zetcd-1_1 | I0918 19:42:34.638955 1 zklog.go:73] Ping(-2,{})
zetcd-1_1 | I0918 19:42:34.639337 1 session.go:122] waitFutureSendResp=&{}
zetcd-1_1 | I0918 19:42:34.640075 1 session.go:142] zk/zkresp=({Xid:-2 Zxid:214 Err:0},&{})
zetcd-1_1 | W0918 19:42:34.640134 1 zk.go:318] xchk failed (zxid mismatch)
zetcd-1_1 | candidate: &{Xid:-2 Zxid:15 Err:0}
zetcd-1_1 | oracle: &{Xid:-2 Zxid:214 Err:0}
zetcd-1_1 | I0918 19:42:34.640206 1 conn.go:131] sendXchk Xid:-2 ZXid:214 Resp:&{}
zetcd-1_1 | I0918 19:42:34.640240 1 conn.go:139] conn.Send(xid=-2, zxid=214, &{})
...If you want in direct zookeeper mode: ZK=zookeeper ZETCD_OPTS="-debug-zkbridge zookeeper:2181 -debug-oracle zk" docker-compose up --force-recreate
heyitsanthony
commented
7 years ago zetcd-1_1 | I0918 19:42:34.303321 1 session.go:122] waitFutureSendResp=&{}
zetcd-1_1 | I0918 19:42:34.305209 1 session.go:142] zk/zkresp=({Xid:-2 Zxid:214 Err:0},&{})
zetcd-1_1 | W0918 19:42:34.305431 1 zk.go:318] xchk failed (zxid mismatch)
zetcd-1_1 | candidate: &{Xid:-2 Zxid:15 Err:0}
zetcd-1_1 | oracle: &{Xid:-2 Zxid:214 Err:0}The zookeeper zxid is much greater than the candidate zxid. Was this with a fresh zookeeper cluster? What is the first xchk error?
a recent version of zetcd
which version / SHA?
wegel
commented
7 years ago which version / SHA?
This is running the latest quay.io/coreos/zetcd, so v0.0.4 and e4352ce3cc940bc5b60bb3dd69c14f16dac2980a.
The zookeeper zxid is much greater than the candidate zxid. Was this with a fresh zookeeper cluster? What is the first xchk error?
docker-compose rm -v
ZK=zetcd ZETCD_OPTS="-debug-zkbridge zookeeper:2181 -debug-oracle zk" docker-compose up --force-recreate | grep 'zetcd\|mesos-master'First xchk error is: zetcd-1_1 | W0925 18:38:03.839920 1 zk.go:318] xchk failed (stat mismatch)
zetcd-2_1 | Running zetcd proxy
zetcd-3_1 | Running zetcd proxy
zetcd-2_1 | Version: v0.0.4
zetcd-3_1 | Version: v0.0.4
zetcd-2_1 | SHA: e4352ce3cc940bc5b60bb3dd69c14f16dac2980a
zetcd-3_1 | SHA: e4352ce3cc940bc5b60bb3dd69c14f16dac2980a
zetcd-2_1 | I0925 18:38:03.285440 1 server.go:128] accepted remote connection "172.18.0.10:60236"
mesos-master-1_1 | WARNING: Logging before InitGoogleLogging() is written to STDERR
mesos-master-3_1 | WARNING: Logging before InitGoogleLogging() is written to STDERR
mesos-master-2_1 | WARNING: Logging before InitGoogleLogging() is written to STDERR
mesos-master-1_1 | I0925 18:38:03.281867 1 main.cpp:232] Build: 2017-09-05 18:08:25 by ubuntu
mesos-master-3_1 | I0925 18:38:03.281867 1 main.cpp:232] Build: 2017-09-05 18:08:25 by ubuntu
zetcd-2_1 | I0925 18:38:03.285626 1 authconn.go:53] auth(&{ProtocolVersion:0 LastZxidSeen:0 TimeOut:10000 SessionID:0 Passwd:[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]})
mesos-master-2_1 | I0925 18:38:03.565781 1 main.cpp:232] Build: 2017-09-05 18:08:25 by ubuntu
mesos-master-1_1 | I0925 18:38:03.281955 1 main.cpp:233] Version: 1.3.1
mesos-master-3_1 | I0925 18:38:03.281957 1 main.cpp:233] Version: 1.3.1
zetcd-1_1 | Running zetcd proxy
zetcd-2_1 | I0925 18:38:03.285559 1 server.go:128] accepted remote connection "172.18.0.11:57018"
mesos-master-2_1 | I0925 18:38:03.565871 1 main.cpp:233] Version: 1.3.1
mesos-master-1_1 | I0925 18:38:03.281960 1 main.cpp:236] Git tag: 1.3.1
mesos-master-1_1 | I0925 18:38:03.281962 1 main.cpp:240] Git SHA: 1beaede8c13f0832d4921121da34f924deec8950
mesos-master-3_1 | I0925 18:38:03.281965 1 main.cpp:236] Git tag: 1.3.1
zetcd-1_1 | Version: v0.0.4
zetcd-2_1 | I0925 18:38:03.285821 1 session.go:54] newSession([zookeeper:2181])
mesos-master-2_1 | I0925 18:38:03.565878 1 main.cpp:236] Git tag: 1.3.1
mesos-master-3_1 | I0925 18:38:03.281968 1 main.cpp:240] Git SHA: 1beaede8c13f0832d4921121da34f924deec8950
zetcd-1_1 | SHA: e4352ce3cc940bc5b60bb3dd69c14f16dac2980a
zetcd_zetcd-3_1 exited with code 1
mesos-master-1_1 | I0925 18:38:03.283102 1 main.cpp:339] Using 'HierarchicalDRF' allocator
mesos-master-2_1 | I0925 18:38:03.565881 1 main.cpp:240] Git SHA: 1beaede8c13f0832d4921121da34f924deec8950
zetcd-2_1 | I0925 18:38:03.285866 1 authconn.go:53] auth(&{ProtocolVersion:0 LastZxidSeen:0 TimeOut:10000 SessionID:0 Passwd:[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]})
zetcd-2_1 | I0925 18:38:03.285835 1 session.go:54] newSession([zookeeper:2181])
mesos-master-3_1 | I0925 18:38:03.283071 1 main.cpp:339] Using 'HierarchicalDRF' allocator
mesos-master-2_1 | I0925 18:38:03.567328 1 main.cpp:339] Using 'HierarchicalDRF' allocator
zetcd-1_1 | I0925 18:38:03.801967 1 server.go:128] accepted remote connection "172.18.0.10:58344"
mesos-master-1_1 | 2017-09-25 18:38:03,283:1(0x7f6158ff6700):ZOO_INFO@log_env@726: Client environment:zookeeper.version=zookeeper C client 3.4.8
zetcd-1_1 | I0925 18:38:03.802009 1 server.go:128] accepted remote connection "172.18.0.12:38574"
zetcd-2_1 | I0925 18:38:03.289081 1 pool.go:92] authresp=&{ProtocolVersion:0 TimeOut:10000 SessionID:7635112897658448644 Passwd:[136 235 95 155 191 75 53 37 196 146 143 160 130 92 205 172]}
mesos-master-3_1 | 2017-09-25 18:38:03,283:1(0x7fe7ee12d700):ZOO_INFO@log_env@726: Client environment:zookeeper.version=zookeeper C client 3.4.8
mesos-master-2_1 | 2017-09-25 18:38:03,567:1(0x7fc13ce3d700):ZOO_INFO@log_env@726: Client environment:zookeeper.version=zookeeper C client 3.4.8
mesos-master-1_1 | 2017-09-25 18:38:03,283:1(0x7f6158ff6700):ZOO_INFO@log_env@730: Client environment:host.name=ca9ede31a76c
zetcd-1_1 | I0925 18:38:03.802040 1 authconn.go:53] auth(&{ProtocolVersion:0 LastZxidSeen:0 TimeOut:10000 SessionID:0 Passwd:[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]})
zetcd-2_1 | I0925 18:38:03.289127 1 pool.go:92] authresp=&{ProtocolVersion:0 TimeOut:10000 SessionID:7635112897658448646 Passwd:[169 235 235 25 245 4 91 22 102 132 160 206 127 193 225 129]}
mesos-master-3_1 | 2017-09-25 18:38:03,283:1(0x7fe7ee12d700):ZOO_INFO@log_env@730: Client environment:host.name=60323ba49be0
mesos-master-2_1 | 2017-09-25 18:38:03,567:1(0x7fc13d63e700):ZOO_INFO@log_env@726: Client environment:zookeeper.version=zookeeper C client 3.4.8
mesos-master-1_1 | 2017-09-25 18:38:03,283:1(0x7f6158ff6700):ZOO_INFO@log_env@737: Client environment:os.name=Linux
mesos-master-3_1 | 2017-09-25 18:38:03,283:1(0x7fe7ee12d700):ZOO_INFO@log_env@737: Client environment:os.name=Linux
mesos-master-2_1 | 2017-09-25 18:38:03,567:1(0x7fc13d63e700):ZOO_INFO@log_env@730: Client environment:host.name=8a0a1462ea5d
zetcd-2_1 | I0925 18:38:03.315459 1 session.go:59] starting the session... id=7635112897658448646
mesos-master-1_1 | 2017-09-25 18:38:03,283:1(0x7f6158ff6700):ZOO_INFO@log_env@738: Client environment:os.arch=4.4.0-97-generic
mesos-master-2_1 | 2017-09-25 18:38:03,567:1(0x7fc13d63e700):ZOO_INFO@log_env@737: Client environment:os.name=Linux
zetcd-2_1 | I0925 18:38:03.315449 1 session.go:90] auth resp OK ({ProtocolVersion:0 TimeOut:10000 SessionID:98721115828125696 Passwd:[254 101 112 25 179 210 58 231 12 84 93 108 180 217 241 117]})
mesos-master-1_1 | 2017-09-25 18:38:03,283:1(0x7f61577f3700):ZOO_INFO@log_env@726: Client environment:zookeeper.version=zookeeper C client 3.4.8
mesos-master-3_1 | 2017-09-25 18:38:03,283:1(0x7fe7ee12d700):ZOO_INFO@log_env@738: Client environment:os.arch=4.4.0-97-generic
zetcd-1_1 | I0925 18:38:03.802052 1 session.go:54] newSession([zookeeper:2181])
mesos-master-2_1 | 2017-09-25 18:38:03,567:1(0x7fc13d63e700):ZOO_INFO@log_env@738: Client environment:os.arch=4.4.0-97-generic
zetcd-2_1 | I0925 18:38:03.315491 1 server.go:46] serving global serial session requests on s=&{Conn:0xc420178960 oracle:0xc420284000 candidate:0xc4201c4380 req:{ProtocolVersion:0 LastZxidSeen:0 TimeOut:10000 SessionID:0 Passwd:[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]} sp:0xc4201aa020}
mesos-master-1_1 | 2017-09-25 18:38:03,283:1(0x7f61577f3700):ZOO_INFO@log_env@730: Client environment:host.name=ca9ede31a76c
mesos-master-3_1 | 2017-09-25 18:38:03,283:1(0x7fe7ee12d700):ZOO_INFO@log_env@739: Client environment:os.version=#120-Ubuntu SMP Tue Sep 19 17:28:18 UTC 2017
zetcd-1_1 | I0925 18:38:03.802021 1 server.go:128] accepted remote connection "172.18.0.11:51616"
mesos-master-2_1 | 2017-09-25 18:38:03,567:1(0x7fc13d63e700):ZOO_INFO@log_env@739: Client environment:os.version=#120-Ubuntu SMP Tue Sep 19 17:28:18 UTC 2017
mesos-master-1_1 | 2017-09-25 18:38:03,283:1(0x7f61577f3700):ZOO_INFO@log_env@737: Client environment:os.name=Linux
zetcd-2_1 | I0925 18:38:03.315819 1 server.go:110] zkreq={xid:1506364684 req:*zetcd.ExistsRequest:&{Path:/mesos Watch:false}}
mesos-master-3_1 | 2017-09-25 18:38:03,284:1(0x7fe7ee12d700):ZOO_INFO@log_env@747: Client environment:user.name=(null)
zetcd-1_1 | I0925 18:38:03.802102 1 session.go:54] newSession([zookeeper:2181])
mesos-master-2_1 | 2017-09-25 18:38:03,567:1(0x7fc13ce3d700):ZOO_INFO@log_env@730: Client environment:host.name=8a0a1462ea5d
zetcd-2_1 | I0925 18:38:03.315860 1 zklog.go:38] Exists(1506364684,{Path:/mesos Watch:false})
mesos-master-1_1 | 2017-09-25 18:38:03,283:1(0x7f61577f3700):ZOO_INFO@log_env@738: Client environment:os.arch=4.4.0-97-generic
mesos-master-3_1 | 2017-09-25 18:38:03,284:1(0x7fe7ee12d700):ZOO_INFO@log_env@755: Client environment:user.home=/root
mesos-master-2_1 | 2017-09-25 18:38:03,567:1(0x7fc13ce3d700):ZOO_INFO@log_env@737: Client environment:os.name=Linux
zetcd-2_1 | I0925 18:38:03.315916 1 zklog.go:38] Exists(1506364684,{Path:/mesos Watch:false})
mesos-master-1_1 | 2017-09-25 18:38:03,283:1(0x7f61577f3700):ZOO_INFO@log_env@739: Client environment:os.version=#120-Ubuntu SMP Tue Sep 19 17:28:18 UTC 2017
zetcd-1_1 | I0925 18:38:03.802178 1 session.go:54] newSession([zookeeper:2181])
mesos-master-3_1 | 2017-09-25 18:38:03,284:1(0x7fe7ee12d700):ZOO_INFO@log_env@767: Client environment:user.dir=/
mesos-master-2_1 | 2017-09-25 18:38:03,567:1(0x7fc13ce3d700):ZOO_INFO@log_env@738: Client environment:os.arch=4.4.0-97-generic
zetcd-2_1 | I0925 18:38:03.315916 1 session.go:122] waitFutureSendResp=&{Path:/mesos Watch:false}
mesos-master-1_1 | 2017-09-25 18:38:03,283:1(0x7f6158ff6700):ZOO_INFO@log_env@739: Client environment:os.version=#120-Ubuntu SMP Tue Sep 19 17:28:18 UTC 2017
zetcd-1_1 | I0925 18:38:03.802132 1 authconn.go:53] auth(&{ProtocolVersion:0 LastZxidSeen:0 TimeOut:10000 SessionID:0 Passwd:[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]})
mesos-master-3_1 | 2017-09-25 18:38:03,284:1(0x7fe7ee12d700):ZOO_INFO@zookeeper_init@800: Initiating client connection, host=zetcd:2181 sessionTimeout=10000 watcher=0x7fe7f8703470 sessionId=0 sessionPasswd=<null> context=0x7fe7d4002d18 flags=0
mesos-master-2_1 | 2017-09-25 18:38:03,567:1(0x7fc13ce3d700):ZOO_INFO@log_env@739: Client environment:os.version=#120-Ubuntu SMP Tue Sep 19 17:28:18 UTC 2017
zetcd-1_1 | I0925 18:38:03.806979 1 pool.go:92] authresp=&{ProtocolVersion:0 TimeOut:10000 SessionID:7635112897658448661 Passwd:[229 37 137 157 237 40 42 1 30 182 88 36 10 179 12 114]}
mesos-master-1_1 | 2017-09-25 18:38:03,283:1(0x7f61577f3700):ZOO_INFO@log_env@747: Client environment:user.name=(null)
zetcd-2_1 | I0925 18:38:03.316147 1 session.go:59] starting the session... id=7635112897658448644
mesos-master-3_1 | I0925 18:38:03.284684 1 master.cpp:436] Master 6027c177-a18e-44af-afa7-9acfc96b9d08 (60323ba49be0) started on 172.18.0.10:5050
mesos-master-1_1 | 2017-09-25 18:38:03,283:1(0x7f61577f3700):ZOO_INFO@log_env@755: Client environment:user.home=/root
zetcd-2_1 | I0925 18:38:03.316181 1 session.go:90] auth resp OK ({ProtocolVersion:0 TimeOut:10000 SessionID:98721115828125697 Passwd:[17 135 106 25 83 30 26 44 1 151 196 151 136 98 6 67]})
zetcd-1_1 | I0925 18:38:03.808911 1 session.go:90] auth resp OK ({ProtocolVersion:0 TimeOut:10000 SessionID:98721115828125699 Passwd:[235 68 118 25 19 135 91 162 23 17 246 64 225 80 221 168]})
mesos-master-2_1 | 2017-09-25 18:38:03,568:1(0x7fc13d63e700):ZOO_INFO@log_env@747: Client environment:user.name=(null)
mesos-master-3_1 | I0925 18:38:03.284704 1 master.cpp:438] Flags at startup: --agent_ping_timeout="15secs" --agent_reregister_timeout="10mins" --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate_agents="false" --authenticate_frameworks="false" --authenticate_http_frameworks="false" --authenticate_http_readonly="false" --authenticate_http_readwrite="false" --authenticators="crammd5" --authorizers="local" --framework_sorter="drf" --help="false" --hostname_lookup="true" --http_authenticators="basic" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_agent_ping_timeouts="5" --max_completed_frameworks="50" --max_completed_tasks_per_framework="1000" --max_unreachable_tasks_per_framework="1000" --port="5050" --quiet="false" --recovery_agent_removal_limit="100%" --registry="in_memory" --registry_fetch_timeout="1mins" --registry_gc_interval="15mins" --registry_max_agent_age="2weeks" --registry_max_agent_count="102400" --registry_store_timeout="20secs" --registry_strict="false" --root_submissions="true" --user_sorter="drf" --version="false" --webui_dir="/usr/share/mesos/webui" --zk="zk://zetcd:2181/mesos" --zk_session_timeout="10secs"
mesos-master-3_1 | I0925 18:38:03.284837 1 master.cpp:490] Master allowing unauthenticated frameworks to register
mesos-master-1_1 | 2017-09-25 18:38:03,283:1(0x7f61577f3700):ZOO_INFO@log_env@767: Client environment:user.dir=/
zetcd-1_1 | I0925 18:38:03.808923 1 session.go:59] starting the session... id=7635112897658448661
zetcd-2_1 | I0925 18:38:03.316550 1 server.go:46] serving global serial session requests on s=&{Conn:0xc420260840 oracle:0xc4202840a0 candidate:0xc420240200 req:{ProtocolVersion:0 LastZxidSeen:0 TimeOut:10000 SessionID:0 Passwd:[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]} sp:0xc4201aa020}
mesos-master-2_1 | 2017-09-25 18:38:03,568:1(0x7fc13d63e700):ZOO_INFO@log_env@755: Client environment:user.home=/root
mesos-master-3_1 | I0925 18:38:03.284843 1 master.cpp:504] Master allowing unauthenticated agents to register
mesos-master-1_1 | 2017-09-25 18:38:03,283:1(0x7f61577f3700):ZOO_INFO@zookeeper_init@800: Initiating client connection, host=zetcd:2181 sessionTimeout=10000 watcher=0x7f6162dcb470 sessionId=0 sessionPasswd=<null> context=0x7f6148000b68 flags=0
mesos-master-2_1 | 2017-09-25 18:38:03,568:1(0x7fc13d63e700):ZOO_INFO@log_env@767: Client environment:user.dir=/
zetcd-1_1 | I0925 18:38:03.808971 1 server.go:46] serving global serial session requests on s=&{Conn:0xc42017ede0 oracle:0xc420154d20 candidate:0xc420190880 req:{ProtocolVersion:0 LastZxidSeen:0 TimeOut:10000 SessionID:0 Passwd:[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]} sp:0xc42017ac70}
mesos-master-3_1 | I0925 18:38:03.284848 1 master.cpp:518] Master allowing HTTP frameworks to register without authentication
zetcd-2_1 | I0925 18:38:03.320117 1 session.go:142] zk/zkresp=({Xid:1506364684 Zxid:2 Err:-101},<nil>)
mesos-master-1_1 | 2017-09-25 18:38:03,284:1(0x7f6158ff6700):ZOO_INFO@log_env@747: Client environment:user.name=(null)
mesos-master-2_1 | 2017-09-25 18:38:03,568:1(0x7fc13d63e700):ZOO_INFO@zookeeper_init@800: Initiating client connection, host=zetcd:2181 sessionTimeout=10000 watcher=0x7fc148415470 sessionId=0 sessionPasswd=<null> context=0x7fc1300017a8 flags=0
mesos-master-3_1 | I0925 18:38:03.284862 1 master.cpp:560] Using default 'crammd5' authenticator
zetcd-2_1 | I0925 18:38:03.320145 1 conn.go:131] sendXchk Xid:1506364684 ZXid:2 Resp:0xc42017c3b0
mesos-master-1_1 | 2017-09-25 18:38:03,284:1(0x7f6158ff6700):ZOO_INFO@log_env@755: Client environment:user.home=/root
mesos-master-3_1 | W0925 18:38:03.284873 1 authenticator.cpp:512] No credentials provided, authentication requests will be refused
mesos-master-1_1 | 2017-09-25 18:38:03,284:1(0x7f6158ff6700):ZOO_INFO@log_env@767: Client environment:user.dir=/
zetcd-1_1 | I0925 18:38:03.809142 1 server.go:110] zkreq={xid:1506364689 req:*zetcd.ExistsRequest:&{Path:/mesos Watch:false}}
mesos-master-2_1 | 2017-09-25 18:38:03,568:1(0x7fc13ce3d700):ZOO_INFO@log_env@747: Client environment:user.name=(null)
zetcd-2_1 | I0925 18:38:03.320547 1 conn.go:139] conn.Send(xid=1506364684, zxid=2, 0xc42017c3b0)
mesos-master-3_1 | I0925 18:38:03.284878 1 authenticator.cpp:519] Initializing server SASL
mesos-master-1_1 | 2017-09-25 18:38:03,284:1(0x7f6158ff6700):ZOO_INFO@zookeeper_init@800: Initiating client connection, host=zetcd:2181 sessionTimeout=10000 watcher=0x7f6162dcb470 sessionId=0 sessionPasswd=<null> context=0x7f6144000988 flags=0
mesos-master-2_1 | 2017-09-25 18:38:03,568:1(0x7fc13ce3d700):ZOO_INFO@log_env@755: Client environment:user.home=/root
zetcd-1_1 | I0925 18:38:03.809181 1 zklog.go:38] Exists(1506364689,{Path:/mesos Watch:false})
mesos-master-3_1 | 2017-09-25 18:38:03,284:1(0x7fe7ef12f700):ZOO_INFO@log_env@726: Client environment:zookeeper.version=zookeeper C client 3.4.8
mesos-master-1_1 | 2017-09-25 18:38:03,285:1(0x7f6154da8700):ZOO_INFO@check_events@1728: initiated connection to server [172.18.0.2:2181]
zetcd-2_1 | I0925 18:38:03.320577 1 server.go:110] zkreq={xid:1506364684 req:*zetcd.ExistsRequest:&{Path:/mesos Watch:false}}
mesos-master-2_1 | 2017-09-25 18:38:03,568:1(0x7fc13ce3d700):ZOO_INFO@log_env@767: Client environment:user.dir=/
zetcd-1_1 | I0925 18:38:03.809187 1 zklog.go:38] Exists(1506364689,{Path:/mesos Watch:false})
mesos-master-3_1 | 2017-09-25 18:38:03,284:1(0x7fe7ef12f700):ZOO_INFO@log_env@730: Client environment:host.name=60323ba49be0
zetcd-2_1 | I0925 18:38:03.320646 1 zklog.go:38] Exists(1506364684,{Path:/mesos Watch:false})
mesos-master-2_1 | 2017-09-25 18:38:03,568:1(0x7fc13ce3d700):ZOO_INFO@zookeeper_init@800: Initiating client connection, host=zetcd:2181 sessionTimeout=10000 watcher=0x7fc148415470 sessionId=0 sessionPasswd=<null> context=0x7fc1240019b8 flags=0
mesos-master-3_1 | 2017-09-25 18:38:03,284:1(0x7fe7ef12f700):ZOO_INFO@log_env@737: Client environment:os.name=Linux
mesos-master-1_1 | I0925 18:38:03.285449 1 master.cpp:436] Master 5548dcd0-ec93-4c43-b259-cc9c79f9f5a5 (ca9ede31a76c) started on 172.18.0.11:5050
zetcd-1_1 | I0925 18:38:03.809466 1 session.go:122] waitFutureSendResp=&{Path:/mesos Watch:false}
zetcd-2_1 | I0925 18:38:03.320648 1 zklog.go:38] Exists(1506364684,{Path:/mesos Watch:false})
mesos-master-3_1 | 2017-09-25 18:38:03,284:1(0x7fe7ef12f700):ZOO_INFO@log_env@738: Client environment:os.arch=4.4.0-97-generic
mesos-master-2_1 | I0925 18:38:03.569576 1 master.cpp:436] Master 309cce98-07d9-4a05-982b-478a8f435252 (8a0a1462ea5d) started on 172.18.0.12:5050
mesos-master-1_1 | I0925 18:38:03.285461 1 master.cpp:438] Flags at startup: --agent_ping_timeout="15secs" --agent_reregister_timeout="10mins" --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate_agents="false" --authenticate_frameworks="false" --authenticate_http_frameworks="false" --authenticate_http_readonly="false" --authenticate_http_readwrite="false" --authenticators="crammd5" --authorizers="local" --framework_sorter="drf" --help="false" --hostname_lookup="true" --http_authenticators="basic" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_agent_ping_timeouts="5" --max_completed_frameworks="50" --max_completed_tasks_per_framework="1000" --max_unreachable_tasks_per_framework="1000" --port="5050" --quiet="false" --recovery_agent_removal_limit="100%" --registry="in_memory" --registry_fetch_timeout="1mins" --registry_gc_interval="15mins" --registry_max_agent_age="2weeks" --registry_max_agent_count="102400" --registry_store_timeout="20secs" --registry_strict="false" --root_submissions="true" --user_sorter="drf" --version="false" --webui_dir="/usr/share/mesos/webui" --zk="zk://zetcd:2181/mesos" --zk_session_timeout="10secs"
zetcd-1_1 | I0925 18:38:03.812379 1 pool.go:92] authresp=&{ProtocolVersion:0 TimeOut:10000 SessionID:7635112897658448663 Passwd:[69 26 228 230 115 245 215 83 131 194 57 26 88 226 10 111]}
mesos-master-3_1 | 2017-09-25 18:38:03,284:1(0x7fe7ef12f700):ZOO_INFO@log_env@739: Client environment:os.version=#120-Ubuntu SMP Tue Sep 19 17:28:18 UTC 2017
mesos-master-1_1 | I0925 18:38:03.285627 1 master.cpp:490] Master allowing unauthenticated frameworks to register
mesos-master-2_1 | I0925 18:38:03.569591 1 master.cpp:438] Flags at startup: --agent_ping_timeout="15secs" --agent_reregister_timeout="10mins" --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate_agents="false" --authenticate_frameworks="false" --authenticate_http_frameworks="false" --authenticate_http_readonly="false" --authenticate_http_readwrite="false" --authenticators="crammd5" --authorizers="local" --framework_sorter="drf" --help="false" --hostname_lookup="true" --http_authenticators="basic" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_agent_ping_timeouts="5" --max_completed_frameworks="50" --max_completed_tasks_per_framework="1000" --max_unreachable_tasks_per_framework="1000" --port="5050" --quiet="false" --recovery_agent_removal_limit="100%" --registry="in_memory" --registry_fetch_timeout="1mins" --registry_gc_interval="15mins" --registry_max_agent_age="2weeks" --registry_max_agent_count="102400" --registry_store_timeout="20secs" --registry_strict="false" --root_submissions="true" --user_sorter="drf" --version="false" --webui_dir="/usr/share/mesos/webui" --zk="zk://zetcd:2181/mesos" --zk_session_timeout="10secs"
zetcd-2_1 | I0925 18:38:03.321838 1 session.go:122] waitFutureSendResp=&{Path:/mesos Watch:false}
zetcd-1_1 | I0925 18:38:03.816550 1 session.go:59] starting the session... id=7635112897658448663
mesos-master-3_1 | 2017-09-25 18:38:03,285:1(0x7fe7ef12f700):ZOO_INFO@log_env@747: Client environment:user.name=(null)
mesos-master-1_1 | I0925 18:38:03.285634 1 master.cpp:504] Master allowing unauthenticated agents to register
mesos-master-2_1 | 2017-09-25 18:38:03,569:1(0x7fc13a3f2700):ZOO_INFO@check_events@1728: initiated connection to server [172.18.0.2:2181]
zetcd-2_1 | I0925 18:38:03.322474 1 session.go:142] zk/zkresp=({Xid:1506364684 Zxid:2 Err:-101},<nil>)
mesos-master-3_1 | 2017-09-25 18:38:03,285:1(0x7fe7ef12f700):ZOO_INFO@log_env@755: Client environment:user.home=/root
mesos-master-1_1 | I0925 18:38:03.285640 1 master.cpp:518] Master allowing HTTP frameworks to register without authentication
zetcd-1_1 | I0925 18:38:03.816626 1 session.go:90] auth resp OK ({ProtocolVersion:0 TimeOut:10000 SessionID:98721115828125700 Passwd:[74 234 88 25 51 1 184 250 225 95 250 25 3 253 67 170]})
zetcd-2_1 | I0925 18:38:03.322563 1 conn.go:131] sendXchk Xid:1506364684 ZXid:2 Resp:0xc42000c270
mesos-master-2_1 | I0925 18:38:03.569718 1 master.cpp:490] Master allowing unauthenticated frameworks to register
zetcd-1_1 | I0925 18:38:03.816652 1 server.go:46] serving global serial session requests on s=&{Conn:0xc4202bc5a0 oracle:0xc4200c8b40 candidate:0xc4204fa080 req:{ProtocolVersion:0 LastZxidSeen:0 TimeOut:10000 SessionID:0 Passwd:[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]} sp:0xc42017ac70}
mesos-master-1_1 | I0925 18:38:03.285652 1 master.cpp:560] Using default 'crammd5' authenticator
mesos-master-2_1 | I0925 18:38:03.569726 1 master.cpp:504] Master allowing unauthenticated agents to register
mesos-master-3_1 | 2017-09-25 18:38:03,285:1(0x7fe7ef12f700):ZOO_INFO@log_env@767: Client environment:user.dir=/
zetcd-2_1 | I0925 18:38:03.322670 1 conn.go:139] conn.Send(xid=1506364684, zxid=2, 0xc42000c270)
zetcd-1_1 | I0925 18:38:03.816760 1 zketcd.go:285] Exists(1506364689) = (zxid=10, resp={Stat:{Czxid:3 Mzxid:3 Ctime:1506364683323773 Mtime:1506364683323773 Version:0 Cversion:2 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:2 Pzxid:7}})
mesos-master-1_1 | W0925 18:38:03.285665 1 authenticator.cpp:512] No credentials provided, authentication requests will be refused
mesos-master-2_1 | I0925 18:38:03.569732 1 master.cpp:518] Master allowing HTTP frameworks to register without authentication
mesos-master-3_1 | 2017-09-25 18:38:03,285:1(0x7fe7ef12f700):ZOO_INFO@zookeeper_init@800: Initiating client connection, host=zetcd:2181 sessionTimeout=10000 watcher=0x7fe7f8703470 sessionId=0 sessionPasswd=<null> context=0x7fe7d0000c88 flags=0
zetcd-2_1 | I0925 18:38:03.322682 1 server.go:110] zkreq={xid:1506364685 req:*zetcd.CreateRequest:&{Path:/mesos Data:[] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:0}}
zetcd-1_1 | I0925 18:38:03.816819 1 session.go:142] zk/zkresp=({Xid:1506364689 Zxid:9 Err:0},&{Stat:{Czxid:3 Mzxid:3 Ctime:1506364683324 Mtime:1506364683324 Version:0 Cversion:2 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:2 Pzxid:7}})
mesos-master-1_1 | I0925 18:38:03.285670 1 authenticator.cpp:519] Initializing server SASL
zetcd-1_1 | I0925 18:38:03.816910 1 conn.go:131] sendXchk Xid:1506364689 ZXid:9 Resp:&{Stat:{Czxid:3 Mzxid:3 Ctime:1506364683324 Mtime:1506364683324 Version:0 Cversion:2 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:2 Pzxid:7}}
mesos-master-2_1 | I0925 18:38:03.569797 1 master.cpp:560] Using default 'crammd5' authenticator
mesos-master-3_1 | 2017-09-25 18:38:03,285:1(0x7fe7ea6e0700):ZOO_INFO@check_events@1728: initiated connection to server [172.18.0.2:2181]
zetcd-2_1 | I0925 18:38:03.322742 1 zklog.go:28] Create(1506364685,{Path:/mesos Data:[] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:0})
zetcd-1_1 | I0925 18:38:03.816968 1 conn.go:139] conn.Send(xid=1506364689, zxid=9, &{Stat:{Czxid:3 Mzxid:3 Ctime:1506364683324 Mtime:1506364683324 Version:0 Cversion:2 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:2 Pzxid:7}})
mesos-master-1_1 | I0925 18:38:03.292635 13 contender.cpp:152] Joining the ZK group
mesos-master-2_1 | W0925 18:38:03.569809 1 authenticator.cpp:512] No credentials provided, authentication requests will be refused
mesos-master-3_1 | I0925 18:38:03.298512 14 contender.cpp:152] Joining the ZK group
zetcd-2_1 | I0925 18:38:03.322773 1 zklog.go:28] Create(1506364685,{Path:/mesos Data:[] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:0})
zetcd-1_1 | I0925 18:38:03.816978 1 server.go:110] zkreq={xid:1506364690 req:*zetcd.ExistsRequest:&{Path:/mesos Watch:false}}
mesos-master-2_1 | I0925 18:38:03.569816 1 authenticator.cpp:519] Initializing server SASL
mesos-master-1_1 | 2017-09-25 18:38:03,315:1(0x7f6154da8700):ZOO_INFO@check_events@1775: session establishment complete on server [172.18.0.2:2181], sessionId=0x15eba54f0c40000, negotiated timeout=10000
mesos-master-3_1 | 2017-09-25 18:38:03,316:1(0x7fe7ea6e0700):ZOO_INFO@check_events@1775: session establishment complete on server [172.18.0.2:2181], sessionId=0x15eba54f0c40001, negotiated timeout=10000
zetcd-2_1 | I0925 18:38:03.323148 1 session.go:122] waitFutureSendResp=&{Path:/mesos Data:[] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:0}
mesos-master-2_1 | 2017-09-25 18:38:03,573:1(0x7fc13a3f2700):ZOO_INFO@check_events@1775: session establishment complete on server [172.18.0.2:2181], sessionId=0x15eba54f0c40002, negotiated timeout=10000
mesos-master-1_1 | I0925 18:38:03.315672 13 group.cpp:340] Group process (zookeeper-group(2)@172.18.0.11:5050) connected to ZooKeeper
zetcd-1_1 | I0925 18:38:03.817000 1 zklog.go:38] Exists(1506364690,{Path:/mesos Watch:false})
mesos-master-3_1 | I0925 18:38:03.316395 10 group.cpp:340] Group process (zookeeper-group(1)@172.18.0.10:5050) connected to ZooKeeper
zetcd-2_1 | I0925 18:38:03.324350 1 zketcd.go:52] Create(1506364685) = (zxid=4); txnresp: {Header:cluster_id:17810356679686510411 member_id:17456590584391395829 revision:4 raft_term:2 Succeeded:true Responses:[response_put:<header:<revision:4 > > response_put:<header:<revision:4 > > response_put:<header:<revision:4 > > response_put:<header:<revision:4 > > response_put:<header:<revision:4 > > response_put:<header:<revision:4 > > response_put:<header:<revision:4 > > response_put:<header:<revision:4 > > response_put:<header:<revision:4 > > ]}
mesos-master-1_1 | I0925 18:38:03.315703 13 group.cpp:830] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
mesos-master-2_1 | I0925 18:38:03.574081 13 group.cpp:340] Group process (zookeeper-group(1)@172.18.0.12:5050) connected to ZooKeeper
mesos-master-3_1 | I0925 18:38:03.316416 10 group.cpp:830] Syncing group operations: queue size (joins, cancels, datas) = (1, 0, 0)
zetcd-1_1 | I0925 18:38:03.817228 1 zklog.go:38] Exists(1506364690,{Path:/mesos Watch:false})
zetcd-2_1 | I0925 18:38:03.329770 1 session.go:142] zk/zkresp=({Xid:1506364685 Zxid:3 Err:0},&{Path:/mesos})
mesos-master-1_1 | I0925 18:38:03.315712 13 group.cpp:418] Trying to create path '/mesos' in ZooKeeper
mesos-master-3_1 | I0925 18:38:03.316426 10 group.cpp:418] Trying to create path '/mesos' in ZooKeeper
zetcd-1_1 | I0925 18:38:03.817342 1 session.go:122] waitFutureSendResp=&{Path:/mesos Watch:false}
mesos-master-2_1 | I0925 18:38:03.574131 13 group.cpp:830] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
zetcd-2_1 | I0925 18:38:03.329825 1 conn.go:131] sendXchk Xid:1506364685 ZXid:3 Resp:&{Path:/mesos}
mesos-master-3_1 | I0925 18:38:03.349277 7 contender.cpp:268] New candidate (id='0') has entered the contest for leadership
mesos-master-2_1 | I0925 18:38:03.574146 13 group.cpp:418] Trying to create path '/mesos' in ZooKeeper
mesos-master-1_1 | I0925 18:38:03.350911 10 detector.cpp:152] Detected a new leader: (id='0')
zetcd-1_1 | I0925 18:38:03.817818 1 session.go:142] zk/zkresp=({Xid:1506364690 Zxid:9 Err:0},&{Stat:{Czxid:3 Mzxid:3 Ctime:1506364683324 Mtime:1506364683324 Version:0 Cversion:2 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:2 Pzxid:7}})
mesos-master-3_1 | 2017-09-25 18:38:03,801:1(0x7fe7e8f1c700):ZOO_INFO@check_events@1728: initiated connection to server [172.18.0.9:2181]
zetcd-2_1 | I0925 18:38:03.330013 1 conn.go:139] conn.Send(xid=1506364685, zxid=3, &{Path:/mesos})
mesos-master-2_1 | I0925 18:38:03.574291 8 contender.cpp:152] Joining the ZK group
zetcd-1_1 | I0925 18:38:03.817864 1 zketcd.go:285] Exists(1506364690) = (zxid=10, resp={Stat:{Czxid:3 Mzxid:3 Ctime:1506364683323773 Mtime:1506364683323773 Version:0 Cversion:2 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:2 Pzxid:7}})
mesos-master-1_1 | I0925 18:38:03.350978 10 group.cpp:699] Trying to get '/mesos/json.info_0000000000' in ZooKeeper
zetcd-1_1 | I0925 18:38:03.817887 1 conn.go:131] sendXchk Xid:1506364690 ZXid:9 Resp:&{Stat:{Czxid:3 Mzxid:3 Ctime:1506364683324 Mtime:1506364683324 Version:0 Cversion:2 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:2 Pzxid:7}}
zetcd-2_1 | I0925 18:38:03.330023 1 server.go:110] zkreq={xid:1506364685 req:*zetcd.CreateRequest:&{Path:/mesos Data:[] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:0}}
mesos-master-2_1 | I0925 18:38:03.582809 9 contender.cpp:268] New candidate (id='1') has entered the contest for leadership
mesos-master-3_1 | 2017-09-25 18:38:03,808:1(0x7fe7e8f1c700):ZOO_INFO@check_events@1775: session establishment complete on server [172.18.0.9:2181], sessionId=0x15eba54f0c40003, negotiated timeout=10000
mesos-master-1_1 | I0925 18:38:03.353984 7 zookeeper.cpp:262] A new leading master (UPID=master@172.18.0.10:5050) is detected
zetcd-1_1 | I0925 18:38:03.818360 1 conn.go:139] conn.Send(xid=1506364690, zxid=9, &{Stat:{Czxid:3 Mzxid:3 Ctime:1506364683324 Mtime:1506364683324 Version:0 Cversion:2 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:2 Pzxid:7}})
zetcd-2_1 | I0925 18:38:03.330069 1 zklog.go:28] Create(1506364685,{Path:/mesos Data:[] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:0})
mesos-master-1_1 | I0925 18:38:03.354035 10 master.cpp:2174] The newly elected leader is master@172.18.0.10:5050 with id 6027c177-a18e-44af-afa7-9acfc96b9d08
mesos-master-3_1 | I0925 18:38:03.809016 14 group.cpp:340] Group process (zookeeper-group(2)@172.18.0.10:5050) connected to ZooKeeper
mesos-master-2_1 | 2017-09-25 18:38:03,801:1(0x7fc139bf1700):ZOO_INFO@check_events@1728: initiated connection to server [172.18.0.9:2181]
zetcd-2_1 | I0925 18:38:03.330160 1 zklog.go:28] Create(1506364685,{Path:/mesos Data:[] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:0})
zetcd-1_1 | I0925 18:38:03.818370 1 server.go:110] zkreq={xid:1506364690 req:*zetcd.GetChildrenRequest:&{Path:/mesos Watch:true}}
mesos-master-3_1 | I0925 18:38:03.809036 14 group.cpp:830] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
mesos-master-1_1 | 2017-09-25 18:38:03,801:1(0x7f612f7fe700):ZOO_INFO@check_events@1728: initiated connection to server [172.18.0.9:2181]
mesos-master-1_1 | 2017-09-25 18:38:03,816:1(0x7f612f7fe700):ZOO_INFO@check_events@1775: session establishment complete on server [172.18.0.9:2181], sessionId=0x15eba54f0c40004, negotiated timeout=10000
mesos-master-1_1 | I0925 18:38:03.816728 13 group.cpp:340] Group process (zookeeper-group(1)@172.18.0.11:5050) connected to ZooKeeper
mesos-master-1_1 | I0925 18:38:03.816751 13 group.cpp:830] Syncing group operations: queue size (joins, cancels, datas) = (1, 0, 0)
zetcd-2_1 | I0925 18:38:03.330322 1 session.go:122] waitFutureSendResp=&{Path:/mesos Data:[] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:0}
mesos-master-3_1 | I0925 18:38:03.809041 14 group.cpp:418] Trying to create path '/mesos' in ZooKeeper
zetcd-1_1 | I0925 18:38:03.818413 1 zklog.go:63] GetChildren(1506364690,{Path:/mesos Watch:true})
zetcd-2_1 | I0925 18:38:03.334496 1 session.go:142] zk/zkresp=({Xid:1506364685 Zxid:4 Err:-110},<nil>)
mesos-master-2_1 | 2017-09-25 18:38:03,855:1(0x7fc139bf1700):ZOO_INFO@check_events@1775: session establishment complete on server [172.18.0.9:2181], sessionId=0x15eba54f0c40005, negotiated timeout=10000
zetcd-1_1 | I0925 18:38:03.818433 1 zklog.go:63] GetChildren(1506364690,{Path:/mesos Watch:true})
mesos-master-3_1 | I0925 18:38:03.819718 7 detector.cpp:152] Detected a new leader: (id='0')
zetcd-1_1 | I0925 18:38:03.818893 1 session.go:122] waitFutureSendResp=&{Path:/mesos Watch:true}
zetcd-2_1 | I0925 18:38:03.334528 1 conn.go:131] sendXchk Xid:1506364685 ZXid:4 Resp:0xc420156010
mesos-master-1_1 | I0925 18:38:03.816758 13 group.cpp:418] Trying to create path '/mesos' in ZooKeeper
mesos-master-3_1 | I0925 18:38:03.819854 7 group.cpp:699] Trying to get '/mesos/json.info_0000000000' in ZooKeeper
mesos-master-2_1 | I0925 18:38:03.856143 14 group.cpp:340] Group process (zookeeper-group(2)@172.18.0.12:5050) connected to ZooKeeper
zetcd-2_1 | I0925 18:38:03.335015 1 conn.go:139] conn.Send(xid=1506364685, zxid=4, 0xc420156010)
zetcd-1_1 | I0925 18:38:03.819304 1 session.go:142] zk/zkresp=({Xid:1506364690 Zxid:9 Err:0},&{Children:[json.info_0000000001 json.info_0000000000]})
mesos-master-1_1 | I0925 18:38:03.823451 7 contender.cpp:268] New candidate (id='2') has entered the contest for leadership
zetcd-1_1 | I0925 18:38:03.819467 1 zketcd.go:436] GetChildren(1506364690) = (zxid=10, resp={Children:[json.info_0000000000 json.info_0000000001]})
mesos-master-3_1 | I0925 18:38:03.841028 10 zookeeper.cpp:262] A new leading master (UPID=master@172.18.0.10:5050) is detected
zetcd-2_1 | I0925 18:38:03.335026 1 server.go:110] zkreq={xid:1506364686 req:*zetcd.GetChildrenRequest:&{Path:/mesos Watch:true}}
zetcd-1_1 | I0925 18:38:03.819480 1 conn.go:131] sendXchk Xid:1506364690 ZXid:9 Resp:&{Children:[json.info_0000000000 json.info_0000000001]}
mesos-master-3_1 | I0925 18:38:03.841104 10 master.cpp:2161] Elected as the leading master!
zetcd-2_1 | I0925 18:38:03.335077 1 zklog.go:63] GetChildren(1506364686,{Path:/mesos Watch:true})
zetcd-2_1 | I0925 18:38:03.335082 1 zklog.go:63] GetChildren(1506364686,{Path:/mesos Watch:true})
mesos-master-3_1 | I0925 18:38:03.841112 10 master.cpp:1700] Recovering from registrar
zetcd-1_1 | I0925 18:38:03.819494 1 conn.go:139] conn.Send(xid=1506364690, zxid=9, &{Children:[json.info_0000000000 json.info_0000000001]})
zetcd-1_1 | I0925 18:38:03.819502 1 server.go:110] zkreq={xid:1506364691 req:*zetcd.CreateRequest:&{Path:/mesos/json.info_ Data:[123 34 97 100 100 114 101 115 115 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 99 97 57 101 100 101 51 49 97 55 54 99 34 44 34 105 112 34 58 34 49 55 50 46 49 56 46 48 46 49 49 34 44 34 112 111 114 116 34 58 53 48 53 48 125 44 34 104 111 115 116 110 97 109 101 34 58 34 99 97 57 101 100 101 51 49 97 55 54 99 34 44 34 105 100 34 58 34 53 53 52 56 100 99 100 48 45 101 99 57 51 45 52 99 52 51 45 98 50 53 57 45 99 99 57 99 55 57 102 57 102 53 97 53 34 44 34 105 112 34 58 49 56 52 53 53 52 49 53 54 44 34 112 105 100 34 58 34 109 97 115 116 101 114 64 49 55 50 46 49 56 46 48 46 49 49 58 53 48 53 48 34 44 34 112 111 114 116 34 58 53 48 53 48 44 34 118 101 114 115 105 111 110 34 58 34 49 46 51 46 49 34 125] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:3}}
zetcd-2_1 | I0925 18:38:03.335272 1 session.go:122] waitFutureSendResp=&{Path:/mesos Watch:true}
mesos-master-2_1 | I0925 18:38:03.856184 14 group.cpp:830] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
mesos-slave-1_1 | 2017-09-25 18:38:04,599:1(0x7f7fc458e700):ZOO_INFO@zookeeper_init@800: Initiating client connection, host=zetcd:2181 sessionTimeout=10000 watcher=0x7f7fce2e46d0 sessionId=0 sessionPasswd=<null> context=0x7f7fac001020 flags=0
mesos-master-3_1 | I0925 18:38:03.841859 9 registrar.cpp:389] Successfully fetched the registry (0B) in 700160ns
zetcd-2_1 | I0925 18:38:03.335875 1 zketcd.go:436] GetChildren(1506364686) = (zxid=5, resp={Children:[]})
mesos-master-2_1 | I0925 18:38:03.856199 14 group.cpp:418] Trying to create path '/mesos' in ZooKeeper
mesos-master-2_1 | I0925 18:38:03.859180 13 detector.cpp:152] Detected a new leader: (id='0')
mesos-master-3_1 | I0925 18:38:03.841910 9 registrar.cpp:493] Applied 1 operations in 6821ns; attempting to update the registry
zetcd-1_1 | I0925 18:38:03.819585 1 zklog.go:28] Create(1506364691,{Path:/mesos/json.info_ Data:[123 34 97 100 100 114 101 115 115 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 99 97 57 101 100 101 51 49 97 55 54 99 34 44 34 105 112 34 58 34 49 55 50 46 49 56 46 48 46 49 49 34 44 34 112 111 114 116 34 58 53 48 53 48 125 44 34 104 111 115 116 110 97 109 101 34 58 34 99 97 57 101 100 101 51 49 97 55 54 99 34 44 34 105 100 34 58 34 53 53 52 56 100 99 100 48 45 101 99 57 51 45 52 99 52 51 45 98 50 53 57 45 99 99 57 99 55 57 102 57 102 53 97 53 34 44 34 105 112 34 58 49 56 52 53 53 52 49 53 54 44 34 112 105 100 34 58 34 109 97 115 116 101 114 64 49 55 50 46 49 56 46 48 46 49 49 58 53 48 53 48 34 44 34 112 111 114 116 34 58 53 48 53 48 44 34 118 101 114 115 105 111 110 34 58 34 49 46 51 46 49 34 125] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:3})
zetcd-2_1 | I0925 18:38:03.336624 1 session.go:142] zk/zkresp=({Xid:1506364686 Zxid:4 Err:0},&{Children:[]})
mesos-slave-1_1 | I0925 18:38:04.599918 174 slave.cpp:199] Flags at startup: --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http_readonly="false" --authenticate_http_readwrite="false" --authenticatee="crammd5" --authentication_backoff_factor="1secs" --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="docker,mesos" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="2mins" --docker_store_dir="/tmp/mesos/store/docker" --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="5mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/mesos/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --http_authenticators="basic" --http_command_executor="false" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/usr/libexec/mesos" --logbufsecs="0" --logging_level="INFO" --master="zk://zetcd:2181/mesos" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --port="5051" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="1secs" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/data"
mesos-master-2_1 | I0925 18:38:03.859254 10 group.cpp:699] Trying to get '/mesos/json.info_0000000000' in ZooKeeper
zetcd-1_1 | I0925 18:38:03.819584 1 zklog.go:28] Create(1506364691,{Path:/mesos/json.info_ Data:[123 34 97 100 100 114 101 115 115 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 99 97 57 101 100 101 51 49 97 55 54 99 34 44 34 105 112 34 58 34 49 55 50 46 49 56 46 48 46 49 49 34 44 34 112 111 114 116 34 58 53 48 53 48 125 44 34 104 111 115 116 110 97 109 101 34 58 34 99 97 57 101 100 101 51 49 97 55 54 99 34 44 34 105 100 34 58 34 53 53 52 56 100 99 100 48 45 101 99 57 51 45 52 99 52 51 45 98 50 53 57 45 99 99 57 99 55 57 102 57 102 53 97 53 34 44 34 105 112 34 58 49 56 52 53 53 52 49 53 54 44 34 112 105 100 34 58 34 109 97 115 116 101 114 64 49 55 50 46 49 56 46 48 46 49 49 58 53 48 53 48 34 44 34 112 111 114 116 34 58 53 48 53 48 44 34 118 101 114 115 105 111 110 34 58 34 49 46 51 46 49 34 125] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:3})
mesos-master-3_1 | I0925 18:38:03.842614 9 registrar.cpp:550] Successfully updated the registry in 690176ns
zetcd-2_1 | I0925 18:38:03.336642 1 conn.go:131] sendXchk Xid:1506364686 ZXid:4 Resp:&{Children:[]}
mesos-master-2_1 | I0925 18:38:03.861359 12 zookeeper.cpp:262] A new leading master (UPID=master@172.18.0.10:5050) is detected
mesos-master-3_1 | I0925 18:38:03.842640 9 registrar.cpp:422] Successfully recovered registrar
zetcd-1_1 | I0925 18:38:03.819845 1 session.go:122] waitFutureSendResp=&{Path:/mesos/json.info_ Data:[123 34 97 100 100 114 101 115 115 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 99 97 57 101 100 101 51 49 97 55 54 99 34 44 34 105 112 34 58 34 49 55 50 46 49 56 46 48 46 49 49 34 44 34 112 111 114 116 34 58 53 48 53 48 125 44 34 104 111 115 116 110 97 109 101 34 58 34 99 97 57 101 100 101 51 49 97 55 54 99 34 44 34 105 100 34 58 34 53 53 52 56 100 99 100 48 45 101 99 57 51 45 52 99 52 51 45 98 50 53 57 45 99 99 57 99 55 57 102 57 102 53 97 53 34 44 34 105 112 34 58 49 56 52 53 53 52 49 53 54 44 34 112 105 100 34 58 34 109 97 115 116 101 114 64 49 55 50 46 49 56 46 48 46 49 49 58 53 48 53 48 34 44 34 112 111 114 116 34 58 53 48 53 48 44 34 118 101 114 115 105 111 110 34 58 34 49 46 51 46 49 34 125] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:3}
zetcd-2_1 | I0925 18:38:03.336860 1 conn.go:139] conn.Send(xid=1506364686, zxid=4, &{Children:[]})
mesos-master-3_1 | I0925 18:38:03.842733 9 master.cpp:1799] Recovered 0 agents from the registry (128B); allowing 10mins for agents to re-register
mesos-master-2_1 | I0925 18:38:03.861413 14 master.cpp:2174] The newly elected leader is master@172.18.0.10:5050 with id 6027c177-a18e-44af-afa7-9acfc96b9d08
zetcd-1_1 | I0925 18:38:03.820583 1 zketcd.go:430] WatchChild (1506364690,11,{Type:4 State:3 Path:/mesos})
zetcd-2_1 | I0925 18:38:03.336989 1 server.go:110] zkreq={xid:1506364686 req:*zetcd.CreateRequest:&{Path:/mesos/json.info_ Data:[123 34 97 100 100 114 101 115 115 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 112 34 58 34 49 55 50 46 49 56 46 48 46 49 48 34 44 34 112 111 114 116 34 58 53 48 53 48 125 44 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 100 34 58 34 54 48 50 55 99 49 55 55 45 97 49 56 101 45 52 52 97 102 45 97 102 97 55 45 57 97 99 102 99 57 54 98 57 100 48 56 34 44 34 105 112 34 58 49 54 55 55 55 54 57 52 48 44 34 112 105 100 34 58 34 109 97 115 116 101 114 64 49 55 50 46 49 56 46 48 46 49 48 58 53 48 53 48 34 44 34 112 111 114 116 34 58 53 48 53 48 44 34 118 101 114 115 105 111 110 34 58 34 49 46 51 46 49 34 125] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:3}}
mesos-master-3_1 | I0925 18:38:05.239614 7 master.cpp:5429] Received register agent message from slave(1)@172.18.0.6:5051 (762a975061aa)
zetcd-2_1 | I0925 18:38:03.337703 1 zklog.go:28] Create(1506364686,{Path:/mesos/json.info_ Data:[123 34 97 100 100 114 101 115 115 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 112 34 58 34 49 55 50 46 49 56 46 48 46 49 48 34 44 34 112 111 114 116 34 58 53 48 53 48 125 44 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 100 34 58 34 54 48 50 55 99 49 55 55 45 97 49 56 101 45 52 52 97 102 45 97 102 97 55 45 57 97 99 102 99 57 54 98 57 100 48 56 34 44 34 105 112 34 58 49 54 55 55 55 54 57 52 48 44 34 112 105 100 34 58 34 109 97 115 116 101 114 64 49 55 50 46 49 56 46 48 46 49 48 58 53 48 53 48 34 44 34 112 111 114 116 34 58 53 48 53 48 44 34 118 101 114 115 105 111 110 34 58 34 49 46 51 46 49 34 125] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:3})
zetcd-2_1 | I0925 18:38:03.337710 1 zklog.go:28] Create(1506364686,{Path:/mesos/json.info_ Data:[123 34 97 100 100 114 101 115 115 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 112 34 58 34 49 55 50 46 49 56 46 48 46 49 48 34 44 34 112 111 114 116 34 58 53 48 53 48 125 44 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 100 34 58 34 54 48 50 55 99 49 55 55 45 97 49 56 101 45 52 52 97 102 45 97 102 97 55 45 57 97 99 102 99 57 54 98 57 100 48 56 34 44 34 105 112 34 58 49 54 55 55 55 54 57 52 48 44 34 112 105 100 34 58 34 109 97 115 116 101 114 64 49 55 50 46 49 56 46 48 46 49 48 58 53 48 53 48 34 44 34 112 111 114 116 34 58 53 48 53 48 44 34 118 101 114 115 105 111 110 34 58 34 49 46 51 46 49 34 125] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:3})
zetcd-1_1 | I0925 18:38:03.820597 1 conn.go:164] connWorkerSend(-1,-1,&{Type:4 State:3 Path:/mesos})
mesos-master-3_1 | I0925 18:38:05.239792 7 master.cpp:5564] Registering agent at slave(1)@172.18.0.6:5051 (762a975061aa) with id 6027c177-a18e-44af-afa7-9acfc96b9d08-S0
zetcd-2_1 | I0925 18:38:03.338110 1 session.go:122] waitFutureSendResp=&{Path:/mesos/json.info_ Data:[123 34 97 100 100 114 101 115 115 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 112 34 58 34 49 55 50 46 49 56 46 48 46 49 48 34 44 34 112 111 114 116 34 58 53 48 53 48 125 44 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 100 34 58 34 54 48 50 55 99 49 55 55 45 97 49 56 101 45 52 52 97 102 45 97 102 97 55 45 57 97 99 102 99 57 54 98 57 100 48 56 34 44 34 105 112 34 58 49 54 55 55 55 54 57 52 48 44 34 112 105 100 34 58 34 109 97 115 116 101 114 64 49 55 50 46 49 56 46 48 46 49 48 58 53 48 53 48 34 44 34 112 111 114 116 34 58 53 48 53 48 44 34 118 101 114 115 105 111 110 34 58 34 49 46 51 46 49 34 125] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:3}
zetcd-1_1 | I0925 18:38:03.820615 1 zketcd.go:52] Create(1506364691) = (zxid=11); txnresp: {Header:cluster_id:17810356679686510411 member_id:17456590584391395829 revision:11 raft_term:2 Succeeded:true Responses:[response_put:<header:<revision:11 > > response_put:<header:<revision:11 > > response_put:<header:<revision:11 > > response_put:<header:<revision:11 > > response_put:<header:<revision:11 > > response_put:<header:<revision:11 > > response_put:<header:<revision:11 > > response_put:<header:<revision:11 > > response_put:<header:<revision:11 > > ]}
mesos-master-3_1 | I0925 18:38:05.239883 12 registrar.cpp:493] Applied 1 operations in 17423ns; attempting to update the registry
zetcd-2_1 | I0925 18:38:03.338887 1 zketcd.go:52] Create(1506364686) = (zxid=6); txnresp: {Header:cluster_id:17810356679686510411 member_id:17456590584391395829 revision:6 raft_term:2 Succeeded:true Responses:[response_put:<header:<revision:6 > > response_put:<header:<revision:6 > > response_put:<header:<revision:6 > > response_put:<header:<revision:6 > > response_put:<header:<revision:6 > > response_put:<header:<revision:6 > > response_put:<header:<revision:6 > > response_put:<header:<revision:6 > > response_put:<header:<revision:6 > > ]}
zetcd-1_1 | I0925 18:38:03.822357 1 session.go:142] zk/zkresp=({Xid:-1 Zxid:-1 Err:0},&{Type:4 State:3 Path:/mesos})
mesos-master-3_1 | I0925 18:38:05.240620 12 registrar.cpp:550] Successfully updated the registry in 720896ns
zetcd-2_1 | I0925 18:38:03.338947 1 zketcd.go:430] WatchChild (1506364686,6,{Type:4 State:3 Path:/mesos})
zetcd-1_1 | I0925 18:38:03.822368 1 session.go:160] zk/zkSessOOB=&{Xid:-1 Zxid:-1 Err:0} &{Type:4 State:3 Path:/mesos}
mesos-master-3_1 | I0925 18:38:05.240911 10 master.cpp:5639] Registered agent 6027c177-a18e-44af-afa7-9acfc96b9d08-S0 at slave(1)@172.18.0.6:5051 (762a975061aa) with cpus(*):8; mem(*):63105; disk(*):203883; ports(*):[31000-32000]
mesos-master-3_1 | I0925 18:38:05.240969 13 hierarchical.cpp:525] Added agent 6027c177-a18e-44af-afa7-9acfc96b9d08-S0 (762a975061aa) with cpus(*):8; mem(*):63105; disk(*):203883; ports(*):[31000-32000] (allocated: {})
zetcd-2_1 | I0925 18:38:03.338964 1 conn.go:164] connWorkerSend(-1,-1,&{Type:4 State:3 Path:/mesos})
zetcd-1_1 | I0925 18:38:03.822376 1 conn.go:164] connWorkerSend(-1,-1,&{Type:4 State:3 Path:/mesos})
mesos-master-3_1 | I0925 18:38:05.241611 11 master.cpp:6324] Received update of agent 6027c177-a18e-44af-afa7-9acfc96b9d08-S0 at slave(1)@172.18.0.6:5051 (762a975061aa) with total oversubscribed resources {}
zetcd-1_1 | I0925 18:38:03.822405 1 conn.go:112] xchkSendOOB response {Type:4 State:3 Path:/mesos}
zetcd-1_1 | I0925 18:38:03.822443 1 session.go:142] zk/zkresp=({Xid:1506364691 Zxid:10 Err:0},&{Path:/mesos/json.info_0000000002})
zetcd-2_1 | I0925 18:38:03.348575 1 session.go:142] zk/zkresp=({Xid:-1 Zxid:-1 Err:0},&{Type:4 State:3 Path:/mesos})
zetcd-2_1 | I0925 18:38:03.348591 1 session.go:160] zk/zkSessOOB=&{Xid:-1 Zxid:-1 Err:0} &{Type:4 State:3 Path:/mesos}
zetcd-2_1 | I0925 18:38:03.348603 1 session.go:142] zk/zkresp=({Xid:1506364686 Zxid:5 Err:0},&{Path:/mesos/json.info_0000000000})
zetcd-1_1 | I0925 18:38:03.822723 1 conn.go:131] sendXchk Xid:1506364691 ZXid:10 Resp:&{Path:/mesos/json.info_0000000002}
zetcd-2_1 | I0925 18:38:03.348620 1 conn.go:131] sendXchk Xid:1506364686 ZXid:5 Resp:&{Path:/mesos/json.info_0000000000}
zetcd-1_1 | I0925 18:38:03.822911 1 conn.go:139] conn.Send(xid=-1, zxid=-1, &{Type:4 State:3 Path:/mesos})
zetcd-2_1 | I0925 18:38:03.348598 1 conn.go:164] connWorkerSend(-1,-1,&{Type:4 State:3 Path:/mesos})
zetcd-2_1 | I0925 18:38:03.348651 1 conn.go:112] xchkSendOOB response {Type:4 State:3 Path:/mesos}
zetcd-1_1 | I0925 18:38:03.823245 1 conn.go:139] conn.Send(xid=1506364691, zxid=10, &{Path:/mesos/json.info_0000000002})
zetcd-1_1 | I0925 18:38:03.824315 1 server.go:110] zkreq={xid:1506364691 req:*zetcd.GetDataRequest:&{Path:/mesos/json.info_0000000000 Watch:false}}
zetcd-1_1 | I0925 18:38:03.824381 1 zklog.go:43] GetData(1506364691,{Path:/mesos/json.info_0000000000 Watch:false})
zetcd-2_1 | I0925 18:38:03.349098 1 conn.go:139] conn.Send(xid=1506364686, zxid=5, &{Path:/mesos/json.info_0000000000})
zetcd-2_1 | I0925 18:38:03.349162 1 conn.go:139] conn.Send(xid=-1, zxid=-1, &{Type:4 State:3 Path:/mesos})
zetcd-1_1 | I0925 18:38:03.824930 1 zklog.go:43] GetData(1506364691,{Path:/mesos/json.info_0000000000 Watch:false})
zetcd-1_1 | I0925 18:38:03.825001 1 session.go:122] waitFutureSendResp=&{Path:/mesos/json.info_0000000000 Watch:false}
zetcd-2_1 | I0925 18:38:03.349475 1 server.go:110] zkreq={xid:1506364687 req:*zetcd.GetChildrenRequest:&{Path:/mesos Watch:true}}
zetcd-1_1 | I0925 18:38:03.825994 1 session.go:142] zk/zkresp=({Xid:1506364691 Zxid:10 Err:0},&{Data:[123 34 97 100 100 114 101 115 115 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 112 34 58 34 49 55 50 46 49 56 46 48 46 49 48 34 44 34 112 111 114 116 34 58 53 48 53 48 125 44 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 100 34 58 34 54 48 50 55 99 49 55 55 45 97 49 56 101 45 52 52 97 102 45 97 102 97 55 45 57 97 99 102 99 57 54 98 57 100 48 56 34 44 34 105 112 34 58 49 54 55 55 55 54 57 52 48 44 34 112 105 100 34 58 34 109 97 115 116 101 114 64 49 55 50 46 49 56 46 48 46 49 48 58 53 48 53 48 34 44 34 112 111 114 116 34 58 53 48 53 48 44 34 118 101 114 115 105 111 110 34 58 34 49 46 51 46 49 34 125] Stat:{Czxid:5 Mzxid:5 Ctime:1506364683338 Mtime:1506364683338 Version:0 Cversion:0 Aversion:0 EphemeralOwner:98721115828125697 DataLength:217 NumChildren:0 Pzxid:5}})
zetcd-2_1 | I0925 18:38:03.349504 1 zklog.go:63] GetChildren(1506364687,{Path:/mesos Watch:true})
zetcd-1_1 | I0925 18:38:03.839862 1 zketcd.go:320] GetData(1506364691) = (zxid=11, resp={Data:[123 34 97 100 100 114 101 115 115 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 112 34 58 34 49 55 50 46 49 56 46 48 46 49 48 34 44 34 112 111 114 116 34 58 53 48 53 48 125 44 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 100 34 58 34 54 48 50 55 99 49 55 55 45 97 49 56 101 45 52 52 97 102 45 97 102 97 55 45 57 97 99 102 99 57 54 98 57 100 48 56 34 44 34 105 112 34 58 49 54 55 55 55 54 57 52 48 44 34 112 105 100 34 58 34 109 97 115 116 101 114 64 49 55 50 46 49 56 46 48 46 49 48 58 53 48 53 48 34 44 34 112 111 114 116 34 58 53 48 53 48 44 34 118 101 114 115 105 111 110 34 58 34 49 46 51 46 49 34 125] Stat:{Czxid:5 Mzxid:5 Ctime:1506364683338367 Mtime:1506364683338367 Version:0 Cversion:0 Aversion:0 EphemeralOwner:7635112897658448644 DataLength:217 NumChildren:0 Pzxid:5}})
zetcd-2_1 | I0925 18:38:03.349542 1 zklog.go:63] GetChildren(1506364687,{Path:/mesos Watch:true})
zetcd-2_1 | I0925 18:38:03.349700 1 session.go:122] waitFutureSendResp=&{Path:/mesos Watch:true}
zetcd-2_1 | I0925 18:38:03.350121 1 session.go:142] zk/zkresp=({Xid:1506364687 Zxid:5 Err:0},&{Children:[json.info_0000000000]})
zetcd-2_1 | I0925 18:38:03.350259 1 zketcd.go:436] GetChildren(1506364687) = (zxid=6, resp={Children:[json.info_0000000000]})
zetcd-1_1 | W0925 18:38:03.839920 1 zk.go:318] xchk failed (stat mismatch)
zetcd-2_1 | I0925 18:38:03.350271 1 conn.go:131] sendXchk Xid:1506364687 ZXid:5 Resp:&{Children:[json.info_0000000000]}
zetcd-1_1 | candidate: &{Data:[123 34 97 100 100 114 101 115 115 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 112 34 58 34 49 55 50 46 49 56 46 48 46 49 48 34 44 34 112 111 114 116 34 58 53 48 53 48 125 44 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 100 34 58 34 54 48 50 55 99 49 55 55 45 97 49 56 101 45 52 52 97 102 45 97 102 97 55 45 57 97 99 102 99 57 54 98 57 100 48 56 34 44 34 105 112 34 58 49 54 55 55 55 54 57 52 48 44 34 112 105 100 34 58 34 109 97 115 116 101 114 64 49 55 50 46 49 56 46 48 46 49 48 58 53 48 53 48 34 44 34 112 111 114 116 34 58 53 48 53 48 44 34 118 101 114 115 105 111 110 34 58 34 49 46 51 46 49 34 125] Stat:{Czxid:5 Mzxid:5 Ctime:1506364683338367 Mtime:1506364683338367 Version:0 Cversion:0 Aversion:0 EphemeralOwner:7635112897658448644 DataLength:217 NumChildren:0 Pzxid:5}}
zetcd-2_1 | I0925 18:38:03.350279 1 conn.go:139] conn.Send(xid=1506364687, zxid=5, &{Children:[json.info_0000000000]})
zetcd-1_1 | oracle: &{Data:[123 34 97 100 100 114 101 115 115 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 112 34 58 34 49 55 50 46 49 56 46 48 46 49 48 34 44 34 112 111 114 116 34 58 53 48 53 48 125 44 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 100 34 58 34 54 48 50 55 99 49 55 55 45 97 49 56 101 45 52 52 97 102 45 97 102 97 55 45 57 97 99 102 99 57 54 98 57 100 48 56 34 44 34 105 112 34 58 49 54 55 55 55 54 57 52 48 44 34 112 105 100 34 58 34 109 97 115 116 101 114 64 49 55 50 46 49 56 46 48 46 49 48 58 53 48 53 48 34 44 34 112 111 114 116 34 58 53 48 53 48 44 34 118 101 114 115 105 111 110 34 58 34 49 46 51 46 49 34 125] Stat:{Czxid:5 Mzxid:5 Ctime:1506364683338 Mtime:1506364683338 Version:0 Cversion:0 Aversion:0 EphemeralOwner:98721115828125697 DataLength:217 NumChildren:0 Pzxid:5}}
zetcd-2_1 | I0925 18:38:03.350287 1 server.go:110] zkreq={xid:1506364687 req:*zetcd.GetChildrenRequest:&{Path:/mesos Watch:true}}
zetcd-1_1 | I0925 18:38:03.840015 1 conn.go:131] sendXchk Xid:1506364691 ZXid:10 Resp:&{Data:[123 34 97 100 100 114 101 115 115 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 112 34 58 34 49 55 50 46 49 56 46 48 46 49 48 34 44 34 112 111 114 116 34 58 53 48 53 48 125 44 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 100 34 58 34 54 48 50 55 99 49 55 55 45 97 49 56 101 45 52 52 97 102 45 97 102 97 55 45 57 97 99 102 99 57 54 98 57 100 48 56 34 44 34 105 112 34 58 49 54 55 55 55 54 57 52 48 44 34 112 105 100 34 58 34 109 97 115 116 101 114 64 49 55 50 46 49 56 46 48 46 49 48 58 53 48 53 48 34 44 34 112 111 114 116 34 58 53 48 53 48 44 34 118 101 114 115 105 111 110 34 58 34 49 46 51 46 49 34 125] Stat:{Czxid:5 Mzxid:5 Ctime:1506364683338 Mtime:1506364683338 Version:0 Cversion:0 Aversion:0 EphemeralOwner:98721115828125697 DataLength:217 NumChildren:0 Pzxid:5}}
zetcd-2_1 | I0925 18:38:03.350320 1 zklog.go:63] GetChildren(1506364687,{Path:/mesos Watch:true})
zetcd-2_1 | I0925 18:38:03.350321 1 zklog.go:63] GetChildren(1506364687,{Path:/mesos Watch:true})
zetcd-2_1 | I0925 18:38:03.350480 1 session.go:122] waitFutureSendResp=&{Path:/mesos Watch:true}
zetcd-1_1 | I0925 18:38:03.840597 1 conn.go:139] conn.Send(xid=1506364691, zxid=10, &{Data:[123 34 97 100 100 114 101 115 115 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 112 34 58 34 49 55 50 46 49 56 46 48 46 49 48 34 44 34 112 111 114 116 34 58 53 48 53 48 125 44 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 100 34 58 34 54 48 50 55 99 49 55 55 45 97 49 56 101 45 52 52 97 102 45 97 102 97 55 45 57 97 99 102 99 57 54 98 57 100 48 56 34 44 34 105 112 34 58 49 54 55 55 55 54 57 52 48 44 34 112 105 100 34 58 34 109 97 115 116 101 114 64 49 55 50 46 49 56 46 48 46 49 48 58 53 48 53 48 34 44 34 112 111 114 116 34 58 53 48 53 48 44 34 118 101 114 115 105 111 110 34 58 34 49 46 51 46 49 34 125] Stat:{Czxid:5 Mzxid:5 Ctime:1506364683338 Mtime:1506364683338 Version:0 Cversion:0 Aversion:0 EphemeralOwner:98721115828125697 DataLength:217 NumChildren:0 Pzxid:5}})
zetcd-2_1 | I0925 18:38:03.350709 1 zketcd.go:436] GetChildren(1506364687) = (zxid=6, resp={Children:[json.info_0000000000]})
zetcd-1_1 | I0925 18:38:03.840803 1 server.go:110] zkreq={xid:1506364693 req:*zetcd.GetChildrenRequest:&{Path:/mesos Watch:true}}
zetcd-2_1 | I0925 18:38:03.350749 1 session.go:142] zk/zkresp=({Xid:1506364687 Zxid:5 Err:0},&{Children:[json.info_0000000000]})
zetcd-2_1 | I0925 18:38:03.350778 1 conn.go:131] sendXchk Xid:1506364687 ZXid:5 Resp:&{Children:[json.info_0000000000]}
zetcd-2_1 | I0925 18:38:03.350788 1 conn.go:139] conn.Send(xid=1506364687, zxid=5, &{Children:[json.info_0000000000]})
zetcd-1_1 | I0925 18:38:03.840893 1 zklog.go:63] GetChildren(1506364693,{Path:/mesos Watch:true})
zetcd-2_1 | I0925 18:38:03.351070 1 server.go:110] zkreq={xid:1506364688 req:*zetcd.GetDataRequest:&{Path:/mesos/json.info_0000000000 Watch:false}}
zetcd-1_1 | I0925 18:38:03.840974 1 zklog.go:63] GetChildren(1506364693,{Path:/mesos Watch:true})
zetcd-1_1 | I0925 18:38:03.841229 1 session.go:122] waitFutureSendResp=&{Path:/mesos Watch:true}
zetcd-2_1 | I0925 18:38:03.351093 1 zklog.go:43] GetData(1506364688,{Path:/mesos/json.info_0000000000 Watch:false})
zetcd-2_1 | I0925 18:38:03.351096 1 zklog.go:43] GetData(1506364688,{Path:/mesos/json.info_0000000000 Watch:false})
zetcd-1_1 | I0925 18:38:03.841671 1 session.go:142] zk/zkresp=({Xid:1506364693 Zxid:10 Err:0},&{Children:[json.info_0000000001 json.info_0000000002 json.info_0000000000]})
zetcd-1_1 | I0925 18:38:03.841970 1 zketcd.go:436] GetChildren(1506364693) = (zxid=11, resp={Children:[json.info_0000000000 json.info_0000000001 json.info_0000000002]})
zetcd-2_1 | I0925 18:38:03.351390 1 zketcd.go:320] GetData(1506364688) = (zxid=6, resp={Data:[123 34 97 100 100 114 101 115 115 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 112 34 58 34 49 55 50 46 49 56 46 48 46 49 48 34 44 34 112 111 114 116 34 58 53 48 53 48 125 44 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 100 34 58 34 54 48 50 55 99 49 55 55 45 97 49 56 101 45 52 52 97 102 45 97 102 97 55 45 57 97 99 102 99 57 54 98 57 100 48 56 34 44 34 105 112 34 58 49 54 55 55 55 54 57 52 48 44 34 112 105 100 34 58 34 109 97 115 116 101 114 64 49 55 50 46 49 56 46 48 46 49 48 58 53 48 53 48 34 44 34 112 111 114 116 34 58 53 48 53 48 44 34 118 101 114 115 105 111 110 34 58 34 49 46 51 46 49 34 125] Stat:{Czxid:5 Mzxid:5 Ctime:1506364683338367 Mtime:1506364683338367 Version:0 Cversion:0 Aversion:0 EphemeralOwner:7635112897658448644 DataLength:217 NumChildren:0 Pzxid:5}})
zetcd-2_1 | I0925 18:38:03.351662 1 session.go:122] waitFutureSendResp=&{Path:/mesos/json.info_0000000000 Watch:false}
zetcd-2_1 | I0925 18:38:03.352866 1 session.go:142] zk/zkresp=({Xid:1506364688 Zxid:5 Err:0},&{Data:[123 34 97 100 100 114 101 115 115 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 112 34 58 34 49 55 50 46 49 56 46 48 46 49 48 34 44 34 112 111 114 116 34 58 53 48 53 48 125 44 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 100 34 58 34 54 48 50 55 99 49 55 55 45 97 49 56 101 45 52 52 97 102 45 97 102 97 55 45 57 97 99 102 99 57 54 98 57 100 48 56 34 44 34 105 112 34 58 49 54 55 55 55 54 57 52 48 44 34 112 105 100 34 58 34 109 97 115 116 101 114 64 49 55 50 46 49 56 46 48 46 49 48 58 53 48 53 48 34 44 34 112 111 114 116 34 58 53 48 53 48 44 34 118 101 114 115 105 111 110 34 58 34 49 46 51 46 49 34 125] Stat:{Czxid:5 Mzxid:5 Ctime:1506364683338 Mtime:1506364683338 Version:0 Cversion:0 Aversion:0 EphemeralOwner:98721115828125697 DataLength:217 NumChildren:0 Pzxid:5}})
zetcd-2_1 | I0925 18:38:03.352912 1 conn.go:131] sendXchk Xid:1506364688 ZXid:5 Resp:&{Data:[123 34 97 100 100 114 101 115 115 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 112 34 58 34 49 55 50 46 49 56 46 48 46 49 48 34 44 34 112 111 114 116 34 58 53 48 53 48 125 44 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 100 34 58 34 54 48 50 55 99 49 55 55 45 97 49 56 101 45 52 52 97 102 45 97 102 97 55 45 57 97 99 102 99 57 54 98 57 100 48 56 34 44 34 105 112 34 58 49 54 55 55 55 54 57 52 48 44 34 112 105 100 34 58 34 109 97 115 116 101 114 64 49 55 50 46 49 56 46 48 46 49 48 58 53 48 53 48 34 44 34 112 111 114 116 34 58 53 48 53 48 44 34 118 101 114 115 105 111 110 34 58 34 49 46 51 46 49 34 125] Stat:{Czxid:5 Mzxid:5 Ctime:1506364683338 Mtime:1506364683338 Version:0 Cversion:0 Aversion:0 EphemeralOwner:98721115828125697 DataLength:217 NumChildren:0 Pzxid:5}}
zetcd-2_1 | I0925 18:38:03.353594 1 conn.go:139] conn.Send(xid=1506364688, zxid=5, &{Data:[123 34 97 100 100 114 101 115 115 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 112 34 58 34 49 55 50 46 49 56 46 48 46 49 48 34 44 34 112 111 114 116 34 58 53 48 53 48 125 44 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 100 34 58 34 54 48 50 55 99 49 55 55 45 97 49 56 101 45 52 52 97 102 45 97 102 97 55 45 57 97 99 102 99 57 54 98 57 100 48 56 34 44 34 105 112 34 58 49 54 55 55 55 54 57 52 48 44 34 112 105 100 34 58 34 109 97 115 116 101 114 64 49 55 50 46 49 56 46 48 46 49 48 58 53 48 53 48 34 44 34 112 111 114 116 34 58 53 48 53 48 44 34 118 101 114 115 105 111 110 34 58 34 49 46 51 46 49 34 125] Stat:{Czxid:5 Mzxid:5 Ctime:1506364683338 Mtime:1506364683338 Version:0 Cversion:0 Aversion:0 EphemeralOwner:98721115828125697 DataLength:217 NumChildren:0 Pzxid:5}})
zetcd-1_1 | I0925 18:38:03.841982 1 conn.go:131] sendXchk Xid:1506364693 ZXid:10 Resp:&{Children:[json.info_0000000000 json.info_0000000001 json.info_0000000002]}
zetcd-2_1 | I0925 18:38:03.569702 1 server.go:128] accepted remote connection "172.18.0.12:51296"
zetcd-1_1 | I0925 18:38:03.841991 1 conn.go:139] conn.Send(xid=1506364693, zxid=10, &{Children:[json.info_0000000000 json.info_0000000001 json.info_0000000002]})
zetcd-2_1 | I0925 18:38:03.569763 1 session.go:54] newSession([zookeeper:2181])
zetcd-1_1 | I0925 18:38:03.841997 1 server.go:110] zkreq={xid:1506364693 req:*zetcd.GetChildrenRequest:&{Path:/mesos Watch:true}}
zetcd-1_1 | I0925 18:38:03.842015 1 zklog.go:63] GetChildren(1506364693,{Path:/mesos Watch:true})
zetcd-2_1 | I0925 18:38:03.569759 1 authconn.go:53] auth(&{ProtocolVersion:0 LastZxidSeen:0 TimeOut:10000 SessionID:0 Passwd:[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]})
zetcd-1_1 | I0925 18:38:03.842069 1 zklog.go:63] GetChildren(1506364693,{Path:/mesos Watch:true})
zetcd-2_1 | I0925 18:38:03.572248 1 pool.go:92] authresp=&{ProtocolVersion:0 TimeOut:10000 SessionID:7635112897658448656 Passwd:[219 244 158 207 57 213 101 54 202 209 220 137 45 126 44 222]}
zetcd-1_1 | I0925 18:38:03.842229 1 session.go:122] waitFutureSendResp=&{Path:/mesos Watch:true}
zetcd-2_1 | I0925 18:38:03.573934 1 session.go:59] starting the session... id=7635112897658448656
zetcd-2_1 | I0925 18:38:03.573942 1 session.go:90] auth resp OK ({ProtocolVersion:0 TimeOut:10000 SessionID:98721115828125698 Passwd:[216 35 124 25 115 59 124 93 34 206 142 21 13 200 200 219]})
zetcd-2_1 | I0925 18:38:03.574038 1 server.go:46] serving global serial session requests on s=&{Conn:0xc420062d20 oracle:0xc420248320 candidate:0xc420240300 req:{ProtocolVersion:0 LastZxidSeen:0 TimeOut:10000 SessionID:0 Passwd:[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]} sp:0xc4201aa020}
zetcd-1_1 | I0925 18:38:03.842524 1 zketcd.go:436] GetChildren(1506364693) = (zxid=11, resp={Children:[json.info_0000000000 json.info_0000000001 json.info_0000000002]})
zetcd-1_1 | I0925 18:38:03.842595 1 session.go:142] zk/zkresp=({Xid:1506364693 Zxid:10 Err:0},&{Children:[json.info_0000000001 json.info_0000000002 json.info_0000000000]})
zetcd-2_1 | I0925 18:38:03.574657 1 server.go:110] zkreq={xid:1506364684 req:*zetcd.ExistsRequest:&{Path:/mesos Watch:false}}
zetcd-1_1 | I0925 18:38:03.842675 1 conn.go:131] sendXchk Xid:1506364693 ZXid:10 Resp:&{Children:[json.info_0000000000 json.info_0000000001 json.info_0000000002]}
zetcd-1_1 | I0925 18:38:03.842691 1 conn.go:139] conn.Send(xid=1506364693, zxid=10, &{Children:[json.info_0000000000 json.info_0000000001 json.info_0000000002]})
zetcd-1_1 | I0925 18:38:03.851582 1 authconn.go:53] auth(&{ProtocolVersion:0 LastZxidSeen:0 TimeOut:10000 SessionID:0 Passwd:[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]})
zetcd-2_1 | I0925 18:38:03.574700 1 zklog.go:38] Exists(1506364684,{Path:/mesos Watch:false})
zetcd-2_1 | I0925 18:38:03.574705 1 zklog.go:38] Exists(1506364684,{Path:/mesos Watch:false})
zetcd-2_1 | I0925 18:38:03.574905 1 session.go:122] waitFutureSendResp=&{Path:/mesos Watch:false}
zetcd-2_1 | I0925 18:38:03.575120 1 zketcd.go:285] Exists(1506364684) = (zxid=7, resp={Stat:{Czxid:3 Mzxid:3 Ctime:1506364683323773 Mtime:1506364683323773 Version:0 Cversion:1 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:1 Pzxid:5}})
zetcd-1_1 | I0925 18:38:03.854149 1 pool.go:92] authresp=&{ProtocolVersion:0 TimeOut:10000 SessionID:7635112897658448669 Passwd:[26 166 126 140 128 88 190 77 191 25 100 43 213 199 132 6]}
zetcd-2_1 | I0925 18:38:03.575796 1 session.go:142] zk/zkresp=({Xid:1506364684 Zxid:6 Err:0},&{Stat:{Czxid:3 Mzxid:3 Ctime:1506364683324 Mtime:1506364683324 Version:0 Cversion:1 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:1 Pzxid:5}})
zetcd-2_1 | I0925 18:38:03.575834 1 conn.go:131] sendXchk Xid:1506364684 ZXid:6 Resp:&{Stat:{Czxid:3 Mzxid:3 Ctime:1506364683324 Mtime:1506364683324 Version:0 Cversion:1 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:1 Pzxid:5}}
zetcd-1_1 | I0925 18:38:03.855934 1 session.go:59] starting the session... id=7635112897658448669
zetcd-1_1 | I0925 18:38:03.855937 1 session.go:90] auth resp OK ({ProtocolVersion:0 TimeOut:10000 SessionID:98721115828125701 Passwd:[93 11 83 25 211 76 151 63 214 162 97 69 214 133 88 119]})
zetcd-1_1 | I0925 18:38:03.855974 1 server.go:46] serving global serial session requests on s=&{Conn:0xc42020aba0 oracle:0xc420154000 candidate:0xc420064380 req:{ProtocolVersion:0 LastZxidSeen:0 TimeOut:10000 SessionID:0 Passwd:[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]} sp:0xc42017ac70}
zetcd-2_1 | I0925 18:38:03.576379 1 conn.go:139] conn.Send(xid=1506364684, zxid=6, &{Stat:{Czxid:3 Mzxid:3 Ctime:1506364683324 Mtime:1506364683324 Version:0 Cversion:1 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:1 Pzxid:5}})
zetcd-2_1 | I0925 18:38:03.576745 1 server.go:110] zkreq={xid:1506364685 req:*zetcd.GetChildrenRequest:&{Path:/mesos Watch:true}}
zetcd-1_1 | I0925 18:38:03.856307 1 server.go:110] zkreq={xid:1506364689 req:*zetcd.ExistsRequest:&{Path:/mesos Watch:false}}
zetcd-2_1 | I0925 18:38:03.576774 1 zklog.go:63] GetChildren(1506364685,{Path:/mesos Watch:true})
zetcd-2_1 | I0925 18:38:03.576792 1 zklog.go:63] GetChildren(1506364685,{Path:/mesos Watch:true})
zetcd-1_1 | I0925 18:38:03.856348 1 zklog.go:38] Exists(1506364689,{Path:/mesos Watch:false})
zetcd-1_1 | I0925 18:38:03.856362 1 zklog.go:38] Exists(1506364689,{Path:/mesos Watch:false})
zetcd-2_1 | I0925 18:38:03.576966 1 session.go:122] waitFutureSendResp=&{Path:/mesos Watch:true}
zetcd-2_1 | I0925 18:38:03.577331 1 session.go:142] zk/zkresp=({Xid:1506364685 Zxid:6 Err:0},&{Children:[json.info_0000000000]})
zetcd-1_1 | I0925 18:38:03.856773 1 session.go:122] waitFutureSendResp=&{Path:/mesos Watch:false}
zetcd-2_1 | I0925 18:38:03.577369 1 zketcd.go:436] GetChildren(1506364685) = (zxid=7, resp={Children:[json.info_0000000000]})
zetcd-2_1 | I0925 18:38:03.577383 1 conn.go:131] sendXchk Xid:1506364685 ZXid:6 Resp:&{Children:[json.info_0000000000]}
zetcd-1_1 | I0925 18:38:03.857050 1 zketcd.go:285] Exists(1506364689) = (zxid=12, resp={Stat:{Czxid:3 Mzxid:3 Ctime:1506364683323773 Mtime:1506364683323773 Version:0 Cversion:3 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:3 Pzxid:10}})
zetcd-2_1 | I0925 18:38:03.577555 1 conn.go:139] conn.Send(xid=1506364685, zxid=6, &{Children:[json.info_0000000000]})
zetcd-2_1 | I0925 18:38:03.577779 1 server.go:110] zkreq={xid:1506364686 req:*zetcd.CreateRequest:&{Path:/mesos/json.info_ Data:[123 34 97 100 100 114 101 115 115 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 56 97 48 97 49 52 54 50 101 97 53 100 34 44 34 105 112 34 58 34 49 55 50 46 49 56 46 48 46 49 50 34 44 34 112 111 114 116 34 58 53 48 53 48 125 44 34 104 111 115 116 110 97 109 101 34 58 34 56 97 48 97 49 52 54 50 101 97 53 100 34 44 34 105 100 34 58 34 51 48 57 99 99 101 57 56 45 48 55 100 57 45 52 97 48 53 45 57 56 50 98 45 52 55 56 97 56 102 52 51 53 50 53 50 34 44 34 105 112 34 58 50 48 49 51 51 49 51 55 50 44 34 112 105 100 34 58 34 109 97 115 116 101 114 64 49 55 50 46 49 56 46 48 46 49 50 58 53 48 53 48 34 44 34 112 111 114 116 34 58 53 48 53 48 44 34 118 101 114 115 105 111 110 34 58 34 49 46 51 46 49 34 125] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:3}}
zetcd-2_1 | I0925 18:38:03.577837 1 zklog.go:28] Create(1506364686,{Path:/mesos/json.info_ Data:[123 34 97 100 100 114 101 115 115 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 56 97 48 97 49 52 54 50 101 97 53 100 34 44 34 105 112 34 58 34 49 55 50 46 49 56 46 48 46 49 50 34 44 34 112 111 114 116 34 58 53 48 53 48 125 44 34 104 111 115 116 110 97 109 101 34 58 34 56 97 48 97 49 52 54 50 101 97 53 100 34 44 34 105 100 34 58 34 51 48 57 99 99 101 57 56 45 48 55 100 57 45 52 97 48 53 45 57 56 50 98 45 52 55 56 97 56 102 52 51 53 50 53 50 34 44 34 105 112 34 58 50 48 49 51 51 49 51 55 50 44 34 112 105 100 34 58 34 109 97 115 116 101 114 64 49 55 50 46 49 56 46 48 46 49 50 58 53 48 53 48 34 44 34 112 111 114 116 34 58 53 48 53 48 44 34 118 101 114 115 105 111 110 34 58 34 49 46 51 46 49 34 125] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:3})
zetcd-2_1 | I0925 18:38:03.577853 1 zklog.go:28] Create(1506364686,{Path:/mesos/json.info_ Data:[123 34 97 100 100 114 101 115 115 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 56 97 48 97 49 52 54 50 101 97 53 100 34 44 34 105 112 34 58 34 49 55 50 46 49 56 46 48 46 49 50 34 44 34 112 111 114 116 34 58 53 48 53 48 125 44 34 104 111 115 116 110 97 109 101 34 58 34 56 97 48 97 49 52 54 50 101 97 53 100 34 44 34 105 100 34 58 34 51 48 57 99 99 101 57 56 45 48 55 100 57 45 52 97 48 53 45 57 56 50 98 45 52 55 56 97 56 102 52 51 53 50 53 50 34 44 34 105 112 34 58 50 48 49 51 51 49 51 55 50 44 34 112 105 100 34 58 34 109 97 115 116 101 114 64 49 55 50 46 49 56 46 48 46 49 50 58 53 48 53 48 34 44 34 112 111 114 116 34 58 53 48 53 48 44 34 118 101 114 115 105 111 110 34 58 34 49 46 51 46 49 34 125] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:3})
zetcd-2_1 | I0925 18:38:03.578256 1 session.go:122] waitFutureSendResp=&{Path:/mesos/json.info_ Data:[123 34 97 100 100 114 101 115 115 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 56 97 48 97 49 52 54 50 101 97 53 100 34 44 34 105 112 34 58 34 49 55 50 46 49 56 46 48 46 49 50 34 44 34 112 111 114 116 34 58 53 48 53 48 125 44 34 104 111 115 116 110 97 109 101 34 58 34 56 97 48 97 49 52 54 50 101 97 53 100 34 44 34 105 100 34 58 34 51 48 57 99 99 101 57 56 45 48 55 100 57 45 52 97 48 53 45 57 56 50 98 45 52 55 56 97 56 102 52 51 53 50 53 50 34 44 34 105 112 34 58 50 48 49 51 51 49 51 55 50 44 34 112 105 100 34 58 34 109 97 115 116 101 114 64 49 55 50 46 49 56 46 48 46 49 50 58 53 48 53 48 34 44 34 112 111 114 116 34 58 53 48 53 48 44 34 118 101 114 115 105 111 110 34 58 34 49 46 51 46 49 34 125] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:3}
zetcd-1_1 | I0925 18:38:03.857150 1 session.go:142] zk/zkresp=({Xid:1506364689 Zxid:11 Err:0},&{Stat:{Czxid:3 Mzxid:3 Ctime:1506364683324 Mtime:1506364683324 Version:0 Cversion:3 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:3 Pzxid:10}})
zetcd-1_1 | I0925 18:38:03.857181 1 conn.go:131] sendXchk Xid:1506364689 ZXid:11 Resp:&{Stat:{Czxid:3 Mzxid:3 Ctime:1506364683324 Mtime:1506364683324 Version:0 Cversion:3 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:3 Pzxid:10}}
zetcd-2_1 | I0925 18:38:03.579968 1 zketcd.go:430] WatchChild (1506364687,8,{Type:4 State:3 Path:/mesos})
zetcd-2_1 | I0925 18:38:03.579880 1 zketcd.go:52] Create(1506364686) = (zxid=8); txnresp: {Header:cluster_id:17810356679686510411 member_id:17456590584391395829 revision:8 raft_term:2 Succeeded:true Responses:[response_put:<header:<revision:8 > > response_put:<header:<revision:8 > > response_put:<header:<revision:8 > > response_put:<header:<revision:8 > > response_put:<header:<revision:8 > > response_put:<header:<revision:8 > > response_put:<header:<revision:8 > > response_put:<header:<revision:8 > > response_put:<header:<revision:8 > > ]}
zetcd-1_1 | I0925 18:38:03.857344 1 conn.go:139] conn.Send(xid=1506364689, zxid=11, &{Stat:{Czxid:3 Mzxid:3 Ctime:1506364683324 Mtime:1506364683324 Version:0 Cversion:3 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:3 Pzxid:10}})
zetcd-2_1 | I0925 18:38:03.579999 1 zketcd.go:430] WatchChild (1506364687,8,{Type:4 State:3 Path:/mesos})
zetcd-2_1 | I0925 18:38:03.579982 1 conn.go:164] connWorkerSend(-1,-1,&{Type:4 State:3 Path:/mesos})
zetcd-2_1 | I0925 18:38:03.580022 1 zketcd.go:430] WatchChild (1506364685,8,{Type:4 State:3 Path:/mesos})
zetcd-2_1 | I0925 18:38:03.580011 1 conn.go:164] connWorkerSend(-1,-1,&{Type:4 State:3 Path:/mesos})
zetcd-2_1 | I0925 18:38:03.580032 1 conn.go:164] connWorkerSend(-1,-1,&{Type:4 State:3 Path:/mesos})
zetcd-2_1 | I0925 18:38:03.580693 1 session.go:142] zk/zkresp=({Xid:-1 Zxid:-1 Err:0},&{Type:4 State:3 Path:/mesos})
zetcd-2_1 | I0925 18:38:03.580707 1 session.go:160] zk/zkSessOOB=&{Xid:-1 Zxid:-1 Err:0} &{Type:4 State:3 Path:/mesos}
zetcd-1_1 | I0925 18:38:03.857536 1 server.go:110] zkreq={xid:1506364690 req:*zetcd.GetChildrenRequest:&{Path:/mesos Watch:true}}
zetcd-2_1 | I0925 18:38:03.580713 1 conn.go:164] connWorkerSend(-1,-1,&{Type:4 State:3 Path:/mesos})
zetcd-2_1 | I0925 18:38:03.580747 1 conn.go:112] xchkSendOOB response {Type:4 State:3 Path:/mesos}
zetcd-1_1 | I0925 18:38:03.857558 1 zklog.go:63] GetChildren(1506364690,{Path:/mesos Watch:true})
zetcd-1_1 | I0925 18:38:03.857571 1 zklog.go:63] GetChildren(1506364690,{Path:/mesos Watch:true})
zetcd-2_1 | I0925 18:38:03.580785 1 session.go:142] zk/zkresp=({Xid:-1 Zxid:-1 Err:0},&{Type:4 State:3 Path:/mesos})
zetcd-2_1 | I0925 18:38:03.580799 1 session.go:160] zk/zkSessOOB=&{Xid:-1 Zxid:-1 Err:0} &{Type:4 State:3 Path:/mesos}
zetcd-2_1 | I0925 18:38:03.580809 1 conn.go:164] connWorkerSend(-1,-1,&{Type:4 State:3 Path:/mesos})
zetcd-1_1 | I0925 18:38:03.857806 1 session.go:122] waitFutureSendResp=&{Path:/mesos Watch:true}
zetcd-2_1 | I0925 18:38:03.580851 1 conn.go:112] xchkSendOOB response {Type:4 State:3 Path:/mesos}
zetcd-2_1 | I0925 18:38:03.580847 1 session.go:142] zk/zkresp=({Xid:-1 Zxid:-1 Err:0},&{Type:4 State:3 Path:/mesos})
zetcd-2_1 | I0925 18:38:03.580864 1 session.go:160] zk/zkSessOOB=&{Xid:-1 Zxid:-1 Err:0} &{Type:4 State:3 Path:/mesos}
zetcd-1_1 | I0925 18:38:03.858166 1 session.go:142] zk/zkresp=({Xid:1506364690 Zxid:11 Err:0},&{Children:[json.info_0000000001 json.info_0000000002 json.info_0000000000]})
zetcd-2_1 | I0925 18:38:03.580876 1 conn.go:164] connWorkerSend(-1,-1,&{Type:4 State:3 Path:/mesos})
zetcd-2_1 | I0925 18:38:03.580938 1 conn.go:112] xchkSendOOB response {Type:4 State:3 Path:/mesos}
zetcd-2_1 | I0925 18:38:03.580981 1 conn.go:139] conn.Send(xid=-1, zxid=-1, &{Type:4 State:3 Path:/mesos})
zetcd-2_1 | I0925 18:38:03.581037 1 session.go:142] zk/zkresp=({Xid:1506364686 Zxid:7 Err:0},&{Path:/mesos/json.info_0000000001})
zetcd-1_1 | I0925 18:38:03.858744 1 zketcd.go:436] GetChildren(1506364690) = (zxid=12, resp={Children:[json.info_0000000000 json.info_0000000001 json.info_0000000002]})
zetcd-1_1 | I0925 18:38:03.858759 1 conn.go:131] sendXchk Xid:1506364690 ZXid:11 Resp:&{Children:[json.info_0000000000 json.info_0000000001 json.info_0000000002]}
zetcd-1_1 | I0925 18:38:03.858958 1 conn.go:139] conn.Send(xid=1506364690, zxid=11, &{Children:[json.info_0000000000 json.info_0000000001 json.info_0000000002]})
zetcd-2_1 | I0925 18:38:03.581079 1 conn.go:131] sendXchk Xid:1506364686 ZXid:7 Resp:&{Path:/mesos/json.info_0000000001}
zetcd-1_1 | I0925 18:38:03.859381 1 server.go:110] zkreq={xid:1506364691 req:*zetcd.GetDataRequest:&{Path:/mesos/json.info_0000000000 Watch:false}}
zetcd-1_1 | I0925 18:38:03.859445 1 zklog.go:43] GetData(1506364691,{Path:/mesos/json.info_0000000000 Watch:false})
zetcd-2_1 | I0925 18:38:03.581564 1 conn.go:139] conn.Send(xid=-1, zxid=-1, &{Type:4 State:3 Path:/mesos})
zetcd-2_1 | I0925 18:38:03.581552 1 conn.go:139] conn.Send(xid=-1, zxid=-1, &{Type:4 State:3 Path:/mesos})
zetcd-1_1 | I0925 18:38:03.859653 1 zklog.go:43] GetData(1506364691,{Path:/mesos/json.info_0000000000 Watch:false})
zetcd-2_1 | I0925 18:38:03.581626 1 conn.go:139] conn.Send(xid=1506364686, zxid=7, &{Path:/mesos/json.info_0000000001})
zetcd-1_1 | I0925 18:38:03.859697 1 session.go:122] waitFutureSendResp=&{Path:/mesos/json.info_0000000000 Watch:false}
zetcd-2_1 | I0925 18:38:03.581779 1 server.go:110] zkreq={xid:1506364688 req:*zetcd.GetChildrenRequest:&{Path:/mesos Watch:true}}
zetcd-2_1 | I0925 18:38:03.581840 1 zklog.go:63] GetChildren(1506364688,{Path:/mesos Watch:true})
zetcd-2_1 | I0925 18:38:03.581843 1 zklog.go:63] GetChildren(1506364688,{Path:/mesos Watch:true})
zetcd-1_1 | I0925 18:38:03.860037 1 zketcd.go:320] GetData(1506364691) = (zxid=12, resp={Data:[123 34 97 100 100 114 101 115 115 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 112 34 58 34 49 55 50 46 49 56 46 48 46 49 48 34 44 34 112 111 114 116 34 58 53 48 53 48 125 44 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 100 34 58 34 54 48 50 55 99 49 55 55 45 97 49 56 101 45 52 52 97 102 45 97 102 97 55 45 57 97 99 102 99 57 54 98 57 100 48 56 34 44 34 105 112 34 58 49 54 55 55 55 54 57 52 48 44 34 112 105 100 34 58 34 109 97 115 116 101 114 64 49 55 50 46 49 56 46 48 46 49 48 58 53 48 53 48 34 44 34 112 111 114 116 34 58 53 48 53 48 44 34 118 101 114 115 105 111 110 34 58 34 49 46 51 46 49 34 125] Stat:{Czxid:5 Mzxid:5 Ctime:1506364683338367 Mtime:1506364683338367 Version:0 Cversion:0 Aversion:0 EphemeralOwner:7635112897658448644 DataLength:217 NumChildren:0 Pzxid:5}})
zetcd-1_1 | I0925 18:38:03.860506 1 session.go:142] zk/zkresp=({Xid:1506364691 Zxid:11 Err:0},&{Data:[123 34 97 100 100 114 101 115 115 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 112 34 58 34 49 55 50 46 49 56 46 48 46 49 48 34 44 34 112 111 114 116 34 58 53 48 53 48 125 44 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 100 34 58 34 54 48 50 55 99 49 55 55 45 97 49 56 101 45 52 52 97 102 45 97 102 97 55 45 57 97 99 102 99 57 54 98 57 100 48 56 34 44 34 105 112 34 58 49 54 55 55 55 54 57 52 48 44 34 112 105 100 34 58 34 109 97 115 116 101 114 64 49 55 50 46 49 56 46 48 46 49 48 58 53 48 53 48 34 44 34 112 111 114 116 34 58 53 48 53 48 44 34 118 101 114 115 105 111 110 34 58 34 49 46 51 46 49 34 125] Stat:{Czxid:5 Mzxid:5 Ctime:1506364683338 Mtime:1506364683338 Version:0 Cversion:0 Aversion:0 EphemeralOwner:98721115828125697 DataLength:217 NumChildren:0 Pzxid:5}})
zetcd-2_1 | I0925 18:38:03.582491 1 session.go:122] waitFutureSendResp=&{Path:/mesos Watch:true}
zetcd-2_1 | I0925 18:38:03.582988 1 session.go:142] zk/zkresp=({Xid:1506364688 Zxid:7 Err:0},&{Children:[json.info_0000000001 json.info_0000000000]})
zetcd-2_1 | I0925 18:38:03.583240 1 zketcd.go:436] GetChildren(1506364688) = (zxid=8, resp={Children:[json.info_0000000000 json.info_0000000001]})
zetcd-2_1 | I0925 18:38:03.583256 1 conn.go:131] sendXchk Xid:1506364688 ZXid:7 Resp:&{Children:[json.info_0000000000 json.info_0000000001]}
zetcd-2_1 | I0925 18:38:03.583267 1 conn.go:139] conn.Send(xid=1506364688, zxid=7, &{Children:[json.info_0000000000 json.info_0000000001]})
zetcd-1_1 | W0925 18:38:03.860941 1 zk.go:318] xchk failed (stat mismatch)
zetcd-2_1 | I0925 18:38:03.583274 1 server.go:110] zkreq={xid:1506364689 req:*zetcd.GetChildrenRequest:&{Path:/mesos Watch:true}}
zetcd-1_1 | candidate: &{Data:[123 34 97 100 100 114 101 115 115 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 112 34 58 34 49 55 50 46 49 56 46 48 46 49 48 34 44 34 112 111 114 116 34 58 53 48 53 48 125 44 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 100 34 58 34 54 48 50 55 99 49 55 55 45 97 49 56 101 45 52 52 97 102 45 97 102 97 55 45 57 97 99 102 99 57 54 98 57 100 48 56 34 44 34 105 112 34 58 49 54 55 55 55 54 57 52 48 44 34 112 105 100 34 58 34 109 97 115 116 101 114 64 49 55 50 46 49 56 46 48 46 49 48 58 53 48 53 48 34 44 34 112 111 114 116 34 58 53 48 53 48 44 34 118 101 114 115 105 111 110 34 58 34 49 46 51 46 49 34 125] Stat:{Czxid:5 Mzxid:5 Ctime:1506364683338367 Mtime:1506364683338367 Version:0 Cversion:0 Aversion:0 EphemeralOwner:7635112897658448644 DataLength:217 NumChildren:0 Pzxid:5}}
zetcd-2_1 | I0925 18:38:03.583312 1 zklog.go:63] GetChildren(1506364689,{Path:/mesos Watch:true})
zetcd-1_1 | oracle: &{Data:[123 34 97 100 100 114 101 115 115 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 112 34 58 34 49 55 50 46 49 56 46 48 46 49 48 34 44 34 112 111 114 116 34 58 53 48 53 48 125 44 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 100 34 58 34 54 48 50 55 99 49 55 55 45 97 49 56 101 45 52 52 97 102 45 97 102 97 55 45 57 97 99 102 99 57 54 98 57 100 48 56 34 44 34 105 112 34 58 49 54 55 55 55 54 57 52 48 44 34 112 105 100 34 58 34 109 97 115 116 101 114 64 49 55 50 46 49 56 46 48 46 49 48 58 53 48 53 48 34 44 34 112 111 114 116 34 58 53 48 53 48 44 34 118 101 114 115 105 111 110 34 58 34 49 46 51 46 49 34 125] Stat:{Czxid:5 Mzxid:5 Ctime:1506364683338 Mtime:1506364683338 Version:0 Cversion:0 Aversion:0 EphemeralOwner:98721115828125697 DataLength:217 NumChildren:0 Pzxid:5}}
zetcd-2_1 | I0925 18:38:03.583348 1 zklog.go:63] GetChildren(1506364689,{Path:/mesos Watch:true})
zetcd-1_1 | I0925 18:38:03.861026 1 conn.go:131] sendXchk Xid:1506364691 ZXid:11 Resp:&{Data:[123 34 97 100 100 114 101 115 115 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 112 34 58 34 49 55 50 46 49 56 46 48 46 49 48 34 44 34 112 111 114 116 34 58 53 48 53 48 125 44 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 100 34 58 34 54 48 50 55 99 49 55 55 45 97 49 56 101 45 52 52 97 102 45 97 102 97 55 45 57 97 99 102 99 57 54 98 57 100 48 56 34 44 34 105 112 34 58 49 54 55 55 55 54 57 52 48 44 34 112 105 100 34 58 34 109 97 115 116 101 114 64 49 55 50 46 49 56 46 48 46 49 48 58 53 48 53 48 34 44 34 112 111 114 116 34 58 53 48 53 48 44 34 118 101 114 115 105 111 110 34 58 34 49 46 51 46 49 34 125] Stat:{Czxid:5 Mzxid:5 Ctime:1506364683338 Mtime:1506364683338 Version:0 Cversion:0 Aversion:0 EphemeralOwner:98721115828125697 DataLength:217 NumChildren:0 Pzxid:5}}
zetcd-2_1 | I0925 18:38:03.583532 1 session.go:122] waitFutureSendResp=&{Path:/mesos Watch:true}
zetcd-1_1 | I0925 18:38:03.861074 1 conn.go:139] conn.Send(xid=1506364691, zxid=11, &{Data:[123 34 97 100 100 114 101 115 115 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 112 34 58 34 49 55 50 46 49 56 46 48 46 49 48 34 44 34 112 111 114 116 34 58 53 48 53 48 125 44 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 100 34 58 34 54 48 50 55 99 49 55 55 45 97 49 56 101 45 52 52 97 102 45 97 102 97 55 45 57 97 99 102 99 57 54 98 57 100 48 56 34 44 34 105 112 34 58 49 54 55 55 55 54 57 52 48 44 34 112 105 100 34 58 34 109 97 115 116 101 114 64 49 55 50 46 49 56 46 48 46 49 48 58 53 48 53 48 34 44 34 112 111 114 116 34 58 53 48 53 48 44 34 118 101 114 115 105 111 110 34 58 34 49 46 51 46 49 34 125] Stat:{Czxid:5 Mzxid:5 Ctime:1506364683338 Mtime:1506364683338 Version:0 Cversion:0 Aversion:0 EphemeralOwner:98721115828125697 DataLength:217 NumChildren:0 Pzxid:5}})
zetcd-2_1 | I0925 18:38:03.583952 1 zketcd.go:436] GetChildren(1506364689) = (zxid=8, resp={Children:[json.info_0000000000 json.info_0000000001]})
zetcd-2_1 | I0925 18:38:03.584119 1 session.go:142] zk/zkresp=({Xid:1506364689 Zxid:7 Err:0},&{Children:[json.info_0000000001 json.info_0000000000]})
zetcd-2_1 | I0925 18:38:03.584152 1 conn.go:131] sendXchk Xid:1506364689 ZXid:7 Resp:&{Children:[json.info_0000000000 json.info_0000000001]}
zetcd-2_1 | I0925 18:38:03.584167 1 conn.go:139] conn.Send(xid=1506364689, zxid=7, &{Children:[json.info_0000000000 json.info_0000000001]})
zetcd-2_1 | I0925 18:38:03.584176 1 server.go:110] zkreq={xid:1506364687 req:*zetcd.GetChildrenRequest:&{Path:/mesos Watch:true}}
zetcd-2_1 | I0925 18:38:03.584190 1 zklog.go:63] GetChildren(1506364687,{Path:/mesos Watch:true})
zetcd-2_1 | I0925 18:38:03.584247 1 zklog.go:63] GetChildren(1506364687,{Path:/mesos Watch:true})
zetcd-2_1 | I0925 18:38:03.584387 1 session.go:122] waitFutureSendResp=&{Path:/mesos Watch:true}
zetcd-2_1 | I0925 18:38:03.584730 1 zketcd.go:436] GetChildren(1506364687) = (zxid=8, resp={Children:[json.info_0000000000 json.info_0000000001]})
zetcd-2_1 | I0925 18:38:03.584783 1 session.go:142] zk/zkresp=({Xid:1506364687 Zxid:7 Err:0},&{Children:[json.info_0000000001 json.info_0000000000]})
zetcd-2_1 | I0925 18:38:03.584818 1 conn.go:131] sendXchk Xid:1506364687 ZXid:7 Resp:&{Children:[json.info_0000000000 json.info_0000000001]}
zetcd-2_1 | I0925 18:38:03.585340 1 conn.go:139] conn.Send(xid=1506364687, zxid=7, &{Children:[json.info_0000000000 json.info_0000000001]})
zetcd-2_1 | I0925 18:38:03.820835 1 zketcd.go:430] WatchChild (1506364689,11,{Type:4 State:3 Path:/mesos})
zetcd-2_1 | I0925 18:38:03.820852 1 zketcd.go:430] WatchChild (1506364688,11,{Type:4 State:3 Path:/mesos})
zetcd-2_1 | I0925 18:38:03.820871 1 zketcd.go:430] WatchChild (1506364687,11,{Type:4 State:3 Path:/mesos})
zetcd-2_1 | I0925 18:38:03.820849 1 conn.go:164] connWorkerSend(-1,-1,&{Type:4 State:3 Path:/mesos})
zetcd-2_1 | I0925 18:38:03.820887 1 conn.go:164] connWorkerSend(-1,-1,&{Type:4 State:3 Path:/mesos})
zetcd-2_1 | I0925 18:38:03.820863 1 conn.go:164] connWorkerSend(-1,-1,&{Type:4 State:3 Path:/mesos})
zetcd-2_1 | I0925 18:38:03.822172 1 session.go:142] zk/zkresp=({Xid:-1 Zxid:-1 Err:0},&{Type:4 State:3 Path:/mesos})
zetcd-2_1 | I0925 18:38:03.822182 1 session.go:160] zk/zkSessOOB=&{Xid:-1 Zxid:-1 Err:0} &{Type:4 State:3 Path:/mesos}
zetcd-2_1 | I0925 18:38:03.822187 1 conn.go:164] connWorkerSend(-1,-1,&{Type:4 State:3 Path:/mesos})
zetcd-2_1 | I0925 18:38:03.822206 1 conn.go:112] xchkSendOOB response {Type:4 State:3 Path:/mesos}
zetcd-2_1 | I0925 18:38:03.822208 1 session.go:142] zk/zkresp=({Xid:-1 Zxid:-1 Err:0},&{Type:4 State:3 Path:/mesos})
zetcd-2_1 | I0925 18:38:03.822222 1 session.go:160] zk/zkSessOOB=&{Xid:-1 Zxid:-1 Err:0} &{Type:4 State:3 Path:/mesos}
zetcd-2_1 | I0925 18:38:03.822235 1 conn.go:164] connWorkerSend(-1,-1,&{Type:4 State:3 Path:/mesos})
zetcd-2_1 | I0925 18:38:03.822225 1 conn.go:139] conn.Send(xid=-1, zxid=-1, &{Type:4 State:3 Path:/mesos})
zetcd-2_1 | I0925 18:38:03.822248 1 conn.go:112] xchkSendOOB response {Type:4 State:3 Path:/mesos}
zetcd-2_1 | I0925 18:38:03.822289 1 session.go:142] zk/zkresp=({Xid:-1 Zxid:-1 Err:0},&{Type:4 State:3 Path:/mesos})
zetcd-2_1 | I0925 18:38:03.822302 1 session.go:160] zk/zkSessOOB=&{Xid:-1 Zxid:-1 Err:0} &{Type:4 State:3 Path:/mesos}
zetcd-2_1 | I0925 18:38:03.822311 1 conn.go:164] connWorkerSend(-1,-1,&{Type:4 State:3 Path:/mesos})
zetcd-2_1 | I0925 18:38:03.822350 1 conn.go:112] xchkSendOOB response {Type:4 State:3 Path:/mesos}
zetcd-2_1 | I0925 18:38:03.822362 1 conn.go:139] conn.Send(xid=-1, zxid=-1, &{Type:4 State:3 Path:/mesos})
zetcd-2_1 | I0925 18:38:03.822676 1 server.go:110] zkreq={xid:1506364688 req:*zetcd.GetChildrenRequest:&{Path:/mesos Watch:true}}
zetcd-2_1 | I0925 18:38:03.822746 1 zklog.go:63] GetChildren(1506364688,{Path:/mesos Watch:true})
zetcd-2_1 | I0925 18:38:03.822792 1 zklog.go:63] GetChildren(1506364688,{Path:/mesos Watch:true})
zetcd-2_1 | I0925 18:38:03.822928 1 conn.go:139] conn.Send(xid=-1, zxid=-1, &{Type:4 State:3 Path:/mesos})
zetcd-2_1 | I0925 18:38:03.823976 1 session.go:122] waitFutureSendResp=&{Path:/mesos Watch:true}
zetcd-2_1 | I0925 18:38:03.824506 1 zketcd.go:436] GetChildren(1506364688) = (zxid=11, resp={Children:[json.info_0000000000 json.info_0000000001 json.info_0000000002]})
zetcd-2_1 | I0925 18:38:03.825105 1 session.go:142] zk/zkresp=({Xid:1506364688 Zxid:10 Err:0},&{Children:[json.info_0000000001 json.info_0000000002 json.info_0000000000]})
zetcd-2_1 | I0925 18:38:03.825164 1 conn.go:131] sendXchk Xid:1506364688 ZXid:10 Resp:&{Children:[json.info_0000000000 json.info_0000000001 json.info_0000000002]}
zetcd-2_1 | I0925 18:38:03.825181 1 conn.go:139] conn.Send(xid=1506364688, zxid=10, &{Children:[json.info_0000000000 json.info_0000000001 json.info_0000000002]})
zetcd-2_1 | I0925 18:38:03.825191 1 server.go:110] zkreq={xid:1506364692 req:*zetcd.GetChildrenRequest:&{Path:/mesos Watch:true}}
zetcd-2_1 | I0925 18:38:03.825216 1 zklog.go:63] GetChildren(1506364692,{Path:/mesos Watch:true})
zetcd-2_1 | I0925 18:38:03.825274 1 zklog.go:63] GetChildren(1506364692,{Path:/mesos Watch:true})
zetcd-2_1 | I0925 18:38:03.826114 1 session.go:122] waitFutureSendResp=&{Path:/mesos Watch:true}
zetcd-2_1 | I0925 18:38:03.826702 1 session.go:142] zk/zkresp=({Xid:1506364692 Zxid:10 Err:0},&{Children:[json.info_0000000001 json.info_0000000002 json.info_0000000000]})
zetcd-2_1 | I0925 18:38:03.840105 1 zketcd.go:436] GetChildren(1506364692) = (zxid=11, resp={Children:[json.info_0000000000 json.info_0000000001 json.info_0000000002]})
zetcd-2_1 | I0925 18:38:03.840146 1 conn.go:131] sendXchk Xid:1506364692 ZXid:10 Resp:&{Children:[json.info_0000000000 json.info_0000000001 json.info_0000000002]}
zetcd-2_1 | I0925 18:38:03.840447 1 conn.go:139] conn.Send(xid=1506364692, zxid=10, &{Children:[json.info_0000000000 json.info_0000000001 json.info_0000000002]})
zetcd-2_1 | I0925 18:38:03.840458 1 server.go:110] zkreq={xid:1506364692 req:*zetcd.GetChildrenRequest:&{Path:/mesos Watch:true}}
zetcd-2_1 | I0925 18:38:03.840502 1 zklog.go:63] GetChildren(1506364692,{Path:/mesos Watch:true})
zetcd-2_1 | I0925 18:38:03.840546 1 zklog.go:63] GetChildren(1506364692,{Path:/mesos Watch:true})
zetcd-2_1 | I0925 18:38:03.840729 1 session.go:122] waitFutureSendResp=&{Path:/mesos Watch:true}
zetcd-2_1 | I0925 18:38:03.841453 1 session.go:142] zk/zkresp=({Xid:1506364692 Zxid:10 Err:0},&{Children:[json.info_0000000001 json.info_0000000002 json.info_0000000000]})
zetcd-2_1 | I0925 18:38:03.841588 1 zketcd.go:436] GetChildren(1506364692) = (zxid=11, resp={Children:[json.info_0000000000 json.info_0000000001 json.info_0000000002]})
zetcd-2_1 | I0925 18:38:03.841640 1 conn.go:131] sendXchk Xid:1506364692 ZXid:10 Resp:&{Children:[json.info_0000000000 json.info_0000000001 json.info_0000000002]}
zetcd-2_1 | I0925 18:38:03.841655 1 conn.go:139] conn.Send(xid=1506364692, zxid=10, &{Children:[json.info_0000000000 json.info_0000000001 json.info_0000000002]})
zetcd-2_1 | I0925 18:38:04.600449 1 server.go:128] accepted remote connection "172.18.0.6:36508"
zetcd-2_1 | I0925 18:38:04.600531 1 session.go:54] newSession([zookeeper:2181])
zetcd-2_1 | I0925 18:38:04.600486 1 authconn.go:53] auth(&{ProtocolVersion:0 LastZxidSeen:0 TimeOut:10000 SessionID:0 Passwd:[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]})
zetcd-2_1 | I0925 18:38:04.643081 1 pool.go:92] authresp=&{ProtocolVersion:0 TimeOut:10000 SessionID:7635112897658448672 Passwd:[22 19 119 85 47 56 92 25 175 245 89 47 165 106 68 31]}
zetcd-2_1 | I0925 18:38:04.870012 1 session.go:59] starting the session... id=7635112897658448672
zetcd-2_1 | I0925 18:38:04.870010 1 session.go:90] auth resp OK ({ProtocolVersion:0 TimeOut:10000 SessionID:98721115828125702 Passwd:[36 168 100 25 243 105 249 112 247 217 43 195 91 235 26 16]})
zetcd-2_1 | I0925 18:38:04.870133 1 server.go:46] serving global serial session requests on s=&{Conn:0xc4201c38c0 oracle:0xc4202483c0 candidate:0xc420064480 req:{ProtocolVersion:0 LastZxidSeen:0 TimeOut:10000 SessionID:0 Passwd:[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]} sp:0xc4201aa020}
zetcd-2_1 | I0925 18:38:04.871057 1 server.go:110] zkreq={xid:1506364685 req:*zetcd.ExistsRequest:&{Path:/mesos Watch:false}}
zetcd-2_1 | I0925 18:38:04.871145 1 zklog.go:38] Exists(1506364685,{Path:/mesos Watch:false})
zetcd-2_1 | I0925 18:38:04.872234 1 zklog.go:38] Exists(1506364685,{Path:/mesos Watch:false})
zetcd-2_1 | I0925 18:38:04.872818 1 session.go:122] waitFutureSendResp=&{Path:/mesos Watch:false}
zetcd-2_1 | I0925 18:38:04.874393 1 zketcd.go:285] Exists(1506364685) = (zxid=13, resp={Stat:{Czxid:3 Mzxid:3 Ctime:1506364683323773 Mtime:1506364683323773 Version:0 Cversion:3 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:3 Pzxid:10}})
zetcd-2_1 | I0925 18:38:04.874428 1 session.go:142] zk/zkresp=({Xid:1506364685 Zxid:12 Err:0},&{Stat:{Czxid:3 Mzxid:3 Ctime:1506364683324 Mtime:1506364683324 Version:0 Cversion:3 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:3 Pzxid:10}})
zetcd-2_1 | I0925 18:38:04.874607 1 conn.go:131] sendXchk Xid:1506364685 ZXid:12 Resp:&{Stat:{Czxid:3 Mzxid:3 Ctime:1506364683324 Mtime:1506364683324 Version:0 Cversion:3 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:3 Pzxid:10}}
zetcd-2_1 | I0925 18:38:04.875100 1 conn.go:139] conn.Send(xid=1506364685, zxid=12, &{Stat:{Czxid:3 Mzxid:3 Ctime:1506364683324 Mtime:1506364683324 Version:0 Cversion:3 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:3 Pzxid:10}})
zetcd-2_1 | I0925 18:38:04.875821 1 server.go:110] zkreq={xid:1506364686 req:*zetcd.GetChildrenRequest:&{Path:/mesos Watch:true}}
zetcd-2_1 | I0925 18:38:04.875911 1 zklog.go:63] GetChildren(1506364686,{Path:/mesos Watch:true})
zetcd-2_1 | I0925 18:38:04.875931 1 zklog.go:63] GetChildren(1506364686,{Path:/mesos Watch:true})
zetcd-2_1 | I0925 18:38:04.876427 1 session.go:122] waitFutureSendResp=&{Path:/mesos Watch:true}
zetcd-2_1 | I0925 18:38:04.877414 1 zketcd.go:436] GetChildren(1506364686) = (zxid=13, resp={Children:[json.info_0000000000 json.info_0000000001 json.info_0000000002]})
zetcd-2_1 | I0925 18:38:04.877553 1 session.go:142] zk/zkresp=({Xid:1506364686 Zxid:12 Err:0},&{Children:[json.info_0000000001 json.info_0000000002 json.info_0000000000]})
zetcd-2_1 | I0925 18:38:04.877661 1 conn.go:131] sendXchk Xid:1506364686 ZXid:12 Resp:&{Children:[json.info_0000000000 json.info_0000000001 json.info_0000000002]}
zetcd-2_1 | I0925 18:38:04.877827 1 conn.go:139] conn.Send(xid=1506364686, zxid=12, &{Children:[json.info_0000000000 json.info_0000000001 json.info_0000000002]})
zetcd-2_1 | I0925 18:38:04.878878 1 server.go:110] zkreq={xid:1506364687 req:*zetcd.GetDataRequest:&{Path:/mesos/json.info_0000000000 Watch:false}}
zetcd-2_1 | I0925 18:38:04.878957 1 zklog.go:43] GetData(1506364687,{Path:/mesos/json.info_0000000000 Watch:false})
zetcd-2_1 | I0925 18:38:04.878985 1 zklog.go:43] GetData(1506364687,{Path:/mesos/json.info_0000000000 Watch:false})
zetcd-2_1 | I0925 18:38:04.880227 1 session.go:122] waitFutureSendResp=&{Path:/mesos/json.info_0000000000 Watch:false}
zetcd-2_1 | I0925 18:38:04.881378 1 zketcd.go:320] GetData(1506364687) = (zxid=13, resp={Data:[123 34 97 100 100 114 101 115 115 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 112 34 58 34 49 55 50 46 49 56 46 48 46 49 48 34 44 34 112 111 114 116 34 58 53 48 53 48 125 44 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 100 34 58 34 54 48 50 55 99 49 55 55 45 97 49 56 101 45 52 52 97 102 45 97 102 97 55 45 57 97 99 102 99 57 54 98 57 100 48 56 34 44 34 105 112 34 58 49 54 55 55 55 54 57 52 48 44 34 112 105 100 34 58 34 109 97 115 116 101 114 64 49 55 50 46 49 56 46 48 46 49 48 58 53 48 53 48 34 44 34 112 111 114 116 34 58 53 48 53 48 44 34 118 101 114 115 105 111 110 34 58 34 49 46 51 46 49 34 125] Stat:{Czxid:5 Mzxid:5 Ctime:1506364683338367 Mtime:1506364683338367 Version:0 Cversion:0 Aversion:0 EphemeralOwner:7635112897658448644 DataLength:217 NumChildren:0 Pzxid:5}})
zetcd-2_1 | I0925 18:38:04.881671 1 session.go:142] zk/zkresp=({Xid:1506364687 Zxid:12 Err:0},&{Data:[123 34 97 100 100 114 101 115 115 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 112 34 58 34 49 55 50 46 49 56 46 48 46 49 48 34 44 34 112 111 114 116 34 58 53 48 53 48 125 44 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 100 34 58 34 54 48 50 55 99 49 55 55 45 97 49 56 101 45 52 52 97 102 45 97 102 97 55 45 57 97 99 102 99 57 54 98 57 100 48 56 34 44 34 105 112 34 58 49 54 55 55 55 54 57 52 48 44 34 112 105 100 34 58 34 109 97 115 116 101 114 64 49 55 50 46 49 56 46 48 46 49 48 58 53 48 53 48 34 44 34 112 111 114 116 34 58 53 48 53 48 44 34 118 101 114 115 105 111 110 34 58 34 49 46 51 46 49 34 125] Stat:{Czxid:5 Mzxid:5 Ctime:1506364683338 Mtime:1506364683338 Version:0 Cversion:0 Aversion:0 EphemeralOwner:98721115828125697 DataLength:217 NumChildren:0 Pzxid:5}})
zetcd-2_1 | I0925 18:38:04.881913 1 conn.go:131] sendXchk Xid:1506364687 ZXid:12 Resp:&{Data:[123 34 97 100 100 114 101 115 115 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 112 34 58 34 49 55 50 46 49 56 46 48 46 49 48 34 44 34 112 111 114 116 34 58 53 48 53 48 125 44 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 100 34 58 34 54 48 50 55 99 49 55 55 45 97 49 56 101 45 52 52 97 102 45 97 102 97 55 45 57 97 99 102 99 57 54 98 57 100 48 56 34 44 34 105 112 34 58 49 54 55 55 55 54 57 52 48 44 34 112 105 100 34 58 34 109 97 115 116 101 114 64 49 55 50 46 49 56 46 48 46 49 48 58 53 48 53 48 34 44 34 112 111 114 116 34 58 53 48 53 48 44 34 118 101 114 115 105 111 110 34 58 34 49 46 51 46 49 34 125] Stat:{Czxid:5 Mzxid:5 Ctime:1506364683338 Mtime:1506364683338 Version:0 Cversion:0 Aversion:0 EphemeralOwner:98721115828125697 DataLength:217 NumChildren:0 Pzxid:5}}
zetcd-2_1 | I0925 18:38:04.882454 1 conn.go:139] conn.Send(xid=1506364687, zxid=12, &{Data:[123 34 97 100 100 114 101 115 115 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 112 34 58 34 49 55 50 46 49 56 46 48 46 49 48 34 44 34 112 111 114 116 34 58 53 48 53 48 125 44 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 100 34 58 34 54 48 50 55 99 49 55 55 45 97 49 56 101 45 52 52 97 102 45 97 102 97 55 45 57 97 99 102 99 57 54 98 57 100 48 56 34 44 34 105 112 34 58 49 54 55 55 55 54 57 52 48 44 34 112 105 100 34 58 34 109 97 115 116 101 114 64 49 55 50 46 49 56 46 48 46 49 48 58 53 48 53 48 34 44 34 112 111 114 116 34 58 53 48 53 48 44 34 118 101 114 115 105 111 110 34 58 34 49 46 51 46 49 34 125] Stat:{Czxid:5 Mzxid:5 Ctime:1506364683338 Mtime:1506364683338 Version:0 Cversion:0 Aversion:0 EphemeralOwner:98721115828125697 DataLength:217 NumChildren:0 Pzxid:5}})
mesos-slave-3_1 | 2017-09-25 18:38:05,471:1(0x7f8ed7695700):ZOO_INFO@zookeeper_init@800: Initiating client connection, host=zetcd:2181 sessionTimeout=10000 watcher=0x7f8ee13eb6d0 sessionId=0 sessionPasswd=<null> context=0x7f8ea8000930 flags=0
mesos-slave-3_1 | I0925 18:38:05.471475 160 slave.cpp:199] Flags at startup: --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http_readonly="false" --authenticate_http_readwrite="false" --authenticatee="crammd5" --authentication_backoff_factor="1secs" --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="docker,mesos" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="2mins" --docker_store_dir="/tmp/mesos/store/docker" --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="5mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/mesos/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --http_authenticators="basic" --http_command_executor="false" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/usr/libexec/mesos" --logbufsecs="0" --logging_level="INFO" --master="zk://zetcd:2181/mesos" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --port="5051" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="1secs" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/data"
zetcd-1_1 | I0925 18:38:05.472110 1 server.go:128] accepted remote connection "172.18.0.8:45918"
zetcd-1_1 | I0925 18:38:05.472156 1 authconn.go:53] auth(&{ProtocolVersion:0 LastZxidSeen:0 TimeOut:10000 SessionID:0 Passwd:[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]})
zetcd-1_1 | I0925 18:38:05.472354 1 session.go:54] newSession([zookeeper:2181])
zetcd-1_1 | I0925 18:38:05.474805 1 pool.go:92] authresp=&{ProtocolVersion:0 TimeOut:10000 SessionID:7635112897658448675 Passwd:[151 209 113 143 15 24 67 150 217 190 75 138 165 188 132 51]}
zetcd-1_1 | I0925 18:38:05.480924 1 session.go:59] starting the session... id=7635112897658448675
zetcd-1_1 | I0925 18:38:05.481106 1 session.go:90] auth resp OK ({ProtocolVersion:0 TimeOut:10000 SessionID:98721115828125703 Passwd:[55 201 94 25 147 181 216 181 236 28 147 238 47 116 47 221]})
zetcd-1_1 | I0925 18:38:05.481149 1 server.go:46] serving global serial session requests on s=&{Conn:0xc42020b200 oracle:0xc4200c9040 candidate:0xc4201f8000 req:{ProtocolVersion:0 LastZxidSeen:0 TimeOut:10000 SessionID:0 Passwd:[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]} sp:0xc42017ac70}
zetcd-1_1 | I0925 18:38:05.481502 1 server.go:110] zkreq={xid:1506364686 req:*zetcd.ExistsRequest:&{Path:/mesos Watch:false}}
zetcd-1_1 | I0925 18:38:05.481537 1 zklog.go:38] Exists(1506364686,{Path:/mesos Watch:false})
zetcd-1_1 | I0925 18:38:05.486987 1 zklog.go:38] Exists(1506364686,{Path:/mesos Watch:false})
zetcd-1_1 | I0925 18:38:05.487991 1 zketcd.go:285] Exists(1506364686) = (zxid=14, resp={Stat:{Czxid:3 Mzxid:3 Ctime:1506364683323773 Mtime:1506364683323773 Version:0 Cversion:3 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:3 Pzxid:10}})
zetcd-1_1 | I0925 18:38:05.488012 1 session.go:122] waitFutureSendResp=&{Path:/mesos Watch:false}
zetcd-1_1 | I0925 18:38:05.488577 1 session.go:142] zk/zkresp=({Xid:1506364686 Zxid:13 Err:0},&{Stat:{Czxid:3 Mzxid:3 Ctime:1506364683324 Mtime:1506364683324 Version:0 Cversion:3 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:3 Pzxid:10}})
zetcd-1_1 | I0925 18:38:05.488622 1 conn.go:131] sendXchk Xid:1506364686 ZXid:13 Resp:&{Stat:{Czxid:3 Mzxid:3 Ctime:1506364683324 Mtime:1506364683324 Version:0 Cversion:3 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:3 Pzxid:10}}
zetcd-1_1 | I0925 18:38:05.488818 1 conn.go:139] conn.Send(xid=1506364686, zxid=13, &{Stat:{Czxid:3 Mzxid:3 Ctime:1506364683324 Mtime:1506364683324 Version:0 Cversion:3 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:3 Pzxid:10}})
zetcd-1_1 | I0925 18:38:05.489071 1 server.go:110] zkreq={xid:1506364687 req:*zetcd.GetChildrenRequest:&{Path:/mesos Watch:true}}
zetcd-1_1 | I0925 18:38:05.489103 1 zklog.go:63] GetChildren(1506364687,{Path:/mesos Watch:true})
zetcd-1_1 | I0925 18:38:05.489105 1 zklog.go:63] GetChildren(1506364687,{Path:/mesos Watch:true})
zetcd-1_1 | I0925 18:38:05.489298 1 session.go:122] waitFutureSendResp=&{Path:/mesos Watch:true}
zetcd-1_1 | I0925 18:38:05.489703 1 session.go:142] zk/zkresp=({Xid:1506364687 Zxid:13 Err:0},&{Children:[json.info_0000000001 json.info_0000000002 json.info_0000000000]})
zetcd-1_1 | I0925 18:38:05.489777 1 zketcd.go:436] GetChildren(1506364687) = (zxid=14, resp={Children:[json.info_0000000000 json.info_0000000001 json.info_0000000002]})
zetcd-1_1 | I0925 18:38:05.489827 1 conn.go:131] sendXchk Xid:1506364687 ZXid:13 Resp:&{Children:[json.info_0000000000 json.info_0000000001 json.info_0000000002]}
zetcd-1_1 | I0925 18:38:05.490144 1 conn.go:139] conn.Send(xid=1506364687, zxid=13, &{Children:[json.info_0000000000 json.info_0000000001 json.info_0000000002]})
zetcd-1_1 | I0925 18:38:05.491120 1 server.go:110] zkreq={xid:1506364688 req:*zetcd.GetDataRequest:&{Path:/mesos/json.info_0000000000 Watch:false}}
zetcd-1_1 | I0925 18:38:05.491157 1 zklog.go:43] GetData(1506364688,{Path:/mesos/json.info_0000000000 Watch:false})
zetcd-1_1 | I0925 18:38:05.491347 1 session.go:122] waitFutureSendResp=&{Path:/mesos/json.info_0000000000 Watch:false}
zetcd-1_1 | I0925 18:38:05.491545 1 zklog.go:43] GetData(1506364688,{Path:/mesos/json.info_0000000000 Watch:false})
zetcd-1_1 | I0925 18:38:05.491876 1 session.go:142] zk/zkresp=({Xid:1506364688 Zxid:13 Err:0},&{Data:[123 34 97 100 100 114 101 115 115 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 112 34 58 34 49 55 50 46 49 56 46 48 46 49 48 34 44 34 112 111 114 116 34 58 53 48 53 48 125 44 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 100 34 58 34 54 48 50 55 99 49 55 55 45 97 49 56 101 45 52 52 97 102 45 97 102 97 55 45 57 97 99 102 99 57 54 98 57 100 48 56 34 44 34 105 112 34 58 49 54 55 55 55 54 57 52 48 44 34 112 105 100 34 58 34 109 97 115 116 101 114 64 49 55 50 46 49 56 46 48 46 49 48 58 53 48 53 48 34 44 34 112 111 114 116 34 58 53 48 53 48 44 34 118 101 114 115 105 111 110 34 58 34 49 46 51 46 49 34 125] Stat:{Czxid:5 Mzxid:5 Ctime:1506364683338 Mtime:1506364683338 Version:0 Cversion:0 Aversion:0 EphemeralOwner:98721115828125697 DataLength:217 NumChildren:0 Pzxid:5}})
zetcd-1_1 | I0925 18:38:05.492134 1 zketcd.go:320] GetData(1506364688) = (zxid=14, resp={Data:[123 34 97 100 100 114 101 115 115 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 112 34 58 34 49 55 50 46 49 56 46 48 46 49 48 34 44 34 112 111 114 116 34 58 53 48 53 48 125 44 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 100 34 58 34 54 48 50 55 99 49 55 55 45 97 49 56 101 45 52 52 97 102 45 97 102 97 55 45 57 97 99 102 99 57 54 98 57 100 48 56 34 44 34 105 112 34 58 49 54 55 55 55 54 57 52 48 44 34 112 105 100 34 58 34 109 97 115 116 101 114 64 49 55 50 46 49 56 46 48 46 49 48 58 53 48 53 48 34 44 34 112 111 114 116 34 58 53 48 53 48 44 34 118 101 114 115 105 111 110 34 58 34 49 46 51 46 49 34 125] Stat:{Czxid:5 Mzxid:5 Ctime:1506364683338367 Mtime:1506364683338367 Version:0 Cversion:0 Aversion:0 EphemeralOwner:7635112897658448644 DataLength:217 NumChildren:0 Pzxid:5}})
zetcd-1_1 | W0925 18:38:05.492233 1 zk.go:318] xchk failed (stat mismatch)
zetcd-1_1 | candidate: &{Data:[123 34 97 100 100 114 101 115 115 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 112 34 58 34 49 55 50 46 49 56 46 48 46 49 48 34 44 34 112 111 114 116 34 58 53 48 53 48 125 44 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 100 34 58 34 54 48 50 55 99 49 55 55 45 97 49 56 101 45 52 52 97 102 45 97 102 97 55 45 57 97 99 102 99 57 54 98 57 100 48 56 34 44 34 105 112 34 58 49 54 55 55 55 54 57 52 48 44 34 112 105 100 34 58 34 109 97 115 116 101 114 64 49 55 50 46 49 56 46 48 46 49 48 58 53 48 53 48 34 44 34 112 111 114 116 34 58 53 48 53 48 44 34 118 101 114 115 105 111 110 34 58 34 49 46 51 46 49 34 125] Stat:{Czxid:5 Mzxid:5 Ctime:1506364683338367 Mtime:1506364683338367 Version:0 Cversion:0 Aversion:0 EphemeralOwner:7635112897658448644 DataLength:217 NumChildren:0 Pzxid:5}}
zetcd-1_1 | oracle: &{Data:[123 34 97 100 100 114 101 115 115 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 112 34 58 34 49 55 50 46 49 56 46 48 46 49 48 34 44 34 112 111 114 116 34 58 53 48 53 48 125 44 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 100 34 58 34 54 48 50 55 99 49 55 55 45 97 49 56 101 45 52 52 97 102 45 97 102 97 55 45 57 97 99 102 99 57 54 98 57 100 48 56 34 44 34 105 112 34 58 49 54 55 55 55 54 57 52 48 44 34 112 105 100 34 58 34 109 97 115 116 101 114 64 49 55 50 46 49 56 46 48 46 49 48 58 53 48 53 48 34 44 34 112 111 114 116 34 58 53 48 53 48 44 34 118 101 114 115 105 111 110 34 58 34 49 46 51 46 49 34 125] Stat:{Czxid:5 Mzxid:5 Ctime:1506364683338 Mtime:1506364683338 Version:0 Cversion:0 Aversion:0 EphemeralOwner:98721115828125697 DataLength:217 NumChildren:0 Pzxid:5}}
zetcd-1_1 | I0925 18:38:05.492414 1 conn.go:131] sendXchk Xid:1506364688 ZXid:13 Resp:&{Data:[123 34 97 100 100 114 101 115 115 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 112 34 58 34 49 55 50 46 49 56 46 48 46 49 48 34 44 34 112 111 114 116 34 58 53 48 53 48 125 44 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 100 34 58 34 54 48 50 55 99 49 55 55 45 97 49 56 101 45 52 52 97 102 45 97 102 97 55 45 57 97 99 102 99 57 54 98 57 100 48 56 34 44 34 105 112 34 58 49 54 55 55 55 54 57 52 48 44 34 112 105 100 34 58 34 109 97 115 116 101 114 64 49 55 50 46 49 56 46 48 46 49 48 58 53 48 53 48 34 44 34 112 111 114 116 34 58 53 48 53 48 44 34 118 101 114 115 105 111 110 34 58 34 49 46 51 46 49 34 125] Stat:{Czxid:5 Mzxid:5 Ctime:1506364683338 Mtime:1506364683338 Version:0 Cversion:0 Aversion:0 EphemeralOwner:98721115828125697 DataLength:217 NumChildren:0 Pzxid:5}}
zetcd-1_1 | I0925 18:38:05.492705 1 conn.go:139] conn.Send(xid=1506364688, zxid=13, &{Data:[123 34 97 100 100 114 101 115 115 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 112 34 58 34 49 55 50 46 49 56 46 48 46 49 48 34 44 34 112 111 114 116 34 58 53 48 53 48 125 44 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 100 34 58 34 54 48 50 55 99 49 55 55 45 97 49 56 101 45 52 52 97 102 45 97 102 97 55 45 57 97 99 102 99 57 54 98 57 100 48 56 34 44 34 105 112 34 58 49 54 55 55 55 54 57 52 48 44 34 112 105 100 34 58 34 109 97 115 116 101 114 64 49 55 50 46 49 56 46 48 46 49 48 58 53 48 53 48 34 44 34 112 111 114 116 34 58 53 48 53 48 44 34 118 101 114 115 105 111 110 34 58 34 49 46 51 46 49 34 125] Stat:{Czxid:5 Mzxid:5 Ctime:1506364683338 Mtime:1506364683338 Version:0 Cversion:0 Aversion:0 EphemeralOwner:98721115828125697 DataLength:217 NumChildren:0 Pzxid:5}})
mesos-slave-2_1 | 2017-09-25 18:38:06,322:1(0x7f42f5b63700):ZOO_INFO@zookeeper_init@800: Initiating client connection, host=zetcd:2181 sessionTimeout=10000 watcher=0x7f42fe0b66d0 sessionId=0 sessionPasswd=<null> context=0x7f42e0001640 flags=0
mesos-slave-2_1 | I0925 18:38:06.322383 171 slave.cpp:199] Flags at startup: --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http_readonly="false" --authenticate_http_readwrite="false" --authenticatee="crammd5" --authentication_backoff_factor="1secs" --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="docker,mesos" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="2mins" --docker_store_dir="/tmp/mesos/store/docker" --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="5mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/mesos/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --http_authenticators="basic" --http_command_executor="false" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/usr/libexec/mesos" --logbufsecs="0" --logging_level="INFO" --master="zk://zetcd:2181/mesos" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --port="5051" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="1secs" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/data"
zetcd-2_1 | I0925 18:38:06.323505 1 server.go:128] accepted remote connection "172.18.0.7:46240"
zetcd-2_1 | I0925 18:38:06.323561 1 session.go:54] newSession([zookeeper:2181])
zetcd-2_1 | I0925 18:38:06.323550 1 authconn.go:53] auth(&{ProtocolVersion:0 LastZxidSeen:0 TimeOut:10000 SessionID:0 Passwd:[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]})
zetcd-2_1 | I0925 18:38:06.326759 1 pool.go:92] authresp=&{ProtocolVersion:0 TimeOut:10000 SessionID:7635112897658448678 Passwd:[136 119 218 75 47 88 75 175 143 12 181 248 203 201 192 245]}
zetcd-2_1 | I0925 18:38:06.328910 1 session.go:59] starting the session... id=7635112897658448678
zetcd-2_1 | I0925 18:38:06.328942 1 session.go:90] auth resp OK ({ProtocolVersion:0 TimeOut:10000 SessionID:98721115828125704 Passwd:[151 110 65 25 179 47 53 14 182 107 151 199 81 32 150 222]})
zetcd-2_1 | I0925 18:38:06.328997 1 server.go:46] serving global serial session requests on s=&{Conn:0xc420210960 oracle:0xc420142140 candidate:0xc420064280 req:{ProtocolVersion:0 LastZxidSeen:0 TimeOut:10000 SessionID:0 Passwd:[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]} sp:0xc4201aa020}
zetcd-2_1 | I0925 18:38:06.334934 1 server.go:110] zkreq={xid:1506364687 req:*zetcd.ExistsRequest:&{Path:/mesos Watch:false}}
zetcd-2_1 | I0925 18:38:06.335034 1 zklog.go:38] Exists(1506364687,{Path:/mesos Watch:false})
zetcd-2_1 | I0925 18:38:06.335050 1 zklog.go:38] Exists(1506364687,{Path:/mesos Watch:false})
zetcd-2_1 | I0925 18:38:06.335339 1 session.go:122] waitFutureSendResp=&{Path:/mesos Watch:false}
zetcd-2_1 | I0925 18:38:06.335622 1 zketcd.go:285] Exists(1506364687) = (zxid=15, resp={Stat:{Czxid:3 Mzxid:3 Ctime:1506364683323773 Mtime:1506364683323773 Version:0 Cversion:3 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:3 Pzxid:10}})
zetcd-2_1 | I0925 18:38:06.335877 1 session.go:142] zk/zkresp=({Xid:1506364687 Zxid:14 Err:0},&{Stat:{Czxid:3 Mzxid:3 Ctime:1506364683324 Mtime:1506364683324 Version:0 Cversion:3 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:3 Pzxid:10}})
zetcd-2_1 | I0925 18:38:06.335922 1 conn.go:131] sendXchk Xid:1506364687 ZXid:14 Resp:&{Stat:{Czxid:3 Mzxid:3 Ctime:1506364683324 Mtime:1506364683324 Version:0 Cversion:3 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:3 Pzxid:10}}
zetcd-2_1 | I0925 18:38:06.336185 1 conn.go:139] conn.Send(xid=1506364687, zxid=14, &{Stat:{Czxid:3 Mzxid:3 Ctime:1506364683324 Mtime:1506364683324 Version:0 Cversion:3 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:3 Pzxid:10}})
zetcd-2_1 | I0925 18:38:06.336450 1 server.go:110] zkreq={xid:1506364688 req:*zetcd.GetChildrenRequest:&{Path:/mesos Watch:true}}
zetcd-2_1 | I0925 18:38:06.336483 1 zklog.go:63] GetChildren(1506364688,{Path:/mesos Watch:true})
zetcd-2_1 | I0925 18:38:06.336496 1 zklog.go:63] GetChildren(1506364688,{Path:/mesos Watch:true})
zetcd-2_1 | I0925 18:38:06.337020 1 session.go:122] waitFutureSendResp=&{Path:/mesos Watch:true}
zetcd-2_1 | I0925 18:38:06.337511 1 zketcd.go:436] GetChildren(1506364688) = (zxid=15, resp={Children:[json.info_0000000000 json.info_0000000001 json.info_0000000002]})
zetcd-2_1 | I0925 18:38:06.337735 1 session.go:142] zk/zkresp=({Xid:1506364688 Zxid:14 Err:0},&{Children:[json.info_0000000001 json.info_0000000002 json.info_0000000000]})
zetcd-2_1 | I0925 18:38:06.337792 1 conn.go:131] sendXchk Xid:1506364688 ZXid:14 Resp:&{Children:[json.info_0000000000 json.info_0000000001 json.info_0000000002]}
zetcd-2_1 | I0925 18:38:06.338085 1 conn.go:139] conn.Send(xid=1506364688, zxid=14, &{Children:[json.info_0000000000 json.info_0000000001 json.info_0000000002]})
zetcd-2_1 | I0925 18:38:06.340081 1 server.go:110] zkreq={xid:1506364689 req:*zetcd.GetDataRequest:&{Path:/mesos/json.info_0000000000 Watch:false}}
zetcd-2_1 | I0925 18:38:06.340130 1 zklog.go:43] GetData(1506364689,{Path:/mesos/json.info_0000000000 Watch:false})
zetcd-2_1 | I0925 18:38:06.340194 1 zklog.go:43] GetData(1506364689,{Path:/mesos/json.info_0000000000 Watch:false})
zetcd-2_1 | I0925 18:38:06.340380 1 session.go:122] waitFutureSendResp=&{Path:/mesos/json.info_0000000000 Watch:false}
zetcd-2_1 | I0925 18:38:06.340759 1 zketcd.go:320] GetData(1506364689) = (zxid=15, resp={Data:[123 34 97 100 100 114 101 115 115 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 112 34 58 34 49 55 50 46 49 56 46 48 46 49 48 34 44 34 112 111 114 116 34 58 53 48 53 48 125 44 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 100 34 58 34 54 48 50 55 99 49 55 55 45 97 49 56 101 45 52 52 97 102 45 97 102 97 55 45 57 97 99 102 99 57 54 98 57 100 48 56 34 44 34 105 112 34 58 49 54 55 55 55 54 57 52 48 44 34 112 105 100 34 58 34 109 97 115 116 101 114 64 49 55 50 46 49 56 46 48 46 49 48 58 53 48 53 48 34 44 34 112 111 114 116 34 58 53 48 53 48 44 34 118 101 114 115 105 111 110 34 58 34 49 46 51 46 49 34 125] Stat:{Czxid:5 Mzxid:5 Ctime:1506364683338367 Mtime:1506364683338367 Version:0 Cversion:0 Aversion:0 EphemeralOwner:7635112897658448644 DataLength:217 NumChildren:0 Pzxid:5}})
zetcd-2_1 | I0925 18:38:06.341008 1 session.go:142] zk/zkresp=({Xid:1506364689 Zxid:14 Err:0},&{Data:[123 34 97 100 100 114 101 115 115 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 112 34 58 34 49 55 50 46 49 56 46 48 46 49 48 34 44 34 112 111 114 116 34 58 53 48 53 48 125 44 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 100 34 58 34 54 48 50 55 99 49 55 55 45 97 49 56 101 45 52 52 97 102 45 97 102 97 55 45 57 97 99 102 99 57 54 98 57 100 48 56 34 44 34 105 112 34 58 49 54 55 55 55 54 57 52 48 44 34 112 105 100 34 58 34 109 97 115 116 101 114 64 49 55 50 46 49 56 46 48 46 49 48 58 53 48 53 48 34 44 34 112 111 114 116 34 58 53 48 53 48 44 34 118 101 114 115 105 111 110 34 58 34 49 46 51 46 49 34 125] Stat:{Czxid:5 Mzxid:5 Ctime:1506364683338 Mtime:1506364683338 Version:0 Cversion:0 Aversion:0 EphemeralOwner:98721115828125697 DataLength:217 NumChildren:0 Pzxid:5}})
zetcd-2_1 | I0925 18:38:06.341059 1 conn.go:131] sendXchk Xid:1506364689 ZXid:14 Resp:&{Data:[123 34 97 100 100 114 101 115 115 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 112 34 58 34 49 55 50 46 49 56 46 48 46 49 48 34 44 34 112 111 114 116 34 58 53 48 53 48 125 44 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 100 34 58 34 54 48 50 55 99 49 55 55 45 97 49 56 101 45 52 52 97 102 45 97 102 97 55 45 57 97 99 102 99 57 54 98 57 100 48 56 34 44 34 105 112 34 58 49 54 55 55 55 54 57 52 48 44 34 112 105 100 34 58 34 109 97 115 116 101 114 64 49 55 50 46 49 56 46 48 46 49 48 58 53 48 53 48 34 44 34 112 111 114 116 34 58 53 48 53 48 44 34 118 101 114 115 105 111 110 34 58 34 49 46 51 46 49 34 125] Stat:{Czxid:5 Mzxid:5 Ctime:1506364683338 Mtime:1506364683338 Version:0 Cversion:0 Aversion:0 EphemeralOwner:98721115828125697 DataLength:217 NumChildren:0 Pzxid:5}}
zetcd-2_1 | I0925 18:38:06.341271 1 conn.go:139] conn.Send(xid=1506364689, zxid=14, &{Data:[123 34 97 100 100 114 101 115 115 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 112 34 58 34 49 55 50 46 49 56 46 48 46 49 48 34 44 34 112 111 114 116 34 58 53 48 53 48 125 44 34 104 111 115 116 110 97 109 101 34 58 34 54 48 51 50 51 98 97 52 57 98 101 48 34 44 34 105 100 34 58 34 54 48 50 55 99 49 55 55 45 97 49 56 101 45 52 52 97 102 45 97 102 97 55 45 57 97 99 102 99 57 54 98 57 100 48 56 34 44 34 105 112 34 58 49 54 55 55 55 54 57 52 48 44 34 112 105 100 34 58 34 109 97 115 116 101 114 64 49 55 50 46 49 56 46 48 46 49 48 58 53 48 53 48 34 44 34 112 111 114 116 34 58 53 48 53 48 44 34 118 101 114 115 105 111 110 34 58 34 49 46 51 46 49 34 125] Stat:{Czxid:5 Mzxid:5 Ctime:1506364683338 Mtime:1506364683338 Version:0 Cversion:0 Aversion:0 EphemeralOwner:98721115828125697 DataLength:217 NumChildren:0 Pzxid:5}})
mesos-master-3_1 | I0925 18:38:06.367161 10 master.cpp:5429] Received register agent message from slave(1)@172.18.0.8:5051 (d270103477f5)
mesos-master-3_1 | I0925 18:38:06.367290 10 master.cpp:5564] Registering agent at slave(1)@172.18.0.8:5051 (d270103477f5) with id 6027c177-a18e-44af-afa7-9acfc96b9d08-S1
mesos-master-3_1 | I0925 18:38:06.367388 10 registrar.cpp:493] Applied 1 operations in 33498ns; attempting to update the registry
mesos-master-3_1 | I0925 18:38:06.367527 10 registrar.cpp:550] Successfully updated the registry in 111104ns
mesos-master-3_1 | I0925 18:38:06.367846 10 master.cpp:5639] Registered agent 6027c177-a18e-44af-afa7-9acfc96b9d08-S1 at slave(1)@172.18.0.8:5051 (d270103477f5) with cpus(*):8; mem(*):63105; disk(*):203883; ports(*):[31000-32000]
mesos-master-3_1 | I0925 18:38:06.367954 10 hierarchical.cpp:525] Added agent 6027c177-a18e-44af-afa7-9acfc96b9d08-S1 (d270103477f5) with cpus(*):8; mem(*):63105; disk(*):203883; ports(*):[31000-32000] (allocated: {})
mesos-master-3_1 | I0925 18:38:06.368860 10 master.cpp:6324] Received update of agent 6027c177-a18e-44af-afa7-9acfc96b9d08-S1 at slave(1)@172.18.0.8:5051 (d270103477f5) with total oversubscribed resources {}
mesos-master-3_1 | I0925 18:38:06.978113 10 master.cpp:5429] Received register agent message from slave(1)@172.18.0.7:5051 (1730f4606017)
mesos-master-3_1 | I0925 18:38:06.978181 10 master.cpp:5564] Registering agent at slave(1)@172.18.0.7:5051 (1730f4606017) with id 6027c177-a18e-44af-afa7-9acfc96b9d08-S2
mesos-master-3_1 | I0925 18:38:06.978257 10 registrar.cpp:493] Applied 1 operations in 16869ns; attempting to update the registry
mesos-master-3_1 | I0925 18:38:06.978386 10 registrar.cpp:550] Successfully updated the registry in 110080ns
mesos-master-3_1 | I0925 18:38:06.978641 10 master.cpp:5639] Registered agent 6027c177-a18e-44af-afa7-9acfc96b9d08-S2 at slave(1)@172.18.0.7:5051 (1730f4606017) with cpus(*):8; mem(*):63105; disk(*):203883; ports(*):[31000-32000]
mesos-master-3_1 | I0925 18:38:06.978716 10 hierarchical.cpp:525] Added agent 6027c177-a18e-44af-afa7-9acfc96b9d08-S2 (1730f4606017) with cpus(*):8; mem(*):63105; disk(*):203883; ports(*):[31000-32000] (allocated: {})
mesos-master-3_1 | I0925 18:38:06.979573 10 master.cpp:6324] Received update of agent 6027c177-a18e-44af-afa7-9acfc96b9d08-S2 at slave(1)@172.18.0.7:5051 (1730f4606017) with total oversubscribed resources {}
zetcd-1_1 | I0925 18:38:07.159536 1 server.go:110] zkreq={xid:-2 req:*zetcd.PingRequest:&{}}
zetcd-2_1 | I0925 18:38:07.159529 1 server.go:110] zkreq={xid:-2 req:*zetcd.PingRequest:&{}}
zetcd-2_1 | I0925 18:38:07.159602 1 zklog.go:73] Ping(-2,{})
zetcd-2_1 | I0925 18:38:07.159634 1 zklog.go:73] Ping(-2,{})
zetcd-2_1 | I0925 18:38:07.159886 1 session.go:122] waitFutureSendResp=&{}
zetcd-1_1 | I0925 18:38:07.159604 1 zklog.go:73] Ping(-2,{})
zetcd-1_1 | I0925 18:38:07.159937 1 zklog.go:73] Ping(-2,{})
zetcd-2_1 | I0925 18:38:07.160649 1 session.go:142] zk/zkresp=({Xid:-2 Zxid:14 Err:0},&{})
zetcd-1_1 | I0925 18:38:07.160309 1 session.go:122] waitFutureSendResp=&{}
zetcd-2_1 | W0925 18:38:07.160690 1 zk.go:318] xchk failed (zxid mismatch)
zetcd-2_1 | candidate: &{Xid:-2 Zxid:7 Err:0}
zetcd-2_1 | oracle: &{Xid:-2 Zxid:14 Err:0}
zetcd-2_1 | I0925 18:38:07.160729 1 conn.go:131] sendXchk Xid:-2 ZXid:14 Resp:&{}
zetcd-2_1 | I0925 18:38:07.160763 1 conn.go:139] conn.Send(xid=-2, zxid=14, &{})
zetcd-2_1 | I0925 18:38:07.160783 1 server.go:110] zkreq={xid:-2 req:*zetcd.PingRequest:&{}}
zetcd-2_1 | I0925 18:38:07.161163 1 zklog.go:73] Ping(-2,{})
zetcd-1_1 | I0925 18:38:07.160954 1 session.go:142] zk/zkresp=({Xid:-2 Zxid:14 Err:0},&{})
zetcd-1_1 | W0925 18:38:07.160974 1 zk.go:318] xchk failed (zxid mismatch)
zetcd-1_1 | candidate: &{Xid:-2 Zxid:11 Err:0}
zetcd-1_1 | oracle: &{Xid:-2 Zxid:14 Err:0}
zetcd-1_1 | I0925 18:38:07.160987 1 conn.go:131] sendXchk Xid:-2 ZXid:14 Resp:&{}
zetcd-2_1 | I0925 18:38:07.161512 1 session.go:122] waitFutureSendResp=&{}
zetcd-2_1 | I0925 18:38:07.161617 1 zklog.go:73] Ping(-2,{})
zetcd-1_1 | I0925 18:38:07.162020 1 conn.go:139] conn.Send(xid=-2, zxid=14, &{})
zetcd-2_1 | I0925 18:38:07.162636 1 session.go:142] zk/zkresp=({Xid:-2 Zxid:14 Err:0},&{})
zetcd-2_1 | W0925 18:38:07.162678 1 zk.go:318] xchk failed (zxid mismatch)
zetcd-2_1 | candidate: &{Xid:-2 Zxid:7 Err:0}
zetcd-2_1 | oracle: &{Xid:-2 Zxid:14 Err:0}
zetcd-2_1 | I0925 18:38:07.162704 1 conn.go:131] sendXchk Xid:-2 ZXid:14 Resp:&{}
zetcd-2_1 | I0925 18:38:07.162936 1 conn.go:139] conn.Send(xid=-2, zxid=14, &{})
zetcd-2_1 | I0925 18:38:07.162968 1 server.go:110] zkreq={xid:-2 req:*zetcd.PingRequest:&{}}
zetcd-2_1 | I0925 18:38:07.163035 1 zklog.go:73] Ping(-2,{})
zetcd-2_1 | I0925 18:38:07.163058 1 zklog.go:73] Ping(-2,{})
zetcd-2_1 | I0925 18:38:07.163996 1 session.go:122] waitFutureSendResp=&{}
zetcd-2_1 | I0925 18:38:07.164583 1 session.go:142] zk/zkresp=({Xid:-2 Zxid:14 Err:0},&{})
zetcd-2_1 | W0925 18:38:07.164619 1 zk.go:318] xchk failed (zxid mismatch)
zetcd-2_1 | candidate: &{Xid:-2 Zxid:7 Err:0}
zetcd-2_1 | oracle: &{Xid:-2 Zxid:14 Err:0}
zetcd-2_1 | I0925 18:38:07.164657 1 conn.go:131] sendXchk Xid:-2 ZXid:14 Resp:&{}
zetcd-2_1 | I0925 18:38:07.164900 1 conn.go:139] conn.Send(xid=-2, zxid=14, &{})
zetcd-1_1 | I0925 18:38:07.175552 1 server.go:110] zkreq={xid:-2 req:*zetcd.PingRequest:&{}}
zetcd-1_1 | I0925 18:38:07.175683 1 zklog.go:73] Ping(-2,{})
zetcd-1_1 | I0925 18:38:07.175932 1 session.go:122] waitFutureSendResp=&{}
zetcd-1_1 | I0925 18:38:07.176181 1 zklog.go:73] Ping(-2,{})
zetcd-1_1 | I0925 18:38:07.176767 1 session.go:142] zk/zkresp=({Xid:-2 Zxid:14 Err:0},&{})
zetcd-1_1 | W0925 18:38:07.176991 1 zk.go:318] xchk failed (zxid mismatch)
zetcd-1_1 | candidate: &{Xid:-2 Zxid:11 Err:0}
zetcd-1_1 | oracle: &{Xid:-2 Zxid:14 Err:0}
zetcd-1_1 | I0925 18:38:07.177235 1 conn.go:131] sendXchk Xid:-2 ZXid:14 Resp:&{}
zetcd-1_1 | I0925 18:38:07.177615 1 conn.go:139] conn.Send(xid=-2, zxid=14, &{})
zetcd-1_1 | I0925 18:38:07.195517 1 server.go:110] zkreq={xid:-2 req:*zetcd.PingRequest:&{}}
zetcd-1_1 | I0925 18:38:07.195666 1 zklog.go:73] Ping(-2,{})
zetcd-1_1 | I0925 18:38:07.195721 1 zklog.go:73] Ping(-2,{})
zetcd-1_1 | I0925 18:38:07.195998 1 session.go:122] waitFutureSendResp=&{}
zetcd-1_1 | I0925 18:38:07.196506 1 session.go:142] zk/zkresp=({Xid:-2 Zxid:14 Err:0},&{})
zetcd-1_1 | W0925 18:38:07.196533 1 zk.go:318] xchk failed (zxid mismatch)
zetcd-1_1 | candidate: &{Xid:-2 Zxid:11 Err:0}
zetcd-1_1 | oracle: &{Xid:-2 Zxid:14 Err:0}
zetcd-1_1 | I0925 18:38:07.196574 1 conn.go:131] sendXchk Xid:-2 ZXid:14 Resp:&{}
zetcd-1_1 | I0925 18:38:07.196598 1 conn.go:139] conn.Send(xid=-2, zxid=14, &{})
marathon_1 | [2017-09-25 18:38:07,872] INFO Starting Marathon 1.5.0-SNAPSHOT/unknown with --master zk://zetcd:2181/mesos --zk zk://zetcd:2181/marathon (mesosphere.marathon.MarathonApp:JMX exporting thread)
zetcd-2_1 | I0925 18:38:08.217260 1 server.go:110] zkreq={xid:-2 req:*zetcd.PingRequest:&{}}
zetcd-2_1 | I0925 18:38:08.217339 1 zklog.go:73] Ping(-2,{})
zetcd-2_1 | I0925 18:38:08.217702 1 zklog.go:73] Ping(-2,{})
zetcd-2_1 | I0925 18:38:08.217939 1 session.go:122] waitFutureSendResp=&{}
zetcd-2_1 | I0925 18:38:08.218410 1 session.go:142] zk/zkresp=({Xid:-2 Zxid:14 Err:0},&{})
zetcd-2_1 | W0925 18:38:08.218448 1 zk.go:318] xchk failed (zxid mismatch)
zetcd-2_1 | candidate: &{Xid:-2 Zxid:12 Err:0}
zetcd-2_1 | oracle: &{Xid:-2 Zxid:14 Err:0}
zetcd-2_1 | I0925 18:38:08.218491 1 conn.go:131] sendXchk Xid:-2 ZXid:14 Resp:&{}
zetcd-2_1 | I0925 18:38:08.218731 1 conn.go:139] conn.Send(xid=-2, zxid=14, &{})
marathon_1 | [2017-09-25 18:38:08,626] INFO Initiating client connection, connectString=zetcd:2181 sessionTimeout=10000 watcher=org.apache.curator.ConnectionState@305289b3 (org.apache.zookeeper.ZooKeeper:JMX exporting thread)
marathon_1 | [2017-09-25 18:38:08,641] INFO Opening socket connection to server zetcd_zetcd-2_1.zetcd_cluster-net/172.18.0.2:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn:JMX exporting thread-SendThread(zetcd_zetcd-2_1.zetcd_cluster-net:2181))
zetcd-2_1 | I0925 18:38:08.645842 1 server.go:128] accepted remote connection "172.18.0.5:40572"
zetcd-2_1 | I0925 18:38:08.645884 1 session.go:54] newSession([zookeeper:2181])
marathon_1 | [2017-09-25 18:38:08,646] INFO Socket connection established to zetcd_zetcd-2_1.zetcd_cluster-net/172.18.0.2:2181, initiating session (org.apache.zookeeper.ClientCnxn:JMX exporting thread-SendThread(zetcd_zetcd-2_1.zetcd_cluster-net:2181))
zetcd-2_1 | I0925 18:38:08.648698 1 authconn.go:53] auth(&{ProtocolVersion:0 LastZxidSeen:0 TimeOut:10000 SessionID:0 Passwd:[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]})
zetcd-2_1 | I0925 18:38:08.651085 1 pool.go:92] authresp=&{ProtocolVersion:0 TimeOut:10000 SessionID:7635112897658448681 Passwd:[191 0 102 207 230 195 224 71 223 255 65 185 152 22 222 59]}
zetcd-2_1 | I0925 18:38:08.656279 1 session.go:59] starting the session... id=7635112897658448681
zetcd-2_1 | I0925 18:38:08.656378 1 session.go:90] auth resp OK ({ProtocolVersion:0 TimeOut:10000 SessionID:98721115828125705 Passwd:[170 143 59 25 83 123 20 83 171 174 254 242 37 169 170 171]})
zetcd-2_1 | I0925 18:38:08.656444 1 server.go:46] serving global serial session requests on s=&{Conn:0xc420261440 oracle:0xc4200c8be0 candidate:0xc4201c4480 req:{ProtocolVersion:0 LastZxidSeen:0 TimeOut:10000 SessionID:0 Passwd:[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]} sp:0xc4201aa020}
marathon_1 | [2017-09-25 18:38:08,660] WARN Connected to an old server; r-o mode will be unavailable (org.apache.zookeeper.ClientCnxnSocket:JMX exporting thread-SendThread(zetcd_zetcd-2_1.zetcd_cluster-net:2181))
marathon_1 | [2017-09-25 18:38:08,660] INFO Session establishment complete on server zetcd_zetcd-2_1.zetcd_cluster-net/172.18.0.2:2181, sessionid = 0x15eba54f0c40009, negotiated timeout = 10000 (org.apache.zookeeper.ClientCnxn:JMX exporting thread-SendThread(zetcd_zetcd-2_1.zetcd_cluster-net:2181))
zetcd-1_1 | I0925 18:38:08.827585 1 server.go:110] zkreq={xid:-2 req:*zetcd.PingRequest:&{}}
zetcd-1_1 | I0925 18:38:08.827662 1 zklog.go:73] Ping(-2,{})
zetcd-1_1 | I0925 18:38:08.827726 1 zklog.go:73] Ping(-2,{})
zetcd-1_1 | I0925 18:38:08.828757 1 session.go:122] waitFutureSendResp=&{}
zetcd-1_1 | I0925 18:38:08.829207 1 session.go:142] zk/zkresp=({Xid:-2 Zxid:15 Err:0},&{})
zetcd-1_1 | W0925 18:38:08.829277 1 zk.go:318] xchk failed (zxid mismatch)
zetcd-1_1 | candidate: &{Xid:-2 Zxid:13 Err:0}
zetcd-1_1 | oracle: &{Xid:-2 Zxid:15 Err:0}
zetcd-1_1 | I0925 18:38:08.829350 1 conn.go:131] sendXchk Xid:-2 ZXid:15 Resp:&{}
zetcd-1_1 | I0925 18:38:08.829613 1 conn.go:139] conn.Send(xid=-2, zxid=15, &{})
zetcd-2_1 | I0925 18:38:09.205591 1 server.go:110] zkreq={xid:1 req:*zetcd.ExistsRequest:&{Path:/marathon Watch:false}}
zetcd-2_1 | I0925 18:38:09.205634 1 zklog.go:38] Exists(1,{Path:/marathon Watch:false})
zetcd-2_1 | I0925 18:38:09.205887 1 session.go:122] waitFutureSendResp=&{Path:/marathon Watch:false}
zetcd-2_1 | I0925 18:38:09.205674 1 zklog.go:38] Exists(1,{Path:/marathon Watch:false})
zetcd-2_1 | I0925 18:38:09.206596 1 session.go:142] zk/zkresp=({Xid:1 Zxid:15 Err:-101},<nil>)
zetcd-2_1 | I0925 18:38:09.206630 1 conn.go:131] sendXchk Xid:1 ZXid:15 Resp:0xc42026a450
zetcd-2_1 | I0925 18:38:09.207105 1 conn.go:139] conn.Send(xid=1, zxid=15, 0xc42026a450)
zetcd-2_1 | I0925 18:38:09.210195 1 server.go:110] zkreq={xid:2 req:*zetcd.CreateRequest:&{Path:/marathon Data:[] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:0}}
zetcd-2_1 | I0925 18:38:09.210273 1 zklog.go:28] Create(2,{Path:/marathon Data:[] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:0})
zetcd-2_1 | I0925 18:38:09.210289 1 zklog.go:28] Create(2,{Path:/marathon Data:[] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:0})
zetcd-2_1 | I0925 18:38:09.210486 1 session.go:122] waitFutureSendResp=&{Path:/marathon Data:[] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:0}
zetcd-2_1 | I0925 18:38:09.211685 1 zketcd.go:52] Create(2) = (zxid=17); txnresp: {Header:cluster_id:17810356679686510411 member_id:17456590584391395829 revision:17 raft_term:2 Succeeded:true Responses:[response_put:<header:<revision:17 > > response_put:<header:<revision:17 > > response_put:<header:<revision:17 > > response_put:<header:<revision:17 > > response_put:<header:<revision:17 > > response_put:<header:<revision:17 > > response_put:<header:<revision:17 > > response_put:<header:<revision:17 > > response_put:<header:<revision:17 > > ]}
zetcd-2_1 | I0925 18:38:09.212816 1 session.go:142] zk/zkresp=({Xid:2 Zxid:16 Err:0},&{Path:/marathon})
zetcd-2_1 | I0925 18:38:09.212850 1 conn.go:131] sendXchk Xid:2 ZXid:16 Resp:&{Path:/marathon}
zetcd-2_1 | I0925 18:38:09.212888 1 conn.go:139] conn.Send(xid=2, zxid=16, &{Path:/marathon})
zetcd-2_1 | I0925 18:38:09.214932 1 server.go:110] zkreq={xid:3 req:*zetcd.ExistsRequest:&{Path:/marathon/state Watch:false}}
zetcd-2_1 | I0925 18:38:09.214970 1 zklog.go:38] Exists(3,{Path:/marathon/state Watch:false})
zetcd-2_1 | I0925 18:38:09.214980 1 zklog.go:38] Exists(3,{Path:/marathon/state Watch:false})
zetcd-2_1 | I0925 18:38:09.215183 1 session.go:122] waitFutureSendResp=&{Path:/marathon/state Watch:false}
zetcd-2_1 | I0925 18:38:09.215657 1 session.go:142] zk/zkresp=({Xid:3 Zxid:16 Err:-101},<nil>)
zetcd-2_1 | I0925 18:38:09.215699 1 conn.go:131] sendXchk Xid:3 ZXid:16 Resp:0xc420214d50
zetcd-2_1 | I0925 18:38:09.215907 1 conn.go:139] conn.Send(xid=3, zxid=16, 0xc420214d50)
zetcd-2_1 | I0925 18:38:09.216285 1 server.go:110] zkreq={xid:4 req:*zetcd.CreateRequest:&{Path:/marathon/state Data:[] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:0}}
zetcd-2_1 | I0925 18:38:09.216345 1 zklog.go:28] Create(4,{Path:/marathon/state Data:[] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:0})
zetcd-2_1 | I0925 18:38:09.216323 1 zklog.go:28] Create(4,{Path:/marathon/state Data:[] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:0})
zetcd-2_1 | I0925 18:38:09.216731 1 session.go:122] waitFutureSendResp=&{Path:/marathon/state Data:[] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:0}
zetcd-2_1 | I0925 18:38:09.217923 1 zketcd.go:52] Create(4) = (zxid=18); txnresp: {Header:cluster_id:17810356679686510411 member_id:17456590584391395829 revision:18 raft_term:2 Succeeded:true Responses:[response_put:<header:<revision:18 > > response_put:<header:<revision:18 > > response_put:<header:<revision:18 > > response_put:<header:<revision:18 > > response_put:<header:<revision:18 > > response_put:<header:<revision:18 > > response_put:<header:<revision:18 > > response_put:<header:<revision:18 > > response_put:<header:<revision:18 > > ]}
zetcd-2_1 | I0925 18:38:09.219064 1 session.go:142] zk/zkresp=({Xid:4 Zxid:17 Err:0},&{Path:/marathon/state})
zetcd-2_1 | I0925 18:38:09.219084 1 conn.go:131] sendXchk Xid:4 ZXid:17 Resp:&{Path:/marathon/state}
zetcd-2_1 | I0925 18:38:09.219106 1 conn.go:139] conn.Send(xid=4, zxid=17, &{Path:/marathon/state})
zetcd-2_1 | I0925 18:38:09.221677 1 server.go:110] zkreq={xid:5 req:*zetcd.GetDataRequest:&{Path:/marathon/state/group/2/root Watch:false}}
zetcd-2_1 | I0925 18:38:09.221717 1 zklog.go:43] GetData(5,{Path:/marathon/state/group/2/root Watch:false})
zetcd-2_1 | I0925 18:38:09.222249 1 zklog.go:43] GetData(5,{Path:/marathon/state/group/2/root Watch:false})
zetcd-2_1 | I0925 18:38:09.222980 1 session.go:122] waitFutureSendResp=&{Path:/marathon/state/group/2/root Watch:false}
zetcd-2_1 | I0925 18:38:09.223635 1 session.go:142] zk/zkresp=({Xid:5 Zxid:17 Err:-101},<nil>)
zetcd-2_1 | I0925 18:38:09.223654 1 conn.go:131] sendXchk Xid:5 ZXid:17 Resp:0xc42013f050
zetcd-2_1 | I0925 18:38:09.223855 1 conn.go:139] conn.Send(xid=5, zxid=17, 0xc42013f050)
zetcd-2_1 | I0925 18:38:09.335283 1 server.go:110] zkreq={xid:6 req:*zetcd.SetDataRequest:&{Path:/marathon/state/group/2/root Data:[10 0 18 20 49 57 55 48 45 48 49 45 48 49 84 48 48 58 48 48 58 48 48 90] Version:-1}}
zetcd-2_1 | I0925 18:38:09.335345 1 zklog.go:48] SetData(6,{Path:/marathon/state/group/2/root Data:[10 0 18 20 49 57 55 48 45 48 49 45 48 49 84 48 48 58 48 48 58 48 48 90] Version:-1})
zetcd-2_1 | I0925 18:38:09.335416 1 zklog.go:48] SetData(6,{Path:/marathon/state/group/2/root Data:[10 0 18 20 49 57 55 48 45 48 49 45 48 49 84 48 48 58 48 48 58 48 48 90] Version:-1})
zetcd-2_1 | I0925 18:38:09.335632 1 session.go:122] waitFutureSendResp=&{Path:/marathon/state/group/2/root Data:[10 0 18 20 49 57 55 48 45 48 49 45 48 49 84 48 48 58 48 48 58 48 48 90] Version:-1}
zetcd-2_1 | I0925 18:38:09.339691 1 session.go:142] zk/zkresp=({Xid:6 Zxid:18 Err:-101},<nil>)
zetcd-2_1 | I0925 18:38:09.339719 1 conn.go:131] sendXchk Xid:6 ZXid:18 Resp:0xc42026a030
zetcd-2_1 | I0925 18:38:09.339915 1 conn.go:139] conn.Send(xid=6, zxid=18, 0xc42026a030)
zetcd-2_1 | I0925 18:38:09.339923 1 server.go:110] zkreq={xid:7 req:*zetcd.SetDataRequest:&{Path:/marathon/state/group/2/root/1970-01-01T00:00:00Z Data:[10 0 18 20 49 57 55 48 45 48 49 45 48 49 84 48 48 58 48 48 58 48 48 90] Version:-1}}
zetcd-2_1 | I0925 18:38:09.339958 1 zklog.go:48] SetData(7,{Path:/marathon/state/group/2/root/1970-01-01T00:00:00Z Data:[10 0 18 20 49 57 55 48 45 48 49 45 48 49 84 48 48 58 48 48 58 48 48 90] Version:-1})
zetcd-2_1 | I0925 18:38:09.340085 1 zklog.go:48] SetData(7,{Path:/marathon/state/group/2/root/1970-01-01T00:00:00Z Data:[10 0 18 20 49 57 55 48 45 48 49 45 48 49 84 48 48 58 48 48 58 48 48 90] Version:-1})
zetcd-2_1 | I0925 18:38:09.340226 1 session.go:122] waitFutureSendResp=&{Path:/marathon/state/group/2/root/1970-01-01T00:00:00Z Data:[10 0 18 20 49 57 55 48 45 48 49 45 48 49 84 48 48 58 48 48 58 48 48 90] Version:-1}
zetcd-2_1 | I0925 18:38:09.344430 1 session.go:142] zk/zkresp=({Xid:7 Zxid:19 Err:-101},<nil>)
zetcd-2_1 | I0925 18:38:09.344458 1 conn.go:131] sendXchk Xid:7 ZXid:19 Resp:0xc42018e730
zetcd-2_1 | I0925 18:38:09.344478 1 conn.go:139] conn.Send(xid=7, zxid=19, 0xc42018e730)
zetcd-2_1 | I0925 18:38:09.353424 1 server.go:110] zkreq={xid:8 req:*zetcd.CreateRequest:&{Path:/marathon/state/group/2/root Data:[10 0 18 20 49 57 55 48 45 48 49 45 48 49 84 48 48 58 48 48 58 48 48 90] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:0}}
zetcd-2_1 | I0925 18:38:09.353496 1 zklog.go:28] Create(8,{Path:/marathon/state/group/2/root Data:[10 0 18 20 49 57 55 48 45 48 49 45 48 49 84 48 48 58 48 48 58 48 48 90] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:0})
zetcd-2_1 | I0925 18:38:09.353610 1 zklog.go:28] Create(8,{Path:/marathon/state/group/2/root Data:[10 0 18 20 49 57 55 48 45 48 49 45 48 49 84 48 48 58 48 48 58 48 48 90] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:0})
zetcd-2_1 | I0925 18:38:09.354079 1 session.go:122] waitFutureSendResp=&{Path:/marathon/state/group/2/root Data:[10 0 18 20 49 57 55 48 45 48 49 45 48 49 84 48 48 58 48 48 58 48 48 90] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:0}
zetcd-2_1 | I0925 18:38:09.358526 1 session.go:142] zk/zkresp=({Xid:8 Zxid:20 Err:-101},<nil>)
zetcd-2_1 | I0925 18:38:09.358581 1 conn.go:131] sendXchk Xid:8 ZXid:20 Resp:0xc4200b8530
zetcd-2_1 | I0925 18:38:09.359038 1 conn.go:139] conn.Send(xid=8, zxid=20, 0xc4200b8530)
zetcd-2_1 | I0925 18:38:09.359067 1 server.go:110] zkreq={xid:9 req:*zetcd.CreateRequest:&{Path:/marathon/state/group/2/root/1970-01-01T00:00:00Z Data:[10 0 18 20 49 57 55 48 45 48 49 45 48 49 84 48 48 58 48 48 58 48 48 90] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:0}}
zetcd-2_1 | I0925 18:38:09.359817 1 zklog.go:28] Create(9,{Path:/marathon/state/group/2/root/1970-01-01T00:00:00Z Data:[10 0 18 20 49 57 55 48 45 48 49 45 48 49 84 48 48 58 48 48 58 48 48 90] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:0})
zetcd-2_1 | I0925 18:38:09.360447 1 zklog.go:28] Create(9,{Path:/marathon/state/group/2/root/1970-01-01T00:00:00Z Data:[10 0 18 20 49 57 55 48 45 48 49 45 48 49 84 48 48 58 48 48 58 48 48 90] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:0})
zetcd-2_1 | I0925 18:38:09.361123 1 session.go:122] waitFutureSendResp=&{Path:/marathon/state/group/2/root/1970-01-01T00:00:00Z Data:[10 0 18 20 49 57 55 48 45 48 49 45 48 49 84 48 48 58 48 48 58 48 48 90] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:0}
zetcd-2_1 | I0925 18:38:09.365474 1 session.go:142] zk/zkresp=({Xid:9 Zxid:21 Err:-101},<nil>)
zetcd-2_1 | I0925 18:38:09.365515 1 conn.go:131] sendXchk Xid:9 ZXid:21 Resp:0xc4200b8630
zetcd-2_1 | I0925 18:38:09.365747 1 conn.go:139] conn.Send(xid=9, zxid=21, 0xc4200b8630)
zetcd-2_1 | I0925 18:38:09.365790 1 server.go:110] zkreq={xid:10 req:*zetcd.ExistsRequest:&{Path:/marathon Watch:false}}
zetcd-2_1 | I0925 18:38:09.365849 1 zklog.go:38] Exists(10,{Path:/marathon Watch:false})
zetcd-2_1 | I0925 18:38:09.366045 1 session.go:122] waitFutureSendResp=&{Path:/marathon Watch:false}
zetcd-2_1 | I0925 18:38:09.366153 1 zklog.go:38] Exists(10,{Path:/marathon Watch:false})
zetcd-2_1 | I0925 18:38:09.366598 1 zketcd.go:285] Exists(10) = (zxid=22, resp={Stat:{Czxid:16 Mzxid:16 Ctime:1506364689211158 Mtime:1506364689211158 Version:0 Cversion:1 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:1 Pzxid:17}})
zetcd-2_1 | I0925 18:38:09.366845 1 session.go:142] zk/zkresp=({Xid:10 Zxid:21 Err:0},&{Stat:{Czxid:16 Mzxid:16 Ctime:1506364689210 Mtime:1506364689210 Version:0 Cversion:1 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:1 Pzxid:17}})
zetcd-2_1 | I0925 18:38:09.366901 1 conn.go:131] sendXchk Xid:10 ZXid:21 Resp:&{Stat:{Czxid:16 Mzxid:16 Ctime:1506364689210 Mtime:1506364689210 Version:0 Cversion:1 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:1 Pzxid:17}}
zetcd-2_1 | I0925 18:38:09.367098 1 conn.go:139] conn.Send(xid=10, zxid=21, &{Stat:{Czxid:16 Mzxid:16 Ctime:1506364689210 Mtime:1506364689210 Version:0 Cversion:1 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:1 Pzxid:17}})
zetcd-2_1 | I0925 18:38:09.368728 1 server.go:110] zkreq={xid:11 req:*zetcd.ExistsRequest:&{Path:/marathon/state Watch:false}}
zetcd-2_1 | I0925 18:38:09.369022 1 zklog.go:38] Exists(11,{Path:/marathon/state Watch:false})
zetcd-2_1 | I0925 18:38:09.369271 1 session.go:122] waitFutureSendResp=&{Path:/marathon/state Watch:false}
zetcd-2_1 | I0925 18:38:09.369025 1 zklog.go:38] Exists(11,{Path:/marathon/state Watch:false})
zetcd-2_1 | I0925 18:38:09.370067 1 session.go:142] zk/zkresp=({Xid:11 Zxid:21 Err:0},&{Stat:{Czxid:17 Mzxid:17 Ctime:1506364689216 Mtime:1506364689216 Version:0 Cversion:0 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:0 Pzxid:17}})
zetcd-2_1 | I0925 18:38:09.373331 1 zketcd.go:285] Exists(11) = (zxid=22, resp={Stat:{Czxid:17 Mzxid:17 Ctime:1506364689217357 Mtime:1506364689217357 Version:0 Cversion:0 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:0 Pzxid:17}})
zetcd-2_1 | I0925 18:38:09.373360 1 conn.go:131] sendXchk Xid:11 ZXid:21 Resp:&{Stat:{Czxid:17 Mzxid:17 Ctime:1506364689216 Mtime:1506364689216 Version:0 Cversion:0 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:0 Pzxid:17}}
zetcd-2_1 | I0925 18:38:09.373379 1 conn.go:139] conn.Send(xid=11, zxid=21, &{Stat:{Czxid:17 Mzxid:17 Ctime:1506364689216 Mtime:1506364689216 Version:0 Cversion:0 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:0 Pzxid:17}})
zetcd-2_1 | I0925 18:38:09.373776 1 server.go:110] zkreq={xid:12 req:*zetcd.ExistsRequest:&{Path:/marathon/state/group Watch:false}}
zetcd-2_1 | I0925 18:38:09.373812 1 zklog.go:38] Exists(12,{Path:/marathon/state/group Watch:false})
zetcd-2_1 | I0925 18:38:09.374301 1 zklog.go:38] Exists(12,{Path:/marathon/state/group Watch:false})
zetcd-2_1 | I0925 18:38:09.375135 1 session.go:122] waitFutureSendResp=&{Path:/marathon/state/group Watch:false}
zetcd-2_1 | I0925 18:38:09.375776 1 session.go:142] zk/zkresp=({Xid:12 Zxid:21 Err:-101},<nil>)
zetcd-2_1 | I0925 18:38:09.375809 1 conn.go:131] sendXchk Xid:12 ZXid:21 Resp:0xc4201562b0
zetcd-2_1 | I0925 18:38:09.376030 1 conn.go:139] conn.Send(xid=12, zxid=21, 0xc4201562b0)
zetcd-2_1 | I0925 18:38:09.376374 1 server.go:110] zkreq={xid:13 req:*zetcd.CreateRequest:&{Path:/marathon/state/group Data:[] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:0}}
zetcd-2_1 | I0925 18:38:09.376450 1 zklog.go:28] Create(13,{Path:/marathon/state/group Data:[] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:0})
zetcd-2_1 | I0925 18:38:09.376766 1 session.go:122] waitFutureSendResp=&{Path:/marathon/state/group Data:[] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:0}
zetcd-2_1 | I0925 18:38:09.376838 1 zklog.go:28] Create(13,{Path:/marathon/state/group Data:[] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:0})
zetcd-2_1 | I0925 18:38:09.378379 1 zketcd.go:52] Create(13) = (zxid=23); txnresp: {Header:cluster_id:17810356679686510411 member_id:17456590584391395829 revision:23 raft_term:2 Succeeded:true Responses:[response_put:<header:<revision:23 > > response_put:<header:<revision:23 > > response_put:<header:<revision:23 > > response_put:<header:<revision:23 > > response_put:<header:<revision:23 > > response_put:<header:<revision:23 > > response_put:<header:<revision:23 > > response_put:<header:<revision:23 > > response_put:<header:<revision:23 > > ]}
zetcd-2_1 | I0925 18:38:09.379193 1 session.go:142] zk/zkresp=({Xid:13 Zxid:22 Err:0},&{Path:/marathon/state/group})
zetcd-2_1 | I0925 18:38:09.379229 1 conn.go:131] sendXchk Xid:13 ZXid:22 Resp:&{Path:/marathon/state/group}
zetcd-2_1 | I0925 18:38:09.379447 1 conn.go:139] conn.Send(xid=13, zxid=22, &{Path:/marathon/state/group})
zetcd-2_1 | I0925 18:38:09.379825 1 server.go:110] zkreq={xid:14 req:*zetcd.ExistsRequest:&{Path:/marathon/state/group/2 Watch:false}}
zetcd-2_1 | I0925 18:38:09.380074 1 zklog.go:38] Exists(14,{Path:/marathon/state/group/2 Watch:false})
zetcd-2_1 | I0925 18:38:09.380272 1 session.go:122] waitFutureSendResp=&{Path:/marathon/state/group/2 Watch:false}
zetcd-2_1 | I0925 18:38:09.380362 1 zklog.go:38] Exists(14,{Path:/marathon/state/group/2 Watch:false})
zetcd-2_1 | I0925 18:38:09.380996 1 session.go:142] zk/zkresp=({Xid:14 Zxid:22 Err:-101},<nil>)
zetcd-2_1 | I0925 18:38:09.381237 1 conn.go:131] sendXchk Xid:14 ZXid:22 Resp:0xc420157b30
zetcd-2_1 | I0925 18:38:09.381278 1 conn.go:139] conn.Send(xid=14, zxid=22, 0xc420157b30)
zetcd-2_1 | I0925 18:38:09.381672 1 server.go:110] zkreq={xid:15 req:*zetcd.CreateRequest:&{Path:/marathon/state/group/2 Data:[] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:0}}
zetcd-2_1 | I0925 18:38:09.381705 1 zklog.go:28] Create(15,{Path:/marathon/state/group/2 Data:[] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:0})
zetcd-2_1 | I0925 18:38:09.381901 1 session.go:122] waitFutureSendResp=&{Path:/marathon/state/group/2 Data:[] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:0}
zetcd-2_1 | I0925 18:38:09.381944 1 zklog.go:28] Create(15,{Path:/marathon/state/group/2 Data:[] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:0})
zetcd-2_1 | I0925 18:38:09.386164 1 zketcd.go:52] Create(15) = (zxid=24); txnresp: {Header:cluster_id:17810356679686510411 member_id:17456590584391395829 revision:24 raft_term:2 Succeeded:true Responses:[response_put:<header:<revision:24 > > response_put:<header:<revision:24 > > response_put:<header:<revision:24 > > response_put:<header:<revision:24 > > response_put:<header:<revision:24 > > response_put:<header:<revision:24 > > response_put:<header:<revision:24 > > response_put:<header:<revision:24 > > response_put:<header:<revision:24 > > ]}
zetcd-2_1 | I0925 18:38:09.386781 1 session.go:142] zk/zkresp=({Xid:15 Zxid:23 Err:0},&{Path:/marathon/state/group/2})
zetcd-2_1 | I0925 18:38:09.387010 1 conn.go:131] sendXchk Xid:15 ZXid:23 Resp:&{Path:/marathon/state/group/2}
zetcd-2_1 | I0925 18:38:09.387028 1 conn.go:139] conn.Send(xid=15, zxid=23, &{Path:/marathon/state/group/2})
zetcd-2_1 | I0925 18:38:09.387589 1 server.go:110] zkreq={xid:16 req:*zetcd.CreateRequest:&{Path:/marathon/state/group/2/root Data:[10 0 18 20 49 57 55 48 45 48 49 45 48 49 84 48 48 58 48 48 58 48 48 90] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:0}}
zetcd-2_1 | I0925 18:38:09.387679 1 zklog.go:28] Create(16,{Path:/marathon/state/group/2/root Data:[10 0 18 20 49 57 55 48 45 48 49 45 48 49 84 48 48 58 48 48 58 48 48 90] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:0})
zetcd-2_1 | I0925 18:38:09.387904 1 zklog.go:28] Create(16,{Path:/marathon/state/group/2/root Data:[10 0 18 20 49 57 55 48 45 48 49 45 48 49 84 48 48 58 48 48 58 48 48 90] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:0})
zetcd-2_1 | I0925 18:38:09.388474 1 session.go:122] waitFutureSendResp=&{Path:/marathon/state/group/2/root Data:[10 0 18 20 49 57 55 48 45 48 49 45 48 49 84 48 48 58 48 48 58 48 48 90] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:0}
zetcd-2_1 | I0925 18:38:09.390134 1 zketcd.go:52] Create(16) = (zxid=25); txnresp: {Header:cluster_id:17810356679686510411 member_id:17456590584391395829 revision:25 raft_term:2 Succeeded:true Responses:[response_put:<header:<revision:25 > > response_put:<header:<revision:25 > > response_put:<header:<revision:25 > > response_put:<header:<revision:25 > > response_put:<header:<revision:25 > > response_put:<header:<revision:25 > > response_put:<header:<revision:25 > > response_put:<header:<revision:25 > > response_put:<header:<revision:25 > > ]}
zetcd-2_1 | I0925 18:38:09.391157 1 session.go:142] zk/zkresp=({Xid:16 Zxid:24 Err:0},&{Path:/marathon/state/group/2/root})
zetcd-2_1 | I0925 18:38:09.391248 1 conn.go:131] sendXchk Xid:16 ZXid:24 Resp:&{Path:/marathon/state/group/2/root}
zetcd-2_1 | I0925 18:38:09.391476 1 conn.go:139] conn.Send(xid=16, zxid=24, &{Path:/marathon/state/group/2/root})
zetcd-2_1 | I0925 18:38:09.391487 1 server.go:110] zkreq={xid:17 req:*zetcd.ExistsRequest:&{Path:/marathon Watch:false}}
zetcd-2_1 | I0925 18:38:09.391517 1 zklog.go:38] Exists(17,{Path:/marathon Watch:false})
zetcd-2_1 | I0925 18:38:09.391673 1 zklog.go:38] Exists(17,{Path:/marathon Watch:false})
zetcd-2_1 | I0925 18:38:09.391695 1 session.go:122] waitFutureSendResp=&{Path:/marathon Watch:false}
zetcd-2_1 | I0925 18:38:09.392178 1 session.go:142] zk/zkresp=({Xid:17 Zxid:24 Err:0},&{Stat:{Czxid:16 Mzxid:16 Ctime:1506364689210 Mtime:1506364689210 Version:0 Cversion:1 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:1 Pzxid:17}})
zetcd-2_1 | I0925 18:38:09.392509 1 zketcd.go:285] Exists(17) = (zxid=25, resp={Stat:{Czxid:16 Mzxid:16 Ctime:1506364689211158 Mtime:1506364689211158 Version:0 Cversion:1 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:1 Pzxid:17}})
zetcd-2_1 | I0925 18:38:09.392534 1 conn.go:131] sendXchk Xid:17 ZXid:24 Resp:&{Stat:{Czxid:16 Mzxid:16 Ctime:1506364689210 Mtime:1506364689210 Version:0 Cversion:1 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:1 Pzxid:17}}
zetcd-2_1 | I0925 18:38:09.392704 1 conn.go:139] conn.Send(xid=17, zxid=24, &{Stat:{Czxid:16 Mzxid:16 Ctime:1506364689210 Mtime:1506364689210 Version:0 Cversion:1 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:1 Pzxid:17}})
zetcd-2_1 | I0925 18:38:09.393140 1 server.go:110] zkreq={xid:18 req:*zetcd.ExistsRequest:&{Path:/marathon/state Watch:false}}
zetcd-2_1 | I0925 18:38:09.393174 1 zklog.go:38] Exists(18,{Path:/marathon/state Watch:false})
zetcd-2_1 | I0925 18:38:09.393273 1 zklog.go:38] Exists(18,{Path:/marathon/state Watch:false})
zetcd-2_1 | I0925 18:38:09.393652 1 zketcd.go:285] Exists(18) = (zxid=25, resp={Stat:{Czxid:17 Mzxid:17 Ctime:1506364689217357 Mtime:1506364689217357 Version:0 Cversion:1 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:1 Pzxid:22}})
zetcd-2_1 | I0925 18:38:09.393652 1 session.go:122] waitFutureSendResp=&{Path:/marathon/state Watch:false}
zetcd-2_1 | I0925 18:38:09.394022 1 session.go:142] zk/zkresp=({Xid:18 Zxid:24 Err:0},&{Stat:{Czxid:17 Mzxid:17 Ctime:1506364689216 Mtime:1506364689216 Version:0 Cversion:1 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:1 Pzxid:22}})
zetcd-2_1 | I0925 18:38:09.394042 1 conn.go:131] sendXchk Xid:18 ZXid:24 Resp:&{Stat:{Czxid:17 Mzxid:17 Ctime:1506364689216 Mtime:1506364689216 Version:0 Cversion:1 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:1 Pzxid:22}}
zetcd-2_1 | I0925 18:38:09.394227 1 conn.go:139] conn.Send(xid=18, zxid=24, &{Stat:{Czxid:17 Mzxid:17 Ctime:1506364689216 Mtime:1506364689216 Version:0 Cversion:1 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:1 Pzxid:22}})
zetcd-2_1 | I0925 18:38:09.394714 1 server.go:110] zkreq={xid:19 req:*zetcd.ExistsRequest:&{Path:/marathon/state/group Watch:false}}
zetcd-2_1 | I0925 18:38:09.394747 1 zklog.go:38] Exists(19,{Path:/marathon/state/group Watch:false})
zetcd-2_1 | I0925 18:38:09.394751 1 zklog.go:38] Exists(19,{Path:/marathon/state/group Watch:false})
zetcd-2_1 | I0925 18:38:09.394958 1 session.go:122] waitFutureSendResp=&{Path:/marathon/state/group Watch:false}
zetcd-2_1 | I0925 18:38:09.395315 1 zketcd.go:285] Exists(19) = (zxid=25, resp={Stat:{Czxid:22 Mzxid:22 Ctime:1506364689377934 Mtime:1506364689377934 Version:0 Cversion:1 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:1 Pzxid:23}})
zetcd-2_1 | I0925 18:38:09.395347 1 session.go:142] zk/zkresp=({Xid:19 Zxid:24 Err:0},&{Stat:{Czxid:22 Mzxid:22 Ctime:1506364689377 Mtime:1506364689377 Version:0 Cversion:1 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:1 Pzxid:23}})
zetcd-2_1 | I0925 18:38:09.395390 1 conn.go:131] sendXchk Xid:19 ZXid:24 Resp:&{Stat:{Czxid:22 Mzxid:22 Ctime:1506364689377 Mtime:1506364689377 Version:0 Cversion:1 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:1 Pzxid:23}}
zetcd-2_1 | I0925 18:38:09.395413 1 conn.go:139] conn.Send(xid=19, zxid=24, &{Stat:{Czxid:22 Mzxid:22 Ctime:1506364689377 Mtime:1506364689377 Version:0 Cversion:1 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:1 Pzxid:23}})
zetcd-2_1 | I0925 18:38:09.395775 1 server.go:110] zkreq={xid:20 req:*zetcd.ExistsRequest:&{Path:/marathon/state/group/2 Watch:false}}
zetcd-2_1 | I0925 18:38:09.395800 1 zklog.go:38] Exists(20,{Path:/marathon/state/group/2 Watch:false})
zetcd-2_1 | I0925 18:38:09.396413 1 session.go:122] waitFutureSendResp=&{Path:/marathon/state/group/2 Watch:false}
zetcd-2_1 | I0925 18:38:09.397126 1 zklog.go:38] Exists(20,{Path:/marathon/state/group/2 Watch:false})
zetcd-2_1 | I0925 18:38:09.397316 1 session.go:142] zk/zkresp=({Xid:20 Zxid:24 Err:0},&{Stat:{Czxid:23 Mzxid:23 Ctime:1506364689382 Mtime:1506364689382 Version:0 Cversion:1 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:1 Pzxid:24}})
zetcd-2_1 | I0925 18:38:09.397884 1 zketcd.go:285] Exists(20) = (zxid=25, resp={Stat:{Czxid:23 Mzxid:23 Ctime:1506364689385664 Mtime:1506364689385664 Version:0 Cversion:1 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:1 Pzxid:24}})
zetcd-2_1 | I0925 18:38:09.397906 1 conn.go:131] sendXchk Xid:20 ZXid:24 Resp:&{Stat:{Czxid:23 Mzxid:23 Ctime:1506364689382 Mtime:1506364689382 Version:0 Cversion:1 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:1 Pzxid:24}}
zetcd-2_1 | I0925 18:38:09.398096 1 conn.go:139] conn.Send(xid=20, zxid=24, &{Stat:{Czxid:23 Mzxid:23 Ctime:1506364689382 Mtime:1506364689382 Version:0 Cversion:1 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:1 Pzxid:24}})
zetcd-2_1 | I0925 18:38:09.398455 1 server.go:110] zkreq={xid:21 req:*zetcd.ExistsRequest:&{Path:/marathon/state/group/2/root Watch:false}}
zetcd-2_1 | I0925 18:38:09.398490 1 zklog.go:38] Exists(21,{Path:/marathon/state/group/2/root Watch:false})
zetcd-2_1 | I0925 18:38:09.398494 1 zklog.go:38] Exists(21,{Path:/marathon/state/group/2/root Watch:false})
zetcd-2_1 | I0925 18:38:09.398688 1 session.go:122] waitFutureSendResp=&{Path:/marathon/state/group/2/root Watch:false}
zetcd-2_1 | I0925 18:38:09.398965 1 zketcd.go:285] Exists(21) = (zxid=25, resp={Stat:{Czxid:24 Mzxid:24 Ctime:1506364689389606 Mtime:1506364689389606 Version:0 Cversion:0 Aversion:0 EphemeralOwner:0 DataLength:24 NumChildren:0 Pzxid:24}})
zetcd-2_1 | I0925 18:38:09.399136 1 session.go:142] zk/zkresp=({Xid:21 Zxid:24 Err:0},&{Stat:{Czxid:24 Mzxid:24 Ctime:1506364689389 Mtime:1506364689389 Version:0 Cversion:0 Aversion:0 EphemeralOwner:0 DataLength:24 NumChildren:0 Pzxid:24}})
zetcd-2_1 | I0925 18:38:09.399158 1 conn.go:131] sendXchk Xid:21 ZXid:24 Resp:&{Stat:{Czxid:24 Mzxid:24 Ctime:1506364689389 Mtime:1506364689389 Version:0 Cversion:0 Aversion:0 EphemeralOwner:0 DataLength:24 NumChildren:0 Pzxid:24}}
zetcd-2_1 | I0925 18:38:09.399466 1 conn.go:139] conn.Send(xid=21, zxid=24, &{Stat:{Czxid:24 Mzxid:24 Ctime:1506364689389 Mtime:1506364689389 Version:0 Cversion:0 Aversion:0 EphemeralOwner:0 DataLength:24 NumChildren:0 Pzxid:24}})
zetcd-2_1 | I0925 18:38:09.399869 1 server.go:110] zkreq={xid:22 req:*zetcd.CreateRequest:&{Path:/marathon/state/group/2/root/1970-01-01T00:00:00Z Data:[10 0 18 20 49 57 55 48 45 48 49 45 48 49 84 48 48 58 48 48 58 48 48 90] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:0}}
zetcd-2_1 | I0925 18:38:09.399904 1 zklog.go:28] Create(22,{Path:/marathon/state/group/2/root/1970-01-01T00:00:00Z Data:[10 0 18 20 49 57 55 48 45 48 49 45 48 49 84 48 48 58 48 48 58 48 48 90] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:0})
zetcd-2_1 | I0925 18:38:09.400276 1 session.go:122] waitFutureSendResp=&{Path:/marathon/state/group/2/root/1970-01-01T00:00:00Z Data:[10 0 18 20 49 57 55 48 45 48 49 45 48 49 84 48 48 58 48 48 58 48 48 90] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:0}
zetcd-2_1 | I0925 18:38:09.400394 1 zklog.go:28] Create(22,{Path:/marathon/state/group/2/root/1970-01-01T00:00:00Z Data:[10 0 18 20 49 57 55 48 45 48 49 45 48 49 84 48 48 58 48 48 58 48 48 90] Acl:[{Perms:31 Scheme:world ID:anyone}] Flags:0})
zetcd-2_1 | I0925 18:38:09.401675 1 zketcd.go:52] Create(22) = (zxid=26); txnresp: {Header:cluster_id:17810356679686510411 member_id:17456590584391395829 revision:26 raft_term:2 Succeeded:true Responses:[response_put:<header:<revision:26 > > response_put:<header:<revision:26 > > response_put:<header:<revision:26 > > response_put:<header:<revision:26 > > response_put:<header:<revision:26 > > response_put:<header:<revision:26 > > response_put:<header:<revision:26 > > response_put:<header:<revision:26 > > response_put:<header:<revision:26 > > ]}
zetcd-2_1 | I0925 18:38:09.405436 1 session.go:142] zk/zkresp=({Xid:22 Zxid:25 Err:0},&{Path:/marathon/state/group/2/root/1970-01-01T00:00:00Z})
zetcd-2_1 | I0925 18:38:09.405467 1 conn.go:131] sendXchk Xid:22 ZXid:25 Resp:&{Path:/marathon/state/group/2/root/1970-01-01T00:00:00Z}
zetcd-2_1 | I0925 18:38:09.405487 1 conn.go:139] conn.Send(xid=22, zxid=25, &{Path:/marathon/state/group/2/root/1970-01-01T00:00:00Z})
zetcd-2_1 | I0925 18:38:09.675555 1 server.go:110] zkreq={xid:-2 req:*zetcd.PingRequest:&{}}
zetcd-2_1 | I0925 18:38:09.675611 1 zklog.go:73] Ping(-2,{})
zetcd-2_1 | I0925 18:38:09.675792 1 zklog.go:73] Ping(-2,{})
zetcd-2_1 | I0925 18:38:09.676108 1 session.go:122] waitFutureSendResp=&{}The xchk failed above might be benign; I'll investigate more and report.
Even if it is benign though, it would be nice if this was fixed if doable; our strategy for replacing Zookeeper is to use zetcd in front as a proxy, and when a sufficient amount of time has passed without any xchk failure, we'll have the confidence to use zetcd directly.
Thanks.
Thanks for 13e6ff383b4; now Leader Election in Mesos seems to work.
The next step is attaching a Mesos-slave to the Master, but the slave never successfully registers. I tried using zetcd in zkbridge mode (didn't specify -oracle, don't know if it's relevant?), but couldn't find any error or warning in zetcd logs while doing so, and the slave registered correctly.
Mesos-master log:
Mesos-slave log:
Sample of zetcd log: