candlerb
commented
6 years ago Just tried it again. Unfortunately I cannot get either a single-node or multi-node setup running.
Single node
# ./jocko broker
2018/06/16 21:06:36 Initializing logging reporter
2018-06-16T21:06:36.095Z INFO jocko/broker.go:109 hello {"id": 0, "broker addr": "0.0.0.0:9092", "serf addr": "0.0.0.0:9094", "raft addr": "127.0.0.1:9093", "id": 0, "raft addr": "127.0.0.1:9093"}
2018/06/16 21:06:36 [INFO] raft: Initial configuration (index=0): []
2018/06/16 21:06:36 [INFO] raft: Node at 127.0.0.1:9093 [Follower] entering Follower state (Leader: "")
2018/06/16 21:06:36 [INFO] serf: EventMemberJoin: jocko ::
2018-06-16T21:06:36.138Z INFO jocko/server.go:71 hello {"id": 0, "broker addr": "0.0.0.0:9092", "serf addr": "0.0.0.0:9094", "raft addr": "127.0.0.1:9093", "server id": 0, "addr": "0.0.0.0:9092"}
2018-06-16T21:06:36.139Z INFO jocko/serf.go:74 adding LAN server {"id": 0, "broker addr": "0.0.0.0:9092", "serf addr": "0.0.0.0:9094", "raft addr": "127.0.0.1:9093", "id": 0, "raft addr": "127.0.0.1:9093", "meta": {"ID":0,"Name":"","Bootstrap":false,"Expect":0,"NonVoter":false,"Status":1,"RaftAddr":"127.0.0.1:9093","SerfLANAddr":"0.0.0.0:9094:8301","BrokerAddr":"0.0.0.0:9092"}}
2018/06/16 21:06:37 [WARN] raft: no known peers, aborting electionI spy something dubious there: "SerfLANAddr":"0.0.0.0:9094:8301"
(8301 exists in the source code as DefaultLANSerfPort)
In another screen I try to create a topic:
# ./jocko topic create --topic test
error code: not controllerBack in the broker screen I see:
2018/06/16 21:06:56 Reporting span 58a4a97d0a5b8bfe:152681c6a74b2a88:58a4a97d0a5b8bfe:1
2018/06/16 21:06:56 Reporting span 58a4a97d0a5b8bfe:29f53d9eec662474:58a4a97d0a5b8bfe:1
2018/06/16 21:06:56 Reporting span 58a4a97d0a5b8bfe:3ab651b5a79db080:58a4a97d0a5b8bfe:1
2018/06/16 21:06:56 Reporting span 58a4a97d0a5b8bfe:6215d9d3872fa0c8:58a4a97d0a5b8bfe:1
2018/06/16 21:06:56 Reporting span 58a4a97d0a5b8bfe:6cd74c36fd162c6:58a4a97d0a5b8bfe:1
2018/06/16 21:06:56 Reporting span 58a4a97d0a5b8bfe:58a4a97d0a5b8bfe:0:1
2018/06/16 21:07:12 Reporting span 8304d1cf33b493f:48fb487a39f54e3:8304d1cf33b493f:1
2018/06/16 21:07:12 Reporting span 8304d1cf33b493f:3119779192068843:8304d1cf33b493f:1
2018/06/16 21:07:12 Reporting span 8304d1cf33b493f:14c0ac963bda4e32:8304d1cf33b493f:1
2018/06/16 21:07:12 Reporting span 8304d1cf33b493f:61ef62b06b1e53f3:8304d1cf33b493f:1
2018/06/16 21:07:12 Reporting span 8304d1cf33b493f:41950f95b4c41d52:8304d1cf33b493f:1
2018/06/16 21:07:12 Reporting span 8304d1cf33b493f:8304d1cf33b493f:0:1
2018/06/16 21:07:13 ERROR: error when flushing the buffer: write udp 127.0.0.1:60767->127.0.0.1:6831: write: connection refusedI don't know what's supposed to be listening on port 6831; this number doesn't appear in the Jocko source code anywhere. And indeed nothing is listening on this port, although jocko has a connected UDP socket to send to 6831:
# netstat -naup
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
udp 0 0 127.0.0.1:60767 127.0.0.1:6831 ESTABLISHED 15769/jocko
udp 0 0 0.0.0.0:68 0.0.0.0:* 256/dhclient
udp6 0 0 :::8301 :::* 15769/jockoCluster
The instructions in _examples/cluster/README.md have invalid flags. I changed them to:
# ./jocko broker \
--data-dir="/tmp/jocko0" \
--broker-addr=127.0.0.1:9001 \
--raft-addr=127.0.0.1:9002 \
--serf-addr=127.0.0.1:9003 \
--id=1
2018/06/16 21:23:41 Initializing logging reporter
2018-06-16T21:23:41.179Z INFO jocko/broker.go:109 hello {"id": 1, "broker addr": "127.0.0.1:9001", "serf addr": "127.0.0.1:9003", "raft addr": "127.0.0.1:9002", "id": 1, "raft addr": "127.0.0.1:9002"}
2018/06/16 21:23:41 [INFO] raft: Initial configuration (index=0): []
2018/06/16 21:23:41 [INFO] raft: Node at 127.0.0.1:9002 [Follower] entering Follower state (Leader: "")
2018/06/16 21:23:41 [INFO] serf: EventMemberJoin: jocko ::
2018-06-16T21:23:41.223Z INFO jocko/server.go:71 hello {"id": 1, "broker addr": "127.0.0.1:9001", "serf addr": "127.0.0.1:9003", "raft addr": "127.0.0.1:9002", "server id": 1, "addr": "127.0.0.1:9001"}
2018-06-16T21:23:41.226Z INFO jocko/serf.go:74 adding LAN server {"id": 1, "broker addr": "127.0.0.1:9001", "serf addr": "127.0.0.1:9003", "raft addr": "127.0.0.1:9002", "id": 1, "raft addr": "127.0.0.1:9002", "meta": {"ID":1,"Name":"","Bootstrap":false,"Expect":0,"NonVoter":false,"Status":1,"RaftAddr":"127.0.0.1:9002","SerfLANAddr":"127.0.0.1:9003:8301","BrokerAddr":"127.0.0.1:9001"}}
2018/06/16 21:23:43 [WARN] raft: no known peers, aborting election(Note: same problem with SerfLANAddr having two ports)
In another screen, trying to add a second broker:
# ./jocko broker \
--data-dir="/tmp/jocko1" \
--broker-addr=127.0.0.1:9101 \
--raft-addr=127.0.0.1:9102 \
--serf-addr=127.0.0.1:9103 \
--join=127.0.0.1:9003 \
--id=2
2018/06/16 21:24:31 Initializing logging reporter
2018-06-16T21:24:31.488Z INFO jocko/broker.go:109 hello {"id": 2, "broker addr": "127.0.0.1:9101", "serf addr": "127.0.0.1:9103", "raft addr": "127.0.0.1:9102", "id": 2, "raft addr": "127.0.0.1:9102"}
2018/06/16 21:24:31 [INFO] raft: Initial configuration (index=0): []
2018/06/16 21:24:31 [INFO] raft: Node at 127.0.0.1:9102 [Follower] entering Follower state (Leader: "")
error starting broker: Failed to create memberlist: Could not set up network transport: Failed to start TCP listener on "127.0.0.1:9103" port 8301: listen tcp :8301: bind: address already in use
# This one exits because it tries to bind to 8301; that port is already in use by the first process.
Analysis
It seems that meta.SerfLANAddr / serf_lan_addr is assembled from SerfLANConfig.MemberlistConfig.BindAddr and SerfLANConfig.MemberlistConfig.BindPort
$ grep -R serf_lan_addr .
./jocko/leader.go: "serf_lan_addr": meta.SerfLANAddr,
./jocko/metadata/metadata.go: SerfLANAddr: m.Tags["serf_lan_addr"],
./jocko/serf.go: config.Tags["serf_lan_addr"] = fmt.Sprintf("%s:%d", b.config.SerfLANConfig.MemberlistConfig.BindAddr, b.config.SerfLANConfig.MemberlistConfig.BindPort)However, BindAddr defaults to both address and port:
./cmd/jocko/main.go: brokerCmd.Flags().StringVar(&brokerCfg.SerfLANConfig.MemberlistConfig.BindAddr, "serf-addr", "0.0.0.0:9094", "Address for Serf to bind on") // TODO: can set addr alone or need to set bind port separately?And BindPort defaults to 8301, and AFAICS cannot be overridden.
$ grep -R DefaultLANSerfPort .
./jocko/config/config.go: DefaultLANSerfPort = 8301
./jocko/config/config.go: conf.SerfLANConfig.MemberlistConfig.BindPort = DefaultLANSerfPort... although in the test suite, it is set explicitly:
./jocko/testing.go: config.SerfLANConfig.MemberlistConfig.BindPort = ports[2]
./jocko/testing.go: s1.config.SerfLANConfig.MemberlistConfig.BindPort)
./testutil/testutil.go: config.SerfLANConfig.MemberlistConfig.BindPort = ports[1]I can't see how this can possibly work outside the test suite.
What I can do is force --serf-addr=127.0.0.1 at which point at least we don't have duplicate ports in SerfLANAddr:
# ./jocko broker --serf-addr=127.0.0.1
2018/06/16 21:31:42 Initializing logging reporter
2018-06-16T21:31:42.136Z INFO jocko/broker.go:109 hello {"id": 0, "broker addr": "0.0.0.0:9092", "serf addr": "127.0.0.1", "raft addr": "127.0.0.1:9093", "id": 0, "raft addr": "127.0.0.1:9093"}
2018/06/16 21:31:42 [INFO] raft: Initial configuration (index=0): []
2018/06/16 21:31:42 [INFO] raft: Node at 127.0.0.1:9093 [Follower] entering Follower state (Leader: "")
2018/06/16 21:31:42 [INFO] serf: EventMemberJoin: jocko 127.0.0.1
2018-06-16T21:31:42.170Z INFO jocko/server.go:71 hello {"id": 0, "broker addr": "0.0.0.0:9092", "serf addr": "127.0.0.1", "raft addr": "127.0.0.1:9093", "server id": 0, "addr": "0.0.0.0:9092"}
2018-06-16T21:31:42.175Z INFO jocko/serf.go:74 adding LAN server {"id": 0, "broker addr": "0.0.0.0:9092", "serf addr": "127.0.0.1", "raft addr": "127.0.0.1:9093", "id": 0, "raft addr": "127.0.0.1:9093", "meta": {"ID":0,"Name":"","Bootstrap":false,"Expect":0,"NonVoter":false,"Status":1,"RaftAddr":"127.0.0.1:9093","SerfLANAddr":"127.0.0.1:8301","BrokerAddr":"0.0.0.0:9092"}}
2018/06/16 21:31:42 [WARN] serf: Failed to re-join any previously known node
2018/06/16 21:31:43 [WARN] raft: no known peers, aborting electionHowever it still fails in the same same way as single node cluster (client says error code: not controller; broker fails writing to UDP port 6831)
justone
travisjeffery
The cluster example uses command line flags which are no longer valid:
--debug--log-dir(should be--data-dir?)--prometheus-addr--serf-members(should be--joinor--join-wan?)So I tried running it like this:
These options are accepted; but no broker is listening on ports 9001, 9101 or 9201, nor is serf listening on 9003, 9103 or 9203.
Captured output:
I tried running the first process under strace. Here are all the lines matching
htons:(I don't see any attempt to open ports 9001 or 9003?)
Here are the lines matching
= -1:The EPERM issues are a bit worrying. Maybe this is a symptom of running within an lxd container (but then again, running in a docker container is supposed to work)