The new ZookeeperPoll watcher runs in a background thread and needs to communicate with the main thread (in order to tell it when to update). Currently, it just calls a callback which will execute in the background thread, and that has some thread safety concerns.
Also, the main thread will read the state of each watcher. The fixes for this are:
Hold a Synapse-level lock when reconfiguring.
Hold a watcher-level lock when getting/setting any watcher state.
In addition, it runs the primary Zookeeper watcher in a background thread. This allows retry logic to not block the main thread.
Tests
[x] unit tests
[x] basic testing: running Synapse with multiple watchers
[x] basic Zookeeper watcher still works
$ grep server haproxy.cfg
timeout server 1m
server i-primary_127.0.0.1:1000 127.0.0.1:1000 id 2 cookie i-primary_127.0.0.1:1000 check inter 2s rise 3 fall 2
server default1_localhost:8422 localhost:8422 id 1 cookie default1_localhost:8422 check inter 2s rise 3 fall 2
$ grep server haproxy.cfg timeout server 1m
server i-primary3_127.0.0.3:1003 127.0.0.3:1003 id 1 cookie i-primary3_127.0.0.3:1003 check inter 2s rise 3 fall 2
server i-primary_127.0.0.1:1000 127.0.0.1:1000 id 2 cookie i-primary_127.0.0.1:1000 check inter 2s rise 3 fall 2
server default1_localhost:8422 localhost:8422 id 1 cookie default1_localhost:8422 check inter 2s rise 3 fall 2
- additional testing will be performed as part of mega-PR #311
## Reviewers
@austin-zhu
Summary
The new ZookeeperPoll watcher runs in a background thread and needs to communicate with the main thread (in order to tell it when to update). Currently, it just calls a callback which will execute in the background thread, and that has some thread safety concerns.
Also, the main thread will read the state of each watcher. The fixes for this are:
In addition, it runs the primary Zookeeper watcher in a background thread. This allows retry logic to not block the main thread.
Tests
$ zookeepercli -servers localhost:2181 -c create /services/service1/0002.mango-test '{"host":"127.0.0.3","port":1003,"name":"i-primary3","labels":{"region":"us-east-1","az":"us-east-1c"}}'
$ grep server haproxy.cfg timeout server 1m server i-primary3_127.0.0.3:1003 127.0.0.3:1003 id 1 cookie i-primary3_127.0.0.3:1003 check inter 2s rise 3 fall 2 server i-primary_127.0.0.1:1000 127.0.0.1:1000 id 2 cookie i-primary_127.0.0.1:1000 check inter 2s rise 3 fall 2 server default1_localhost:8422 localhost:8422 id 1 cookie default1_localhost:8422 check inter 2s rise 3 fall 2