airbnb / synapse

A transparent service discovery framework for connecting an SOA
MIT License
2.07k stars 251 forks source link

Thread Safety #321

Closed panchr closed 4 years ago

panchr commented 4 years ago

Summary

The new ZookeeperPoll watcher runs in a background thread and needs to communicate with the main thread (in order to tell it when to update). Currently, it just calls a callback which will execute in the background thread, and that has some thread safety concerns.

Also, the main thread will read the state of each watcher. The fixes for this are:

  1. Hold a Synapse-level lock when reconfiguring.
  2. Hold a watcher-level lock when getting/setting any watcher state.

In addition, it runs the primary Zookeeper watcher in a background thread. This allows retry logic to not block the main thread.

Tests

$ zookeepercli -servers localhost:2181 -c create /services/service1/0002.mango-test '{"host":"127.0.0.3","port":1003,"name":"i-primary3","labels":{"region":"us-east-1","az":"us-east-1c"}}'

$ grep server haproxy.cfg timeout server 1m server i-primary3_127.0.0.3:1003 127.0.0.3:1003 id 1 cookie i-primary3_127.0.0.3:1003 check inter 2s rise 3 fall 2 server i-primary_127.0.0.1:1000 127.0.0.1:1000 id 2 cookie i-primary_127.0.0.1:1000 check inter 2s rise 3 fall 2 server default1_localhost:8422 localhost:8422 id 1 cookie default1_localhost:8422 check inter 2s rise 3 fall 2


- additional testing will be performed as part of mega-PR #311 

## Reviewers
@austin-zhu