Allow resolver to send notifications to MultiWatcher

[panchr]

Summary

A resolver can independently trigger a change of the backends. For example, the new S3ToggleResolver can result in a new set of backends if the S3 file changes. That is, say we have two watchers with the following backends:

• A --> [host_1, host_2]
• B --> [host_3, host_4]

Say in S3, the toggle file originally says to only take backends from A. Then, that file changes to only take backends from B. S3ToggleResolver will notice this change, and change its reported backends to [host_3, host_4].

Before this PR, those changes were not reflected because the underlying watchers themselves had not changed their backends. That is, neither watcher A nor watcher B's backends changed. Instead, what changed is how we resolve the backends between the two.

In order for that change to be reflected in Synapse (and the generated configuration for HAProxy), the resolver needs to be able to tell the main loop that something has changed.

Todo

• [x] new unit tests
• [x] manual testing

Tests

• [x] unit tests
• [x] changing S3 file reflects in haproxy.cfg file change In this test, the primary watcher reads a ZK path that has one instance: i-primary. The secondary watcher reads a ZK path that has two instances: i-secondary and i-secondary2.

1. Haproxy config before changing S3 file (reading primary):

   $ grep "server i-" haproxy.cfg
       server i-primary_127.0.0.1:1000 127.0.0.1:1000 id 1 cookie i-primary_127.0.0.1:1000 check inter 2s rise 3 fall 2

2. Change S3 file to read entirely from secondary:

   
   $ echo "---
   # primary cluster
   primary: 0

secondary cluster

secondary: 100\n" > s3-toggle-resolver.yaml

$ AWS_ACCESS_KEY_ID=minioadmin AWS_SECRET_ACCESS_KEY=minioadmin aws --endpoint-url http://localhost:9000 s3 cp s3-toggle-resolver.yaml s3://synapse-test/s3-toggle-resolver.yaml

3. Synapse reads the new file and reconfigures HAProxy:

I, [2020-03-12T15:21:19.930178 #98585] INFO -- Synapse::ServiceWatcher::Resolver::S3ToggleResolver: synapse: s3 toggle resolver: read s3 file: {"primary"=>0, "secondary"=>100} I, [2020-03-12T15:21:19.930239 #98585] INFO -- Synapse::ServiceWatcher::Resolver::S3ToggleResolver: synapse: s3 toggle resolver: chose watcher secondary I, [2020-03-12T15:21:19.930436 #98585] INFO -- Synapse::ServiceWatcher::MultiWatcher: synapse: discovered 2 backends for service service1 I, [2020-03-12T15:21:19.930464 #98585] INFO -- Synapse::ServiceWatcher::MultiWatcher: synapse: no config_for_generator data from service1 for service service1; keep existing config_for_generator I, [2020-03-12T15:21:20.770567 #98585] INFO -- Synapse::Synapse: synapse: configuring haproxy I, [2020-03-12T15:21:20.771367 #98585] INFO -- Synapse::ConfigGenerator::Haproxy: synapse: restart required because config_for_generator changed. before: {}, after: {"server_options"=>"check inter 2s rise 3 fall 2", "server_port_override"=>nil, "backend"=>[], "frontend"=>[], "listen"=>["mode http", "option httpchk /health", "http-check expect string OK"], "port"=>3213, "bind_options"=>"ssl no-sslv3 crt /path/to/cert/example.pem ciphers ECDHE-ECDSA-CHACHA20-POLY1305"}

4. HAProxy reflects new backends:
```bash
$ grep "server i-" haproxy.cfg
        server i-secondary_127.0.0.2:2000 127.0.0.2:2000 id 3 cookie i-secondary_127.0.0.2:2000 check inter 2s rise 3 fall 2
        server i-secondary2_127.0.0.3:2001 127.0.0.3:2001 id 2 cookie i-secondary2_127.0.0.3:2001 check inter 2s rise 3 fall 2

Reviewers

@austin-zhu @bsherrod