search

mdevilliers / redishappy

Redis Sentinel high availabillity daemon
Apache License 2.0
114 stars 23 forks source link

redishappy-consul can't get SRV record from consul #63

Open webervin opened 7 years ago

webervin commented 7 years ago

Hello, have 3 separate VM's. each has redis server and sentinel. on one machine redishappy-consul is also installed (found no tutorial how to run it on multiple machines at same time). I seem to get no errors, but also only mention of 'redis-cluster' in consul seem to be deregistering of service.

dig output: 

$ dig @127.0.0.1 -p 8600 redis-cluster.service.consul SRV

; <<>> DiG 9.9.5-3ubuntu0.13-Ubuntu <<>> @127.0.0.1 -p 8600 redis-cluster.service.consul SRV
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 20669
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;redis-cluster.service.consul.  IN  SRV

;; AUTHORITY SECTION:
consul.         0   IN  SOA ns.consul. postmaster.consul. 1489610327 3600 600 86400 0

;; Query time: 2 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Wed Mar 15 20:38:47 UTC 2017
;; MSG SIZE  rcvd: 96

sentinel config: 

sentinel monitor redis-cluster 10.128.0.2 18142 2
sentinel failover-timeout redis-cluster 60000
sentinel auth-pass redis-cluster secretpass
sentinel config-epoch redis-cluster 6
sentinel leader-epoch redis-cluster 6
# Generated by CONFIG REWRITE
port 26379
dir "/"
sentinel known-slave redis-cluster 10.128.0.8 18142
sentinel known-slave redis-cluster 10.128.0.9 18142
sentinel known-sentinel redis-cluster 10.128.0.9 26379 3e492f6e8a7763aeb03dbda90e36b67da6a5315b
sentinel known-sentinel redis-cluster 10.128.0.2 26379 7280519877b15ad97c098718752c57cf4e4d243d

sentinel current-epoch 6

  /etc/redishappy-consul/config.json: 

{
  "SentinelTCPKeepAlive": 0,
  "Clusters" :[
  {
    "Name" : "redis-cluster",
    "ExternalPort" : 18142
  }],
  "Sentinels" : [
      {"Host" : "127.0.0.1", "Port" : 26379}
    ],
  "Consul" : {
    "Address" : "127.0.0.1:8500",
    "Services" : [
          {
        "Cluster" : "redis-cluster",
         # what node should be here? let's put current master name as visible to consul
        "Node" : "staging-redis1",
         # copy paste datacenter name from consul config
        "Datacenter": "gce-staging",
        "Tags": ["redis", "master"]
      }
    ]
  }
}

/var/log/redishappy-consul/redis-happy.log: 

INFO: 2017/03/15 19:46:18 manager.go:112: Resync Topology: types.MasterDetails{ExternalPort:18142, Name:"redis-cluster", Ip:"10.128.0.2", Port:18142}
TRACE: 2017/03/15 19:46:18 throttle.go:39: Deduped +switch-master event, as I've (probably) already done the same thing. redis-cluster : 10.128.0.2 : 18142
TRACE: 2017/03/15 19:46:19 sentinels.go:108: Sentinel ping : {"host":"10.128.0.2","port":26379}
TRACE: 2017/03/15 19:46:19 sentinels.go:109: Sentinel state : {"sentinels":{"10.128.0.2:26379":{"sentinelLocation":"10.128.0.2:26379","lastUpdated":"2017-03-15T19:46:18.176549989Z","clusters":null,"state":3},"10.128.0.9:26379":{"sentinelLocation":"10.128.0.9:26379","lastUpdated":"2017-03-15T19:46:18.176551009Z","clusters":null,"state":4},"127.0.0.1:26379":{"sentinelLocation":"127.0.0.1:26379","lastUpdated":"2017-03-15T19:46:19.1768335Z","clusters":["redis-cluster"],"state":3}}}.
TRACE: 2017/03/15 19:46:19 sentinels.go:74: Sentinel added : {"host":"10.128.0.8","port":26379}
TRACE: 2017/03/15 19:46:19 sentinels.go:108: Sentinel ping : {"host":"10.128.0.9","port":26379}
TRACE: 2017/03/15 19:46:19 sentinels.go:109: Sentinel state : {"sentinels":{"10.128.0.2:26379":{"sentinelLocation":"10.128.0.2:26379","lastUpdated":"2017-03-15T19:46:19.177752606Z","clusters":["redis-cluster"],"state":3},"10.128.0.8:26379":{"sentinelLocation":"10.128.0.8:26379","lastUpdated":"2017-03-15T19:46:19.178304986Z","clusters":null,"state":4},"10.128.0.9:26379":{"sentinelLocation":"10.128.0.9:26379","lastUpdated":"2017-03-15T19:46:18.176551009Z","clusters":null,"state":3},"127.0.0.1:26379":{"sentinelLocation":"127.0.0.1:26379","lastUpdated":"2017-03-15T19:46:19.1768335Z","clusters":["redis-cluster"],"state":3}}}.
INFO: 2017/03/15 19:46:19 monitor.go:129: Subscription Message : Channel : +switch-master : 1
TRACE: 2017/03/15 19:46:19 monitor.go:147: Subscription Message : Firing a ConnectionEvent
TRACE: 2017/03/15 19:46:19 manager.go:108: types.MasterDetailsCollection{items:map[string]*types.MasterDetails{"redis-cluster":(*types.MasterDetails)(0xc820124330)}}
INFO: 2017/03/15 19:46:19 manager.go:112: Resync Topology: types.MasterDetails{ExternalPort:18142, Name:"redis-cluster", Ip:"10.128.0.2", Port:18142}
TRACE: 2017/03/15 19:46:19 throttle.go:39: Deduped +switch-master event, as I've (probably) already done the same thing. redis-cluster : 10.128.0.2 : 18142
TRACE: 2017/03/15 19:46:20 sentinels.go:108: Sentinel ping : {"host":"10.128.0.8","port":26379}
TRACE: 2017/03/15 19:46:20 sentinels.go:109: Sentinel state : {"sentinels":{"10.128.0.2:26379":{"sentinelLocation":"10.128.0.2:26379","lastUpdated":"2017-03-15T19:46:20.178629935Z","clusters":["redis-cluster"],"state":3},"10.128.0.8:26379":{"sentinelLocation":"10.128.0.8:26379","lastUpdated":"2017-03-15T19:46:19.178304986Z","clusters":null,"state":3},"10.128.0.9:26379":{"sentinelLocation":"10.128.0.9:26379","lastUpdated":"2017-03-15T19:46:19.178458595Z","clusters":["redis-cluster"],"state":3},"127.0.0.1:26379":{"sentinelLocation":"127.0.0.1:26379","lastUpdated":"2017-03-15T19:46:20.177467597Z","clusters":["redis-cluster"],"state":3}}}.

some http checks: 

$ curl http://127.0.0.1:8000/api/configuration
{"SentinelTCPKeepAlive":0,"clusters":[{"name":"redis-cluster","externalPort":18142}],"sentinels":[{"host":"127.0.0.1","port":26379}],"consul":{"address":"127.0.0.1:8500","services":[{"cluster":"redis-cluster","node":"staging-redis2","tags":["redis","master"],"datacenter":"gce-staging"}]},"HAProxy":{}}

$ curl http://127.0.0.1:8000/api/topology
[{"externalPort":18142,"name":"redis-cluster","ip":"10.128.0.2","port":18142}]
$ curl  --silent http://127.0.0.1:8500/v1/catalog/services 2>/dev/null| grep redis-cluster | wc -l
0

in consul log only mentions for 'redis-cluster' are deregistering of service: 

ubuntu@staging-redis2:~$ sudo  grep redis-cluster /var/log/upstart/consul.log
    2017/03/15 19:36:01 [INFO] agent: Deregistered service 'redishappy-consul-redis-cluster'
    2017/03/15 19:37:09 [INFO] agent: Deregistered service 'redishappy-consul-redis-cluster'
    2017/03/15 19:41:39 [INFO] agent: Deregistered service 'redishappy-consul-redis-cluster'
    2017/03/15 19:44:33 [INFO] agent: Deregistered service 'redishappy-consul-redis-cluster'
    2017/03/15 19:47:05 [INFO] agent: Deregistered service 'redishappy-consul-redis-cluster'