search

cloud-gov / kubernetes-broker

Broker for kubernetes based services
Apache License 2.0
5 stars 6 forks source link

Add preStop hook, update plans, fix custom images #57

Closed rogeruiz closed 7 years ago

rogeruiz commented 7 years ago

This triggers a manual failover whenever a Master Redis server shutsdown.

pairing with @linuxbozo, @jmcarp

rogeruiz commented 7 years ago

So we've got another race condition here. I managed to get 4 iterations of tests to pass with Redis + Sentinel, but I'm still seeing some issues when the Replica that's been elected a Primary hasn't synchronized and can't seem to get their config rewritten because of a permission denied error.

1:S 11 Aug 04:48:36.239 * MASTER <-> SLAVE sync started
1:M 11 Aug 04:48:40.736 * Discarding previously cached master state.
1:M 11 Aug 04:48:40.736 * MASTER MODE enabled (user request from 'id=4 addr=local-ip:56530 fd=5 name=sentinel-b5bbc6db-cmd age=101 idle=0 flags=x db=0 sub=0 psub=0 multi=3 qbuf=0 qbuf-free=32768 obl=36 oll=0 omem=0 events=r cmd=exec')
1:M 11 Aug 04:48:40.737 # CONFIG REWRITE failed: Permission denied
1:M 11 Aug 04:48:41.894 * Slave local-ip:6379 asks for synchronization
1:M 11 Aug 04:48:41.894 * Full resync requested by slave local-ip:6379

Looking at the pods for Redis Server it looks like like redis-server commands runs as redis instead of root.

rogeruiz commented 7 years ago

@cnelson awesome. should this be happening as part of the pre-stop for servers, or do you think it's fine for it to just happen on the start of sentinels? Because of the IP reuse, we were seeing weird slave properties because at one point they were servers, am I understanding the problem right?