Add a cli switch/configuration param to "safe start a slave"

[GoogleCodeExporter]

Hi all!

Our current setup is:

- we use redis to store information of websites
- apache queries redis to know where to fetch the file for the url requested
- there's a local redis instance on every webserver machine, slave of a master 
on another machine

When you reboot the webserver machine, there's a chance this chain of events 
could happen (even if we set apache init script as dependent on redis init 
script completion):

1. machine is restarted and the init process is starting to execute init scripts
2. redis is started first (since apache init depends on it)
3. redis is syncing from master (or reading from AOF and then sync with master) 
but has already returned the prompt (fetching data in the backgroud) so other 
init scripts can start
4. apache is started and start receiving requests
5. some requests fail (f.e. 404), specifically those of the part of the data 
not fetched yet.

we'd like to avoid this situation :)

What we have in mind is something like a '--slave-safe-start' cli switch or 
'slave-safe-start 1' in redis.conf, to make redis-server not return the prompt 
until it fetches all the data. We can even think of an optional parameters, 
'timeout', that returns the prompt after x seconds (in case of problems, f.e.)

Alternatively, we can add a busy loop in the redis init scrip that runs 
'redis-cli info' and checking that both 'loading' and 'master_sync_in_progress' 
are 0 before letting the script complete. But we consider it a workaround, and 
not a proper solution.

What do you think?

Cheers,
Sandro

Original issue reported on code.google.com by matrixhasu on 6 Jul 2011 at 8:35

[GoogleCodeExporter]

The workaround we're going to apply is :

>>>
timeout 3m tail -f /var/log/redis/redis-server.log | grep -m1 -q 'MASTER <-> 
SLAVE sync: Finished with success' &
echo -n "started, waiting for master-slave sync... "
if wait $!
then
       echo "done"
else
       echo "failed"
fi
<<<

that

- tail (in background) the log files (at max for 3 minutes) checking it the 
sync has finished succesfully
- it waits for the background process to complete

It's ugly as hell, it's a bashism ($! is bash-specific) and it's based on the 
fact the sync takes time so from when we start teh redis-server and we start 
tailing the log files the sync hasn't completed yet.

Of course, a proper support on redis-server side is still our preferred 
solution.

Cheers,
Sandro

Original comment by matrixhasu on 13 Jul 2011 at 9:32