Eventual replication, including partially async initial confirmations

[vaizki]

NOTE: I am fully aware that the Disque project may be merged into the redis repo soon as a module of redis 4.2 but I wanted to contribute this idea into disque now so that this requirement (which is very real for me - your mileage may vary) might make it into the new & shiny redis module.

First attempt at eventual replication, combining synchronous and asychronous replication into one ADDJOB operation.

Developed (hastily) to support a scenario with a 2-node system which is geographically distributed and must support at-least-once semantics but also operate each site in 'island mode' where only 1 node may be available for synchronous replication but when inter-node connectivity is restored the jobs should be replicated to the other node. In this scenario jobs are added with REPLICATE 2 SYNC 1 to immediately unblock if the local node is alive but eventually replicate to the other node if it becomes available.

Adds a SYNC parameter to ADDJOB, which is mutually exclusive with ASYNC.

A job with SYNC specified will perform synchronous (block client) replication to N nodes. REPLICATE will still control the amount of nodes that are desired for replication.

When ADDJOB with SYNC is received, REPLJOB messages will be sent to the amount of nodes controlled by REPLICATE, but the operation will unblock only when N nodes have confirmed that they have replicated. The rest of the replication is async, replies will be sent to the origin and they will be added to the confirmed_hosts until full replication has been achieved (sync + async). The confirmed_hosts dict will be released when full confirmation has been achieved, which stops any future broadcasts.

This additional replication currently happens in the standard awake timer so the RETRY parameter in effect controls how often the original node attempts to gain new replications for this job. This may (more likely will) change in the future.

NOTE: This patch completely respects that Disque jobs are immutable, thus when the job unblocks it has been written to the AOF and will not gain knowledge of additional confirmed nodes later. Only in-memory state will change after that. The on-the-wire replication protocol and AOF format & logic are unchanged, so a rolling upgrade to this version should be fine.

Because the confirmed_nodes dict is not serialized to the AOF, any node restarting will have a NULL dict in which case they will not rebroadcast the job to additional nodes. Thus if the original node accepting the job is restarted the message may never reach full replication.

Some test cases have been added to cover the basics of the new functionality.

P.S. I made my first ever PR earlier today and hopefully this one will fare better..

[vaizki]

The check that has failed here.. has never failed in my local tests. It's probably because the node is reported as unreachable before the ADDJOB reaches it.. AFAIK has nothing to do with the changes I made but rather with the lagginess of the CI but I'm happy to be proven wrong.

[vaizki]

Also please see my comments in #189 to understand the motivation and use case for this.

[vaizki]

Closing this as we have moved on from disque.. and so has Redis and antirez. Thanks for all the fish!