compress traffic between replicators

[GoogleCodeExporter]

1. To which tool/application/daemon will this feature apply?

Tungsten Replicator

2. Describe the feature in general

When Tungsten sends transactions across the network, it should be able to 
compress large chunks of data, to reduce the network traffic.

3. Describe the feature interface

No interface. The feature should work transparently.

4. Give an idea (if applicable) of a possible implementation

We set a maximum size that should be sent uncompressed (say, 100kb) . When the 
slave requests an amount of data that is larger than the  maximum amount, the 
data is compressed and sent across.
The receiving server will detect that the data is compressed, and it will 
uncompress it before saving it to the thl file.

A simpler implementation would be to compress everything, no matter how small.

5. Describe pros and cons of this feature.

5a. Why the world will be a better place with this feature.

The network traffic will be greatly reduced. When the slave lag depends on the 
size of the transactions, this feature will make the lag disappear.

5b. What hardship will the human race have to endure if this feature is
implemented.

There are two drawbacks:
* We need to implement the feature carefully, in such a way that the compressed 
block of THL data is never mistaken for a plain block.
* When we start using this feature, we need to upgrade all replicators in the 
cluster, slaves first. In the case of multiple masters, all replicators must be 
upgraded at once.

6. Notes

Original issue reported on code.google.com by g.maxia on 5 Sep 2013 at 3:53

[GoogleCodeExporter]

This is partially related to:

Issue 459 - Add support for compressing THL files when not in use

Original comment by linas.vi...@continuent.com on 10 Sep 2013 at 9:52

[GoogleCodeExporter]

There won't be a 2.1.3.

Original comment by linas.vi...@continuent.com on 17 Sep 2013 at 10:13

• Added labels: fixedin-2.2.0
• Removed labels: FixedIn-2.1.3

[GoogleCodeExporter]

Might need for Hadoop.

Original comment by linas.vi...@continuent.com on 20 Nov 2013 at 3:51

• Removed labels: fixedin-2.2.0

[GoogleCodeExporter]

Please note that without compressing, using protobuf could help divide the 
network load by 3 (this is what I got by restoring a dump of 
https://launchpad.net/test-db through replication) : 512MB for current 
serialization vs 180 MB using protobuf.

Original comment by stephane...@continuent.com on 13 Dec 2013 at 1:54

[GoogleCodeExporter]

How much work is it to switch between JAVA serialization and Protobuf? Sounds 
like a major change.

Original comment by linas.vi...@continuent.com on 13 Dec 2013 at 1:57