Implement systematic configuration for heterogeneous replication

[GoogleCodeExporter]

1. To which tool/application/daemon will this feature apply?

Tungsten Replicator

2. Describe the feature in general

Replication from MySQL and Oracle to other DBMS types is complex to configure 
due to the number of filters that must be enabled.  This feature will reduce 
the configuration to a couple of simple tpm options. 

3. Describe the feature interface

Tpm will support the following installation pattern to enable a master 
replicator or cluster of replicators to serve as a source for heterogeneous 
replication.

[east]
master=db1
slaves=db2,db3
topology=clustered
enable-heterogeneous-master=true
...  

No other parameters are necessary besides enable-heterogeneous-master. 

Tpm will also support the following installation pattern to enable a 
heterogeneous slave. 

[east_slaves]
topology=cluster-slave
master=rep1
master-dataservice=east
enable-heterogeneous-slave=true 
... 

4. Give an idea (if applicable) of a possible implementation

The heterogeneous master will automatically install the pkey, colnames, 
settostring, and enumtostring to generate column and pkey metdata and convert 
data.  Strings will still be replicated using bytes, which ensures homogeneous 
replication works fully. 

The heterogeneous slave will automatically install the fixmysqlstrings filter 
to convert MySQL strings from bytes to UTF8.  This filter also correctly types 
binary values. 

5. Describe pros and cons of this feature.

5a. Why the world will be a better place with this feature.

Simplifies installation of heterogeneous data loading. 

5b. What hardship will the human race have to endure if this feature is
implemented.

Needs very careful testing to ensure the filters do not affect MySQL 
homogeneous replication and that fixmysqlstrings correctly translate bytes to 
utf8. 

6. Notes

Original issue reported on code.google.com by robert.h...@continuent.com on 31 Dec 2014 at 2:04

[GoogleCodeExporter]

This issue was updated by revision r2734.

Added support for tpm --enable-heterogeneous-master and 
--enable-heterogeneous-slave options. This update installs filters correctly 
for heterogeneous replication but does not interfere with with homogeneous 
replication.  The THL protobuf definition has been extended to store the type 
description name, so that colnames does not have to run in downstream 
replicators. 

Original comment by robert.h...@continuent.com on 31 Dec 2014 at 2:30

[GoogleCodeExporter]

Robert, you say "added", but these options already existed. In fact, we have 
been promoting them in documentation for a while. Hence, it is a behavioural 
change and we need to make sure that old installations, which are already using 
these options, work the same after upgrade.

Original comment by linas.vi...@continuent.com on 31 Dec 2014 at 9:18

[GoogleCodeExporter]

This issue was updated by revision r2735.

Update the tungsten_provision_thl script so it may run from a cluster slave for 
provisioning heterogeneous slaves. CONT-51

Original comment by jeffm...@gmail.com on 31 Dec 2014 at 8:32

[GoogleCodeExporter]

This issue was updated by revision r2736.

Update CONT-51
- Enable the fixmysqlstrings filter for all non-mysql slaves
- Disable the bytes for strings feature only when --enable-heterogeneous-master 
or --enable-heterogeneous-service is set for non-clustered servers

Original comment by jeffm...@gmail.com on 31 Dec 2014 at 8:39

[GoogleCodeExporter]

This issue was updated by revision r2737.

Corrected error in protobuf serialization that inadvertently converted 
byte-encoded statements to strings. 

Original comment by robert.h...@continuent.com on 1 Jan 2015 at 3:38

[GoogleCodeExporter]

This issue was updated by revision r2738.

Do not try to bind key values, which have been added to make heterogeneous 
cluster slave to work, for INSERTs.

Original comment by linas.vi...@continuent.com on 2 Jan 2015 at 4:37

[GoogleCodeExporter]

This issue was updated by revision r2739.

Implemented optimizations to ensure extractors consistently provide metadata on 
DBMS type as well as whether strings are converted to UTF8. Added MySQL filter 
optimizations to avoid unnecessary iterations when DBMS type is different or 
when strings are already converted.  Added helper methods on ReplDBMSEvent and 
OneRowChange to make filter programming easier (e.g., to set/get event 
options). 

Original comment by robert.h...@continuent.com on 6 Jan 2015 at 5:11

[GoogleCodeExporter]

This issue was updated by revision r2740.

Added a missed bit of code to set strings and dbms_type options on single 
events from ParallelExtractor. 

Original comment by robert.h...@continuent.com on 6 Jan 2015 at 6:29

[GoogleCodeExporter]

This issue was updated by revision r2751.

Added metadata and fixes to fixmysqlstrings.js to permit correct translation of 
strings when MySQL data set is not UTF8.  Added unit tests to check THL file 
upwards compatibility between releases including THL data files to make tests 
work for two recent versions. Fixed calls in pipeline tests that cause failures 
in builds due to race conditions. Tweaked build.xml to copy test data files 
into builds directory in a more systematic way. 

Original comment by robert.h...@continuent.com on 10 Jan 2015 at 5:33

[GoogleCodeExporter]

This issue was updated by revision r2754.

Corrected a failure that occurs when values like enum or set are converted to 
string before reaching fixmysqlstrings filter.  The filter now detects strings 
that have already been altered and ignores them. 

Original comment by robert.h...@continuent.com on 13 Jan 2015 at 6:15

[GoogleCodeExporter]

Original comment by linas.vi...@continuent.com on 19 Jan 2015 at 2:20

• Added labels: FixedIn-4.0.1
• Removed labels: FixedIn-3.1.0