search

zxs / tungsten-replicator

Automatically exported from code.google.com/p/tungsten-replicator
0 stars 0 forks source link

Develop tools to detect and repair differences between contents of DBMS servers #618

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
1. To which tool/application/daemon will this feature apply?

Tungsten Replicator

2. Describe the feature in general

In any replicated system it is important to be able to do two basic things: 

a.) At any time and with few resources as possible rapidly compare the contents 
of two DBMS servers to see if they are equivalent and if not identify the 
differences as accurately as possible.  

b.) Provide a simple and effective way of repairing differences between 
servers, for example by choosing particular rows, tables, databases, or servers 
as the master version and propagating them automatically to other locations.  
This procedure should handle inserted, updated, and deleted data.  It should 
also work as far as possible on live systems with running applications. 

3. Describe the feature interface

The feature interface has yet to be determined but would presumably start with 
a command line interface.  It should have an API as well that permits it to be 
invoked remotely as well as embedded in graphical tools. 

4. Give an idea (if applicable) of a possible implementation

TBD

5. Describe pros and cons of this feature.

5a. Why the world will be a better place with this feature.

Replicated systems can become inconsistent for a variety of reasons.  Having 
ways of detecting and curing such inconsistencies enables effective testing in 
development and support of operational systems. 

5b. What hardship will the human race have to endure if this feature is
implemented.

Data comparison and repair is a hard problem that must work effectively on live 
systems with large quantities of data.  There are many bad solutions, so we 
need to think carefully and proceed incrementally. 

6. Notes

Some additional requirements: 

1.) Work effectively on very large quantities of data, e.g., 500M records. 
2.) Avoid designs that assume SQL or relational stores. 
3.) Work safely on production systems. 
4.) Take advantage where appropriate of Tungsten replicator to aid in data 
repair.

Original issue reported on code.google.com by robert.h...@continuent.com on 17 Jun 2013 at 8:06

GoogleCodeExporter commented 9 years ago 
Set Linas as owner, as he is current project manager for replicator.  This will 
be a team effort to fix.

Original comment by robert.h...@continuent.com on 17 Jun 2013 at 8:06

GoogleCodeExporter commented 9 years ago 
Some reading related to this problem
http://www.drdobbs.com/taming-the-distributed-database-problem/199101871

Original comment by g.maxia on 17 Jun 2013 at 8:14

GoogleCodeExporter commented 9 years ago

Original comment by linas.vi...@continuent.com on 18 Jun 2013 at 1:04

GoogleCodeExporter commented 9 years ago

Original comment by linas.vi...@continuent.com on 19 Jun 2013 at 4:10

GoogleCodeExporter commented 9 years ago

Original comment by linas.vi...@continuent.com on 11 Jul 2013 at 1:55

GoogleCodeExporter commented 9 years ago 
We'll need THL search option (Issue 644) to align missing/corrupt data with THL.

Original comment by linas.vi...@continuent.com on 18 Jul 2013 at 8:17

GoogleCodeExporter commented 9 years ago

Original comment by linas.vi...@continuent.com on 20 Aug 2013 at 9:35

GoogleCodeExporter commented 9 years ago

Original comment by linas.vi...@continuent.com on 26 Aug 2013 at 1:54

GoogleCodeExporter commented 9 years ago 
There won't be a 2.1.3.

Original comment by linas.vi...@continuent.com on 17 Sep 2013 at 10:13

GoogleCodeExporter commented 9 years ago

Original comment by linas.vi...@continuent.com on 18 Dec 2013 at 1:38

GoogleCodeExporter commented 9 years ago

Original comment by linas.vi...@continuent.com on 6 Feb 2014 at 3:43

GoogleCodeExporter commented 9 years ago 
These are not at the focus of 3.0.0.

Original comment by linas.vi...@continuent.com on 26 May 2014 at 2:21

GoogleCodeExporter commented 9 years ago 
Will not use third version digit for normal releases anymore. It will only be 
increment for maintenance ones.

Original comment by linas.vi...@continuent.com on 26 May 2014 at 5:01

GoogleCodeExporter commented 9 years ago

Original comment by linas.vi...@continuent.com on 19 Jan 2015 at 2:20