GoogleCodeExporter
commented
9 years ago This feature is now implemented. Here is a description including notes on
testing.
1. Filter implementations. This feature adds two filters as described in the
feature description.
schemachange - A Java filter to detect schema changes by parsing statements and
create permanent annotations on the statements. This filter also detects
truncate statements. Here is the general behavior.
a.) Any statement that affects the logical schema of the table, such as
CREATE/DROP SCHEMA, CREATE/DROP TABLE, or ALTER causes the annotation
##schema_change to be added to the event metadata.
b.) Similarly any statement that is a table TRUNCATE operate causes the
annotation ##truncate to be added to the event metadata.
c.) In both cases the operation type, schema name, and table name (if
appropriate) are added to the statement metadata.
monitorschemachange - A Javascript filter to log changes detected by the
schemachange filter and optionally force a commit when such a change is
processed.
2. Installation. Filters can be installed anywhere that is convenient with the
caveat that schemachange must run before monitorschemachange for the latter to
work. Here is a sample installation.
a.) Install schemachange on master as in the following example. This way the
schema annotations are visible in the log.
tools/tpm install batch1 \
...
--svc-extractor-filters=colnames,pkey,schemachange \
...
--start-and-report
b.) Install monitorschemachange on the slave applier stage. This logs the
changes on the slave and causes a commit only in the final stage.
tools/tpm install batch1 \
...
--svc-applier-filters=monitorschemachange \
...
--start-and-report
3. To test schemachange filter operation, enter the following types of
statements:
CREATE SCHEMA bar
DROP SCHEMA bar
CREATE TABLE foo
DROP TABLE foo
ALTER TABLE foo ADD COLUMN data1 varchar(30)
ALTER TABLE foo DROP COLUMN data1
All of these should generate annotations on transactions stored in the THL like
the following example for ALTER TABLE. Note the 'schema_change' annotation on
the transaction and the ##operation, ##schema, and ##table on the statement.
SEQ# = 2 / FRAG# = 0 (last frag)
- TIME = 2014-05-05 21:57:24.0
- EPOCH# = 0
- EVENTID = mysql-bin.000287:0000000000000639;70
- SOURCEID = logos1
- METADATA =
[mysql_server_id=1;unsafe_for_block_commit;dbms_type=mysql;service=batch;shard=t
est;schema_change]
- TYPE = com.continuent.tungsten.replicator.event.ReplDBMSEvent
- OPTIONS = [##charset = UTF-8, autocommit = 1, sql_auto_is_null = 0,
foreign_key_checks = 1, unique_checks = 1, sql_mode = 'NO_ENGINE_SUBSTITUTION',
character_set_client = 33, collation_connection = 33, collation_server = 33,
##schema = test, ##table = foo, ##operation = ALTER TABLE]
- SCHEMA =
- SQL(0) = alter table test.foo add column data1 varchar(30)
You can also enter a truncate table command, which should generate similar
annotations:
SEQ# = 3 / FRAG# = 0 (last frag)
- TIME = 2014-05-05 21:59:35.0
- EPOCH# = 0
- EVENTID = mysql-bin.000287:0000000000000719;71
- SOURCEID = logos1
- METADATA =
[mysql_server_id=1;dbms_type=mysql;service=batch;shard=test;truncate]
- TYPE = com.continuent.tungsten.replicator.event.ReplDBMSEvent
- OPTIONS = [##charset = UTF-8, autocommit = 1, sql_auto_is_null = 0,
foreign_key_checks = 1, unique_checks = 1, sql_mode = 'NO_ENGINE_SUBSTITUTION',
character_set_client = 33, collation_connection = 33, collation_server = 33,
##schema = test, ##table = foo, ##operation = TRUNCATE]
- SCHEMA =
- SQL(0) = truncate test.foo
4.) To test monitorschemachange operation, look for the following:
a.) The slave replicator log should have messages like the following for schema
change and truncate commands:
INFO | jvm 1 | 2014/05/05 22:01:06 | 2014-05-06 05:01:06,422 [batch -
q-to-dbms-0] INFO filter.JavaScriptFilter SCHEMA CHANGE: seqno=2 schema=test
table=foo operation=ALTER TABLE sql=[alter table test.foo add column data1
varchar(30)]
The default behavior of the monitorschemachange filter is to force a schema
change, which results in the following message:
INFO | jvm 1 | 2014/05/05 22:09:30 | 2014-05-06 05:09:30,551 [batch -
q-to-dbms-0] INFO filter.JavaScriptFilter Forcing commit due to schema change:
seqno=2
Original comment by robert.h...@continuent.com on 6 May 2014 at 5:13
Original issue reported on code.google.com by
robert.h...@continuent.comon 14 Apr 2014 at 5:36