epermana / tungsten-replicator

Automatically exported from code.google.com/p/tungsten-replicator
1 stars 0 forks source link

Unknown column '' in 'field list' and un-replicated ALTER/CREATEs #773

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago
What steps will reproduce the problem?

1. This seems to happen randomly over time, so it's hard to reproduce 
synthetically, but does happen frequently under our production load.
2. Usually happens after a schema has been altered or created.
3. This seems to happen either after multiple ALTERs on the same schema, or a 
CREATE then ALTER on the same schema. 
4 Only started happening to us after using any 2.1.x, 2.0.6 was fine.

What is the expected output?

Schema ALTERs and CREATEs should be replicated correctly and consistently.

What do you see instead?

So this is the interesting bit: 

In the THL we can see that there were CREATE statements for the tables, on both 
the master and the slave, so that part seems to be fine. We do not see any 
errors related to the CREATE on either side, however the slave does *not* 
create the table. We only discover its failed when suddenly we see:

In the case of a missing CREATE:
INFO   | jvm 1    | 2013/12/09 12:47:01 | 2013-12-09 12:47:01,946 [alpha - 
q-to-dbms-12] WARN  applier.JdbcApplier No column information found for table 
(perhap
s table is missing?): xxx_com.activitystreamtype
INFO   | jvm 1    | 2013/12/09 12:47:01 | 2013-12-09 12:47:01,946 [alpha - 
q-to-dbms-12] ERROR pipeline.SingleThreadStageTask Stage task failed: 
stage=q-to-dbms
 seqno=191364390 fragno=0
INFO   | jvm 1    | 2013/12/09 12:47:01 | java.lang.NullPointerException
INFO   | jvm 1    | 2013/12/09 12:47:01 |       at 
com.continuent.tungsten.replicator.applier.MySQLDrizzleApplier.setObject(MySQLDr
izzleApplier.java:327)
INFO   | jvm 1    | 2013/12/09 12:47:01 |       at 
com.continuent.tungsten.replicator.applier.JdbcApplier.bindColumnValues(JdbcAppl
ier.java:540)
INFO   | jvm 1    | 2013/12/09 12:47:01 |       at 
com.continuent.tungsten.replicator.applier.JdbcApplier.applyOneRowChangePrepared
(JdbcApplier.java:1005)
INFO   | jvm 1    | 2013/12/09 12:47:01 |       at 
com.continuent.tungsten.replicator.applier.JdbcApplier.applyRowChangeData(JdbcAp
plier.java:1234)
INFO   | jvm 1    | 2013/12/09 12:47:01 |       at 
com.continuent.tungsten.replicator.applier.JdbcApplier.apply(JdbcApplier.java:13
27)
INFO   | jvm 1    | 2013/12/09 12:47:01 |       at 
com.continuent.tungsten.replicator.applier.ApplierWrapper.apply(ApplierWrapper.j
ava:101)
INFO   | jvm 1    | 2013/12/09 12:47:01 |       at 
com.continuent.tungsten.replicator.pipeline.SingleThreadStageTask.apply(SingleTh
readStageTask.java:706)
INFO   | jvm 1    | 2013/12/09 12:47:01 |       at 
com.continuent.tungsten.replicator.pipeline.SingleThreadStageTask.runTask(Single
ThreadStageTask.java:484)
INFO   | jvm 1    | 2013/12/09 12:47:01 |       at 
com.continuent.tungsten.replicator.pipeline.SingleThreadStageTask.run(SingleThre
adStageTask.java:169)
INFO   | jvm 1    | 2013/12/09 12:47:01 |       at 
java.lang.Thread.run(Thread.java:724)

In the case of a missed ALTER:
INFO   | jvm 1    | 2013/12/09 12:54:43 | 2013-12-09 12:54:43,903 [alpha - 
q-to-dbms-12] ERROR pipeline.SingleThreadStageTask Event application failed: 
seqno=191475492 fragno=0 message=java.sql.SQLSyntaxErrorException: Unknown 
column '' in 'field list'
INFO   | jvm 1    | 2013/12/09 12:54:43 | 
com.continuent.tungsten.replicator.applier.ApplierException: 
java.sql.SQLSyntaxErrorException: Unknown column '' in 'field list'
INFO   | jvm 1    | 2013/12/09 12:54:43 |       at 
com.continuent.tungsten.replicator.applier.JdbcApplier.applyOneRowChangePrepared
(JdbcApplier.java:1053)
INFO   | jvm 1    | 2013/12/09 12:54:43 |       at 
com.continuent.tungsten.replicator.applier.JdbcApplier.applyRowChangeData(JdbcAp
plier.java:1234)
INFO   | jvm 1    | 2013/12/09 12:54:43 |       at 
com.continuent.tungsten.replicator.applier.JdbcApplier.apply(JdbcApplier.java:13
27)
INFO   | jvm 1    | 2013/12/09 12:54:43 |       at 
com.continuent.tungsten.replicator.applier.ApplierWrapper.apply(ApplierWrapper.j
ava:101)
INFO   | jvm 1    | 2013/12/09 12:54:43 |       at 
com.continuent.tungsten.replicator.pipeline.SingleThreadStageTask.apply(SingleTh
readStageTask.java:706)
INFO   | jvm 1    | 2013/12/09 12:54:43 |       at 
com.continuent.tungsten.replicator.pipeline.SingleThreadStageTask.runTask(Single
ThreadStageTask.java:484)
INFO   | jvm 1    | 2013/12/09 12:54:43 |       at 
com.continuent.tungsten.replicator.pipeline.SingleThreadStageTask.run(SingleThre
adStageTask.java:169)
INFO   | jvm 1    | 2013/12/09 12:54:43 |       at 
java.lang.Thread.run(Thread.java:724)
INFO   | jvm 1    | 2013/12/09 12:54:43 | Caused by: 
java.sql.SQLSyntaxErrorException: Unknown column '' in 'field list'
INFO   | jvm 1    | 2013/12/09 12:54:43 |       at 
org.drizzle.jdbc.internal.SQLExceptionMapper.get(SQLExceptionMapper.java:99)
INFO   | jvm 1    | 2013/12/09 12:54:43 |       at 
org.drizzle.jdbc.DrizzlePreparedStatement.executeUpdate(DrizzlePreparedStatement
.java:128)
INFO   | jvm 1    | 2013/12/09 12:54:43 |       at 
com.continuent.tungsten.replicator.applier.JdbcApplier.applyOneRowChangePrepared
(JdbcApplier.java:1017)
INFO   | jvm 1    | 2013/12/09 12:54:43 |       ... 7 more
INFO   | jvm 1    | 2013/12/09 12:54:43 | Caused by: 
org.drizzle.jdbc.internal.common.QueryException: Unknown column '' in 'field 
list'
INFO   | jvm 1    | 2013/12/09 12:54:43 |       at 
org.drizzle.jdbc.internal.mysql.MySQLProtocol.executeQuery(MySQLProtocol.java:46
5)
INFO   | jvm 1    | 2013/12/09 12:54:43 |       at 
org.drizzle.jdbc.DrizzlePreparedStatement.executeUpdate(DrizzlePreparedStatement
.java:125)
INFO   | jvm 1    | 2013/12/09 12:54:43 |       ... 8 more

What version of the product are you using?

tungsten-replicator-2.1.0-343 with 
Percona-Server-server-55-5.5.32-rel31.0.549.rhel6.x86_64

On what operating system?

Centos 6.4

java version "1.7.0_40"
Java(TM) SE Runtime Environment (build 1.7.0_40-b43)
Java HotSpot(TM) 64-Bit Server VM (build 24.0-b56, mixed mode)

Please provide any additional information below.

Notes:

We're running Master-Master replication, using row based replication.

I'm going to try to collect a better sequence of events the next time we see 
this happen.

Original issue reported on code.google.com by bran...@seibelnet.ca on 12 Dec 2013 at 2:28

GoogleCodeExporter commented 9 years ago
Please try to provide a simple set of SQL events that can trigger this behavior.
Do the table or column names contain non alphanumeric characters?

Original comment by g.maxia on 12 Dec 2013 at 2:36

GoogleCodeExporter commented 9 years ago
We are having the same issue.

Version: Tungsten Replicator 2.1.0 build 343, MySQL 5.5
Topology: Master-Master

Error:
pendingExceptionMessage: java.sql.SQLSyntaxErrorException: Unknown column '' in 
'field list'
Failing statement : INSERT INTO `testing`.`test` ( `id` , `image` , `` )  
VALUES (  ?  ,  UNHEX( ? )   ,  ?  ) 

How to Reproduce:
1. Node A & B are in sync
2. Alter table "test" in Node A by adding one new column
3. Verify both Node A & B table schema are in sync
4. Run an INSERT statement to table "test" in Node B using PHP
5. Insert will be success in Node B using PHP, but the replicator sync will 
stop with the error above

We also tested the follow steps and NO error occur:
1. Node A & B are in sync
2. Alter table "test" in Node A by adding one new column
3. Verify both Node A & B table schema are in sync
4. Use mysql command line and run "select * from test"
4. Run an INSERT statement in Node B using PHP
5. Insert will be success in BOTH Node A & B using PHP

Just guessing:
It seems the replicator has cached the table structure.

Thank you.

Original comment by fran...@franfran.com on 14 Aug 2014 at 10:42

GoogleCodeExporter commented 9 years ago
I can't reproduce this issue using the steps listed above. I am using 
Tungsten-replicator 3.0.0-286. I would appreciate some more information. I 
believe this problem may happen when the alter table and the subsequent insert 
are too close to each other. I had no luck in finding the right sequence of 
events that lead to the problem described in this issue.

Original comment by g.maxia on 18 Aug 2014 at 3:37