search

is00hcw / tungsten-replicator

Automatically exported from code.google.com/p/tungsten-replicator
0 stars 1 forks source link

Generated message class "...$ProtobufStatementData" missing method "getQueryBytes". #1067

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
# What steps will reproduce the problem?

Perform an Oracle to MySQL provisioning as outlined here:
http://docs.continuent.com/tungsten-replicator-3.0/deployment-oracle-provision.h
tml

# What is the expected output?

Parallel extraction to complete successfully.

# What do you see instead?

The extractor started without a problem, but halted after writing a few 
thousand events to thl.

INFO   | jvm 1    | 2014/12/04 20:56:51 | 2014-12-04 20:56:51,869 [cookbook - 
prov-q-to-thl-0] WARN  thl.serializer.ProtobufSerializer Unimplemented type -3
INFO   | jvm 1    | 2014/12/04 20:56:52 | 2014-12-04 20:56:52,169 [cookbook - 
prov-q-to-thl-0] ERROR pipeline.SingleThreadStageTask Stage task failed: 
stage=prov-q-to-thl seqno=2151 fragno=0
INFO   | jvm 1    | 2014/12/04 20:56:52 | java.lang.ExceptionInInitializerError
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.continuent.tungsten.replicator.thl.protobuf.TungstenProtos$ProtobufOneRowCha
nge$ProtobufRowValue$ProtobufColumnVal.internalGetFieldAccessorTable(TungstenPro
tos.java:2823)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.google.protobuf.GeneratedMessage.getDescriptorForType(GeneratedMessage.java:
97)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.google.protobuf.AbstractMessage$Builder.findMissingFields(AbstractMessage.ja
va:623)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.google.protobuf.AbstractMessage$Builder.findMissingFields(AbstractMessage.ja
va:614)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.google.protobuf.AbstractMessage$Builder.newUninitializedMessageException(Abs
tractMessage.java:605)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.continuent.tungsten.replicator.thl.protobuf.TungstenProtos$ProtobufOneRowCha
nge$ProtobufRowValue$ProtobufColumnVal$Builder.build(TungstenProtos.java:3407)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.continuent.tungsten.replicator.thl.protobuf.TungstenProtos$ProtobufOneRowCha
nge$ProtobufRowValue$Builder.addColumnValue(TungstenProtos.java:4463)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.continuent.tungsten.replicator.thl.serializer.ProtobufSerializer.serializeRo
ws(ProtobufSerializer.java:503)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.continuent.tungsten.replicator.thl.serializer.ProtobufSerializer.serializeEv
ent(ProtobufSerializer.java:222)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.continuent.tungsten.replicator.thl.log.LogEventReplWriter.write(LogEventRepl
Writer.java:79)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.continuent.tungsten.replicator.thl.log.LogConnection.store(LogConnection.jav
a:777)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.continuent.tungsten.replicator.thl.THLStoreApplier.apply(THLStoreApplier.jav
a:151)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.continuent.tungsten.replicator.pipeline.SingleThreadStageTask.apply(SingleTh
readStageTask.java:797)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.continuent.tungsten.replicator.pipeline.SingleThreadStageTask.runTask(Single
ThreadStageTask.java:530)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.continuent.tungsten.replicator.pipeline.SingleThreadStageTask.run(SingleThre
adStageTask.java:179)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
java.lang.Thread.run(Thread.java:745)
INFO   | jvm 1    | 2014/12/04 20:56:52 | Caused by: 
java.lang.RuntimeException: Generated message class 
"com.continuent.tungsten.replicator.thl.protobuf.TungstenProtos$ProtobufStatemen
tData" missing method "getQueryBytes".
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.google.protobuf.GeneratedMessage.getMethodOrDie(GeneratedMessage.java:1359)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.google.protobuf.GeneratedMessage.access$1300(GeneratedMessage.java:57)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.google.protobuf.GeneratedMessage$FieldAccessorTable$SingularFieldAccessor.<i
nit>(GeneratedMessage.java:1484)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.google.protobuf.GeneratedMessage$FieldAccessorTable.<init>(GeneratedMessage.
java:1432)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.continuent.tungsten.replicator.thl.protobuf.TungstenProtos$1.assignDescripto
rs(TungstenProtos.java:16461)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.google.protobuf.Descriptors$FileDescriptor.internalBuildGeneratedFileFrom(De
scriptors.java:298)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.continuent.tungsten.replicator.thl.protobuf.TungstenProtos.<clinit>(Tungsten
Protos.java:16506)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       ... 16 more
INFO   | jvm 1    | 2014/12/04 20:56:52 | Caused by: 
java.lang.NoSuchMethodException: 
com.continuent.tungsten.replicator.thl.protobuf.TungstenProtos$ProtobufStatement
Data.getQueryBytes()
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
java.lang.Class.getMethod(Class.java:1670)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.google.protobuf.GeneratedMessage.getMethodOrDie(GeneratedMessage.java:1357)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       ... 22 more
INFO   | jvm 1    | 2014/12/04 20:56:52 | 2014-12-04 20:56:52,213 [cookbook - 
prov-q-to-thl-0] INFO  pipeline.SingleThreadStageTask Performing emergency 
rollback of applied changes
INFO   | jvm 1    | 2014/12/04 20:56:52 | 2014-12-04 20:56:52,213 [cookbook - 
prov-q-to-thl-0] INFO  pipeline.SingleThreadStageTask Dispatching error event: 
Stage task failed: stage=prov-q-to-thl seqno=2151 fragno=0
INFO   | jvm 1    | 2014/12/04 20:56:52 | 2014-12-04 20:56:52,213 [cookbook - 
prov-q-to-thl-0] INFO  pipeline.SingleThreadStageTask Terminating processing 
for stage task thread
INFO   | jvm 1    | 2014/12/04 20:56:52 | 2014-12-04 20:56:52,213 [cookbook - 
prov-q-to-thl-0] INFO  pipeline.SingleThreadStageTask Last successfully 
processed event prior to termination: seqno=2151 eventid=ora:455798449
INFO   | jvm 1    | 2014/12/04 20:56:52 | 2014-12-04 20:56:52,213 [cookbook - 
prov-q-to-thl-0] INFO  pipeline.SingleThreadStageTask Task event count: 2010
INFO   | jvm 1    | 2014/12/04 20:56:52 | 2014-12-04 20:56:52,228 [cookbook - 
pool-2-thread-1] ERROR management.OpenReplicatorManager Received error 
notification, shutting down services :
INFO   | jvm 1    | 2014/12/04 20:56:52 | Stage task failed: 
stage=prov-q-to-thl seqno=2151 fragno=0
INFO   | jvm 1    | 2014/12/04 20:56:52 | java.lang.ExceptionInInitializerError
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.continuent.tungsten.replicator.thl.protobuf.TungstenProtos$ProtobufOneRowCha
nge$ProtobufRowValue$ProtobufColumnVal.internalGetFieldAccessorTable(TungstenPro
tos.java:2823)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.google.protobuf.GeneratedMessage.getDescriptorForType(GeneratedMessage.java:
97)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.google.protobuf.AbstractMessage$Builder.findMissingFields(AbstractMessage.ja
va:623)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.google.protobuf.AbstractMessage$Builder.findMissingFields(AbstractMessage.ja
va:614)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.google.protobuf.AbstractMessage$Builder.newUninitializedMessageException(Abs
tractMessage.java:605)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.continuent.tungsten.replicator.thl.protobuf.TungstenProtos$ProtobufOneRowCha
nge$ProtobufRowValue$ProtobufColumnVal$Builder.build(TungstenProtos.java:3407)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.continuent.tungsten.replicator.thl.protobuf.TungstenProtos$ProtobufOneRowCha
nge$ProtobufRowValue$Builder.addColumnValue(TungstenProtos.java:4463)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.continuent.tungsten.replicator.thl.serializer.ProtobufSerializer.serializeRo
ws(ProtobufSerializer.java:503)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.continuent.tungsten.replicator.thl.serializer.ProtobufSerializer.serializeEv
ent(ProtobufSerializer.java:222)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.continuent.tungsten.replicator.thl.log.LogEventReplWriter.write(LogEventRepl
Writer.java:79)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.continuent.tungsten.replicator.thl.log.LogConnection.store(LogConnection.jav
a:777)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.continuent.tungsten.replicator.thl.THLStoreApplier.apply(THLStoreApplier.jav
a:151)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.continuent.tungsten.replicator.pipeline.SingleThreadStageTask.apply(SingleTh
readStageTask.java:797)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.continuent.tungsten.replicator.pipeline.SingleThreadStageTask.runTask(Single
ThreadStageTask.java:530)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.continuent.tungsten.replicator.pipeline.SingleThreadStageTask.run(SingleThre
adStageTask.java:179)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
java.lang.Thread.run(Thread.java:745)
INFO   | jvm 1    | 2014/12/04 20:56:52 | Caused by: 
java.lang.RuntimeException: Generated message class 
"com.continuent.tungsten.replicator.thl.protobuf.TungstenProtos$ProtobufStatemen
tData" missing method "getQueryBytes".
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.google.protobuf.GeneratedMessage.getMethodOrDie(GeneratedMessage.java:1359)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.google.protobuf.GeneratedMessage.access$1300(GeneratedMessage.java:57)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.google.protobuf.GeneratedMessage$FieldAccessorTable$SingularFieldAccessor.<i
nit>(GeneratedMessage.java:1484)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.google.protobuf.GeneratedMessage$FieldAccessorTable.<init>(GeneratedMessage.
java:1432)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.continuent.tungsten.replicator.thl.protobuf.TungstenProtos$1.assignDescripto
rs(TungstenProtos.java:16461)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.google.protobuf.Descriptors$FileDescriptor.internalBuildGeneratedFileFrom(De
scriptors.java:298)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.continuent.tungsten.replicator.thl.protobuf.TungstenProtos.<clinit>(Tungsten
Protos.java:16506)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       ... 16 more
INFO   | jvm 1    | 2014/12/04 20:56:52 | Caused by: 
java.lang.NoSuchMethodException: 
com.continuent.tungsten.replicator.thl.protobuf.TungstenProtos$ProtobufStatement
Data.getQueryBytes()
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
java.lang.Class.getMethod(Class.java:1670)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       at 
com.google.protobuf.GeneratedMessage.getMethodOrDie(GeneratedMessage.java:1357)
INFO   | jvm 1    | 2014/12/04 20:56:52 |       ... 22 more
INFO   | jvm 1    | 2014/12/04 20:56:52 | 2014-12-04 20:56:52,229 [cookbook - 
pool-2-thread-1] WARN  management.OpenReplicatorManager Performing emergency 
service shutdown

# What version of the product are you using?

Tungsten Replicator: 3.0.0-524

# On what operating system?

CentOS release 6.6 (Final)

# Notes:

Judging from the error message, I'm suspecting that some post-build 
optimization tool like
Proguard has stripped away the required name(s).

ref: 
http://stackoverflow.com/questions/14098225/what-are-features-of-using-proguard-
in-project-with-protocol-buffers

Original issue reported on code.google.com by ento.ent...@gmail.com on 4 Dec 2014 at 12:48

GoogleCodeExporter commented 9 years ago 
I take back my notes about Proguard. I've managed to reproduce the same 
exception in a unit test by trying to serialize a row change data with a column 
value of type java.sql.VARBINARY.

In my case I can work around this issue by ignoring the offending table.

Original comment by ento.ent...@gmail.com on 5 Dec 2014 at 10:45

Attachments:

GoogleCodeExporter commented 9 years ago 
It turned out that the table being replicated was a CDC table.

I've built a modified tungsten-replicator.jar which logs serialization errors 
more verbosely and the table in question was a CDC table.

> 2014-12-08 14:59:37,428 [cookbook - prov-q-to-thl-0] ERROR 
thl.serializer.ProtobufSerializer Failure while storing MYSCHEMA_PUB.CT_USER

To work around this issue, provide a csv file in the form of 
"schema,table,chunk_size" and specify it as 
`replicator.extractor.parallel-extractor.ChunkDefinitionFile` property.

I request that:

- Parallel extractor should ignore CDC tables
- The ChunkDefinitionFile property should be documented

Original comment by ento.ent...@gmail.com on 8 Dec 2014 at 8:01