search

zendesk / maxwell

Maxwell's daemon, a mysql-to-json kafka producer
https://maxwells-daemon.io/
Other
4.03k stars 1.01k forks source link

polardb-x(aliyun) failed to execute the create index statement #1927

Closed wanghangyu817 closed 1 year ago

wanghangyu817 commented 2 years ago

Execute SQL ： CREATE INDEX device_id_idx ON event_tracking_info_extra (event, create_time)

Exception:

31077 [DEBUG] C3P0PooledConnectionPool: Testing PooledConnection [com.mchange.v2.c3p0.impl.NewPooledConnection@38d72334] on CHECKOUT.
31078 [DEBUG] C3P0PooledConnectionPool: Test of PooledConnection [com.mchange.v2.c3p0.impl.NewPooledConnection@38d72334] on CHECKOUT has SUCCEEDED.
31085 [DEBUG] SchemaChange: SQL_PARSE <- "# POLARX_ORIGIN_SQL=CREATE INDEX device_id_idx ON event_tracking_info_extra (event, create_time)
# POLARX_TSO=xxxxx
CREATE INDEX device_id_idx ON event_tracking_info_extra (event, create_time)"
31156 [ERROR] MysqlParserListener: (statement CREATE INDEX device_id_idx ON event_tracking_info_extra ( event , create_time ))
31156 [ERROR] SchemaChange: Error parsing SQL: '# POLARX_ORIGIN_SQL=CREATE INDEX device_id_idx ON event_tracking_info_extra (event, create_time)
# POLARX_TSO=xxxxx
CREATE INDEX device_id_idx ON event_tracking_info_extra (event, create_time)'
31156 [ERROR] AbstractSchemaStore: Error on bin log position Position[BinlogPosition[binlog.000001:329876], lastHeartbeat=1666677625205]
com.zendesk.maxwell.schema.ddl.MaxwellSQLSyntaxError: CREATE
        at com.zendesk.maxwell.schema.ddl.MysqlParserListener.visitErrorNode(MysqlParserListener.java:93)
        at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:17)
        at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28)
        at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28)
        at com.zendesk.maxwell.schema.ddl.SchemaChange.parseSQL(SchemaChange.java:94)
        at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:109)
        at com.zendesk.maxwell.schema.AbstractSchemaStore.resolveSQL(AbstractSchemaStore.java:50)
        at com.zendesk.maxwell.schema.MysqlSchemaStore.processSQL(MysqlSchemaStore.java:103)
        at com.zendesk.maxwell.replication.BinlogConnectorReplicator.processQueryEvent(BinlogConnectorReplicator.java:336)
        at com.zendesk.maxwell.replication.BinlogConnectorReplicator.processQueryEvent(BinlogConnectorReplicator.java:358)
        at com.zendesk.maxwell.replication.BinlogConnectorReplicator.getRow(BinlogConnectorReplicator.java:675)
        at com.zendesk.maxwell.replication.BinlogConnectorReplicator.work(BinlogConnectorReplicator.java:186)
        at com.zendesk.maxwell.util.RunLoopProcess.runLoop(RunLoopProcess.java:34)
        at com.zendesk.maxwell.Maxwell.startInner(Maxwell.java:256)
        at com.zendesk.maxwell.Maxwell.start(Maxwell.java:183)
        at com.zendesk.maxwell.Maxwell.main(Maxwell.java:287)
31157 [INFO] BinlogConnectorReplicator: Binlog disconnected.
31158 [INFO] TaskManager: Stopping 3 tasks
31158 [ERROR] TaskManager: cause: 
com.zendesk.maxwell.schema.ddl.MaxwellSQLSyntaxError: CREATE
        at com.zendesk.maxwell.schema.ddl.MysqlParserListener.visitErrorNode(MysqlParserListener.java:93) ~[maxwell-1.35.0.jar:1.35.0]
        at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:17) ~[antlr4-runtime-4.8-1.jar:4.8-1]
        at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28) ~[antlr4-runtime-4.8-1.jar:4.8-1]
        at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28) ~[antlr4-runtime-4.8-1.jar:4.8-1]
        at com.zendesk.maxwell.schema.ddl.SchemaChange.parseSQL(SchemaChange.java:94) ~[maxwell-1.35.0.jar:1.35.0]
        at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:109) ~[maxwell-1.35.0.jar:1.35.0]
        at com.zendesk.maxwell.schema.AbstractSchemaStore.resolveSQL(AbstractSchemaStore.java:50) ~[maxwell-1.35.0.jar:1.35.0]
        at com.zendesk.maxwell.schema.MysqlSchemaStore.processSQL(MysqlSchemaStore.java:103) ~[maxwell-1.35.0.jar:1.35.0]
        at com.zendesk.maxwell.replication.BinlogConnectorReplicator.processQueryEvent(BinlogConnectorReplicator.java:336) ~[maxwell-1.35.0.jar:1.35.0]
        at com.zendesk.maxwell.replication.BinlogConnectorReplicator.processQueryEvent(BinlogConnectorReplicator.java:358) ~[maxwell-1.35.0.jar:1.35.0]
        at com.zendesk.maxwell.replication.BinlogConnectorReplicator.getRow(BinlogConnectorReplicator.java:675) ~[maxwell-1.35.0.jar:1.35.0]
        at com.zendesk.maxwell.replication.BinlogConnectorReplicator.work(BinlogConnectorReplicator.java:186) ~[maxwell-1.35.0.jar:1.35.0]
        at com.zendesk.maxwell.util.RunLoopProcess.runLoop(RunLoopProcess.java:34) ~[maxwell-1.35.0.jar:1.35.0]
        at com.zendesk.maxwell.Maxwell.startInner(Maxwell.java:256) ~[maxwell-1.35.0.jar:1.35.0]
        at com.zendesk.maxwell.Maxwell.start(Maxwell.java:183) ~[maxwell-1.35.0.jar:1.35.0]
        at com.zendesk.maxwell.Maxwell.main(Maxwell.java:287) ~[maxwell-1.35.0.jar:1.35.0]
31170 [INFO] TaskManager: Stopping: com.zendesk.maxwell.schema.PositionStoreThread@89866c7
31170 [DEBUG] StoppableTaskState: com.zendesk.maxwell.schema.PositionStoreThread requestStop() called (in state: RUNNING)
31170 [INFO] TaskManager: Stopping: com.zendesk.maxwell.bootstrap.BootstrapController@1e486e4
31170 [DEBUG] StoppableTaskState: com.zendesk.maxwell.bootstrap.BootstrapController requestStop() called (in state: RUNNING)
31170 [INFO] TaskManager: Stopping: com.zendesk.maxwell.replication.BinlogConnectorReplicator@6300e18c
31170 [DEBUG] StoppableTaskState: com.zendesk.maxwell.replication.BinlogConnectorReplicator requestStop() called (in state: STOPPED)
31170 [DEBUG] TaskManager: Awaiting stop of: com.zendesk.maxwell.schema.PositionStoreThread@89866c7
com.zendesk.maxwell.schema.ddl.MaxwellSQLSyntaxError: CREATE
        at com.zendesk.maxwell.schema.ddl.MysqlParserListener.visitErrorNode(MysqlParserListener.java:93)
        at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:17)
        at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28)
        at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28)
        at com.zendesk.maxwell.schema.ddl.SchemaChange.parseSQL(SchemaChange.java:94)
        at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:109)
        at com.zendesk.maxwell.schema.AbstractSchemaStore.resolveSQL(AbstractSchemaStore.java:50)
        at com.zendesk.maxwell.schema.MysqlSchemaStore.processSQL(MysqlSchemaStore.java:103)
        at com.zendesk.maxwell.replication.BinlogConnectorReplicator.processQueryEvent(BinlogConnectorReplicator.java:336)
        at com.zendesk.maxwell.replication.BinlogConnectorReplicator.processQueryEvent(BinlogConnectorReplicator.java:358)
        at com.zendesk.maxwell.replication.BinlogConnectorReplicator.getRow(BinlogConnectorReplicator.java:675)
        at com.zendesk.maxwell.replication.BinlogConnectorReplicator.work(BinlogConnectorReplicator.java:186)
        at com.zendesk.maxwell.util.RunLoopProcess.runLoop(RunLoopProcess.java:34)
        at com.zendesk.maxwell.Maxwell.startInner(Maxwell.java:256)
        at com.zendesk.maxwell.Maxwell.start(Maxwell.java:183)
        at com.zendesk.maxwell.Maxwell.main(Maxwell.java:287)
31271 [DEBUG] TaskManager: Awaiting stop of: com.zendesk.maxwell.bootstrap.BootstrapController@1e486e4
31371 [DEBUG] TaskManager: Awaiting stop of: com.zendesk.maxwell.replication.BinlogConnectorReplicator@6300e18c
31471 [INFO] TaskManager: Stopped all tasks
osheroff commented 1 year ago

what version of maxwell? and can you recover the exact original SQL?

wanghangyu817 commented 1 year ago

I am currently using version 1.35.0, and when I test 1.38.0, I get the same error message, raw sql: CREATE INDEX device_id_idx ON event_tracking_info_extra(event, create_time). It is found in debug that a specific comment has been added to the executed sql, which is the cause of the error

osheroff commented 1 year ago

can you run liike show binlog events from ... and recover the exact original SQL? Your PR would just make it so that maxwell will never process any sql that contained a # comment style comment, which isn't right. I can't reproduce your issue without the full, original sql (including comments.)

wanghangyu817 commented 1 year ago

image

I get sql :

# POLARX_ORIGIN_SQL=CREATE INDEX device_id_idx ON event_tracking_info_extra (event, create_time) # POLARX_TSO=698905756181096044815201227773638819850000000000000000 CREATE INDEX device_id_idx ON event_tracking_info_extra (event, create_time)

image

wanghangyu817 commented 1 year ago

If '--output_ddl=false' does the processing in pr solve the problem (which I'm using now and don't need to output ddl), or does the processing in pr cause other problems

osheroff commented 1 year ago

no that won't solve your problem.

On Wed, Nov 2, 2022 at 6:31 AM Rain丨xiyang @.***> wrote:

If '--output_ddl=false' does the processing in pr solve the problem (which I'm using now and don't need to output ddl), or does the processing in pr cause other problems

— Reply to this email directly, view it on GitHub https://github.com/zendesk/maxwell/issues/1927#issuecomment-1300411048, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAB7P5EPZW4XAR7XEVXMQ63WGJULJANCNFSM6AAAAAARN37FXI . You are receiving this because you commented.Message ID: @.***>

wanghangyu817 commented 1 year ago

I'm currently using pr code in a production environment, and I'm worried about other issues. I think only ddl statements go into this regular expression

osheroff commented 1 year ago

fixed in latest