zendesk / maxwell

Maxwell's daemon, a mysql-to-json kafka producer
https://maxwells-daemon.io/
Other
4.05k stars 1.01k forks source link

Maxwell shutdown on 'bool' and 'boolean' #67

Closed akshayi1 closed 9 years ago

akshayi1 commented 9 years ago

Same as #64, #65 and #66, Maxwell can't handle 'bool' and (just to be completely sure) 'boolean'. This occurs at the create table stage. To get Maxwell to run again, I had to purge binary logs before now() and delete from maxwell.positions.

BOOL:

bin/maxwell --user='root' --password='root' --host='127.0.0.1' --producer=stdout
Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/share/java/jayatanaag.jar 
10:40:30,183 INFO  Maxwell - Maxwell is capturing initial schema
10:40:30,185 DEBUG SchemaStore - Capturing schema
10:40:30,217 DEBUG SchemaStore - Finished capturing schema
10:40:30,294 INFO  TransportImpl - connecting to host: 127.0.0.1, port: 3306
10:40:30,343 INFO  TransportImpl - connected to host: 127.0.0.1, port: 3306, context: AbstractTransport.Context[threadId=31,scramble=}_EvG<6<gt16X;]DyW(],protocolVersion=10,serverHost=127.0.0.1,serverPort=3306,serverStatus=2,serverCollation=8,serverVersion=5.6.24-0ubuntu2-log,serverCapabilities=63487]
10:40:30,344 INFO  AuthenticatorImpl - start to login, user: root, host: 127.0.0.1, port: 3306
10:40:30,348 INFO  AuthenticatorImpl - login successfully, user: root, detail: OKPacket[packetMarker=0,affectedRows=0,insertId=0,serverStatus=2,warningCount=0,message=<null>]
10:40:31,175 DEBUG SchemaPosition - Writing binlog position to maxwell.positions: BinlogPosition[mysql-logs.000003:12471]
10:55:36,558 DEBUG SchemaChange - SQL_PARSE <- "create table mmf_test(col1 smallint, col2 int, col3 bigint,  col4 float, col5 date, col6 datetime, col7 decimal(10,2), col8 varchar(500), col9 text, col10 bool)"
line 1:155 no viable alternative at input 'bool'
10:55:36,590 ERROR MysqlParserListener - (data_type bool)
10:55:36,590 ERROR SchemaChange - Error parsing SQL: 'create table mmf_test(col1 smallint, col2 int, col3 bigint,  col4 float, col5 date, col6 datetime, col7 decimal(10,2), col8 varchar(500), col9 text, col10 bool)'
com.zendesk.maxwell.schema.ddl.MaxwellSQLSyntaxError: bool
    at com.zendesk.maxwell.schema.ddl.MysqlParserListener.visitErrorNode(MysqlParserListener.java:85)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:41)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:65)
    at com.zendesk.maxwell.MaxwellParser.processQueryEvent(MaxwellParser.java:206)
    at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:192)
    at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:198)
    at com.zendesk.maxwell.MaxwellParser.run(MaxwellParser.java:98)
    at com.zendesk.maxwell.Maxwell.run(Maxwell.java:69)
    at com.zendesk.maxwell.Maxwell.main(Maxwell.java:75)
2015-06-18 17:55:36,635 FATAL Unable to register shutdown hook because JVM is shutting down.

BOOLEAN:

bin/maxwell --user='root' --password='root' --host='127.0.0.1' --producer=stdout
Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/share/java/jayatanaag.jar 
10:58:46,071 INFO  Maxwell - Maxwell is capturing initial schema
10:58:46,073 DEBUG SchemaStore - Capturing schema
10:58:46,111 DEBUG SchemaStore - Finished capturing schema
10:58:47,020 INFO  TransportImpl - connecting to host: 127.0.0.1, port: 3306
10:58:47,044 INFO  TransportImpl - connected to host: 127.0.0.1, port: 3306, context: AbstractTransport.Context[threadId=35,scramble=pWD+Jt,w6D5z*_~!&7#J,protocolVersion=10,serverHost=127.0.0.1,serverPort=3306,serverStatus=2,serverCollation=8,serverVersion=5.6.24-0ubuntu2-log,serverCapabilities=63487]
10:58:47,045 INFO  AuthenticatorImpl - start to login, user: root, host: 127.0.0.1, port: 3306
10:58:47,054 INFO  AuthenticatorImpl - login successfully, user: root, detail: OKPacket[packetMarker=0,affectedRows=0,insertId=0,serverStatus=2,warningCount=0,message=<null>]
10:58:47,063 DEBUG SchemaPosition - Writing binlog position to maxwell.positions: BinlogPosition[mysql-logs.000003:18202]
10:58:55,665 DEBUG SchemaChange - SQL_PARSE <- "create table mmf_test(col1 smallint, col2 int, col3 bigint,  col4 float, col5 date, col6 datetime, col7 decimal(10,2), col8 varchar(500), col9 text, col10 boolean)"
line 1:155 no viable alternative at input 'boolean'
10:58:55,712 ERROR MysqlParserListener - (data_type boolean)
10:58:55,712 ERROR SchemaChange - Error parsing SQL: 'create table mmf_test(col1 smallint, col2 int, col3 bigint,  col4 float, col5 date, col6 datetime, col7 decimal(10,2), col8 varchar(500), col9 text, col10 boolean)'
com.zendesk.maxwell.schema.ddl.MaxwellSQLSyntaxError: boolean
    at com.zendesk.maxwell.schema.ddl.MysqlParserListener.visitErrorNode(MysqlParserListener.java:85)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:41)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:65)
    at com.zendesk.maxwell.MaxwellParser.processQueryEvent(MaxwellParser.java:206)
    at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:192)
    at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:198)
    at com.zendesk.maxwell.MaxwellParser.run(MaxwellParser.java:98)
    at com.zendesk.maxwell.Maxwell.run(Maxwell.java:69)
    at com.zendesk.maxwell.Maxwell.main(Maxwell.java:75)
akshayi1 commented 9 years ago

For this, I tried to do:

ColumnDef.java

public static ColumnDef build(String tableName, String name, String encoding, String type, int pos, boolean signed, String enumValues[]) {
        type = unalias_type(type);

        switch(type) {

        case "tinyint":
        case "smallint":
        case "mediumint":
        case "bit":
        case "int":
            return new IntColumnDef(tableName, name, type, pos, signed);
....
....
....

static private String unalias_type(String type) {
        switch(type) {
            case "bool":
            case "boolean":
            case "bit":
                return "bit";
            case "int1":
                return "tinyint";
            case "int2":
                return "smallint";
            case "int3":
                return "mediumint";
            case "int4":
            case "integer":
                return "int";
            case "int8":
                return "bigint";
            default:
                return type;
        }
    }

IntColumnDef.java

public boolean matchesMysqlType(int type) {
        switch(this.bits) {
        case 1:
            return type == MySQLConstants.TYPE_BIT;
        case 8:
            return type == MySQLConstants.TYPE_TINY;
        case 16:
            return type == MySQLConstants.TYPE_SHORT;
        case 24:
            return type == MySQLConstants.TYPE_INT24;
        case 32:
            return type == MySQLConstants.TYPE_LONG;
        default:
            return false;
        }
    }

    private final static int bitsFromType(String type) {
        switch(type) {
        case "bit":
            return 1;
        case "tinyint":
            return 8;
        case "smallint":
            return 16;
        case "mediumint":
            return 24;
        case "int":
            return 32;
        default:
            return 0;
        }
    }

I tried a few combinations of the same - without the bit and so on, and nothing seemed to work. It was almost as if my changed code did not even exist! I've kinda given up on getting bool to work.

osheroff commented 9 years ago

working on this one.

akshayi1 commented 9 years ago

68 fixes this. Thanks!