zendesk / maxwell

Maxwell's daemon, a mysql-to-json kafka producer
https://maxwells-daemon.io/
Other
4.02k stars 1.01k forks source link

Maxwell crash when bootstrapping a table. #369

Closed xmlking closed 8 years ago

xmlking commented 8 years ago

When try to bootstrap a table by inserting record into maxwell.bootstrap table ,Maxwell crashed with following errors: We are using Maxwell 1.1.1 wonder if it a bug or specific to my environment? Any help is appreciated -Sumo

15:19:46,869 ERROR SchemaScavenger - SchemaScavenger thread aborting after exception: java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction
02:39:43,336 WARN  MaxwellReplicator - no heartbeat heard from server in 2003ms.  restarting replication.
02:39:43,344 INFO  TransportImpl - disconnected from mysql_host:3306
02:39:43,345 INFO  TransportImpl - connecting to host: mysql_host, port: 3306
02:39:47,826 INFO  TransportImpl - connected to host: mysql_host, port: 3306, context: AbstractTransport.Context[threadId=24418,scramble=\i{MN}MFWes4E>H.Lt%U,protocolVersion=10,serverHost=mysql_host,serverPort=3306,serverStatus=2,serverCollation=8,serverVersion=5.6.24-72.2-25.11-log,serverCapabilities=65535]
02:39:47,826 INFO  AuthenticatorImpl - start to login, user: maxwell_user, host: mysql_host, port: 3306
02:39:47,840 INFO  AuthenticatorImpl - login successfully, user: maxwell_user, detail: OKPacket[packetMarker=0,affectedRows=0,insertId=0,serverStatus=2,warningCount=0,message=<null>]
com.google.code.or.net.TransportException: Lock wait timeout exceeded; try restarting transaction
        at com.google.code.or.net.impl.Query.getFirst(Query.java:40)
        at com.google.code.or.OpenReplicator.setupChecksumState(OpenReplicator.java:268)
        at com.google.code.or.OpenReplicator.start(OpenReplicator.java:109)
        at com.zendesk.maxwell.MaxwellReplicator.ensureReplicatorThread(MaxwellReplicator.java:94)
        at com.zendesk.maxwell.MaxwellReplicator.getRow(MaxwellReplicator.java:280)
        at com.zendesk.maxwell.MaxwellReplicator.work(MaxwellReplicator.java:115)
        at com.zendesk.maxwell.RunLoopProcess.runLoop(RunLoopProcess.java:31)
        at com.zendesk.maxwell.Maxwell.run(Maxwell.java:101)
        at com.zendesk.maxwell.Maxwell.main(Maxwell.java:107)
02:39:54,257 ERROR SchemaPosition - received SQLException while trying to save to maxwell.positions:
02:39:54,257 ERROR SchemaPosition - WSREP has not yet prepared node for application use

Other bootstrap error:

11:42:19,082 INFO  AuthenticatorImpl - start to login, user: maxwell_user, host: mysql_host, port: 3306
11:42:19,086 INFO  AuthenticatorImpl - login successfully, user: maxwell_user, detail: OKPacket[packetMarker=0,affectedRows=0,insertId=0,serverStatus=2,warningCount=0,message=<null>]
11:52:19,151 WARN  Selector - Error in I/O with dbsrs1496/10.207.146.57
12:58:10,252 INFO  MaxwellReplicator - bootstrapping started for mydb.user_tab, binlog position is BinlogPosition[binarylog.042355:33264]
java.sql.SQLException: Cannot convert value '2016-06-13 05:56:59.000' from column 4 to TIMESTAMP.
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1055)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:956)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:926)
        at com.mysql.jdbc.ResultSetRow.getTimestampFast(ResultSetRow.java:1321)
        at com.mysql.jdbc.BufferRow.getTimestampFast(BufferRow.java:573)
        at com.mysql.jdbc.ResultSetImpl.getTimestampInternal(ResultSetImpl.java:6617)
        at com.mysql.jdbc.ResultSetImpl.getTimestamp(ResultSetImpl.java:5943)
        at com.mysql.jdbc.ResultSetImpl.getObject(ResultSetImpl.java:4901)
        at com.zendesk.maxwell.bootstrap.SynchronousBootstrapper.setRowValues(SynchronousBootstrapper.java:221)
        at com.zendesk.maxwell.bootstrap.SynchronousBootstrapper.startBootstrap(SynchronousBootstrapper.java:58)
        at com.zendesk.maxwell.bootstrap.AsynchronousBootstrapper$1.run(AsynchronousBootstrapper.java:70)
        at java.lang.Thread.run(Thread.java:745)
ADMINISTRATOR0016-06-13 05:56:59.000hts/jsp/security/systemMenu.jsp2016-06-13 05:56:59.000
        at com.mysql.jdbc.StringUtils.getInt(StringUtils.java:756)
        at com.mysql.jdbc.ResultSetRow.getTimestampFast(ResultSetRow.java:1106)
        ... 8 more
12:58:10,388 INFO  TransportImpl - disconnected from mysql_host:3306
12:58:10,397 INFO  SchemaPosition - Storing final position: BinlogPosition[binarylog.042355:33264]

Other bootstrap error:

13:09:30,344 INFO  MaxwellReplicator - bootstrapping started for mydb.user_tab, binlog position is BinlogPosition[binarylog.042356:22106]
java.sql.SQLException: Cannot convert value '2016-06-13 05:56:59.000' from column 4 to TIMESTAMP.
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1055)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:956)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:926)
        at com.mysql.jdbc.ResultSetRow.getTimestampFast(ResultSetRow.java:1321)
        at com.mysql.jdbc.BufferRow.getTimestampFast(BufferRow.java:573)
        at com.mysql.jdbc.ResultSetImpl.getTimestampInternal(ResultSetImpl.java:6617)
        at com.mysql.jdbc.ResultSetImpl.getTimestamp(ResultSetImpl.java:5943)
        at com.mysql.jdbc.ResultSetImpl.getObject(ResultSetImpl.java:4901)
        at com.zendesk.maxwell.bootstrap.SynchronousBootstrapper.setRowValues(SynchronousBootstrapper.java:221)
        at com.zendesk.maxwell.bootstrap.SynchronousBootstrapper.startBootstrap(SynchronousBootstrapper.java:58)
        at com.zendesk.maxwell.bootstrap.AsynchronousBootstrapper$1.run(AsynchronousBootstrapper.java:70)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NumberFormatException: ^MADMINISTRATOR^ZADMINISTRATOR - God Rights^\/jsp/security/systemMenu.jsp^W2016-06-13 05:56:59.000^MADMINISTRATOR^W2016-06-13 05:56:59.000^MADMINISTRATOR^A0^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
        at com.mysql.jdbc.StringUtils.getInt(StringUtils.java:756)
        at com.mysql.jdbc.ResultSetRow.getTimestampFast(ResultSetRow.java:1106)
        ... 8 more
13:09:30,459 INFO  TransportImpl - disconnected from dbsrs0879:3306
13:09:30,469 INFO  SchemaPosition - Storing final position: BinlogPosition[binarylog.042356:22106]
osheroff commented 8 years ago

what's the schema for the table you're trying to bootstrap? and what version of mysql?

The second error makes a certain degree of sense, maybe something up with different versions of mysql server / java client.

The first error is completely baffling. There should really be no way that that line triggers a lock-wait timeout, as it's just:

            List<String> cols = query.getFirst("SELECT @@global.binlog_checksum");
xmlking commented 8 years ago

Environment

innodb_version  5.6.24-72.2
version 5.6.24-72.2-25.11-log
version_comment Percona XtraDB Cluster binary (GPL) 5.6.24-25.11, Revision, wsrep_25.11
version_compile_machine x86_64
version_compile_os  Linux

Maxwell's config.properties

log_level=INFO
user=maxwell
password=xxxxxxxxxx
host=mysql_host
port=3306

producer=kafka
kafka.acks=1
kafka.bootstrap.servers=kafka_host1:9092,kafka_host2:9092,kafka_host3:9092
kafka_topic=maxwell
include_dbs=mydb

bootstrapper=async

Table 1 schema: (first error) has 80k records

CREATE TABLE zipcode
(
    zip CHAR(5) DEFAULT '' NOT NULL,
    city VARCHAR(50) DEFAULT '' NOT NULL,
    countyFIPS CHAR(5) DEFAULT '' NOT NULL,
    state CHAR(2),
    latitudeDeg DECIMAL(20,14),
    longitudeDeg DECIMAL(20,14),
    metroServiceArea CHAR(4),
    areaCode VARCHAR(16),
    stateName VARCHAR(64),
    zipType CHAR(1),
    timeZone VARCHAR(16),
    utc DECIMAL(3,1),
    dst CHAR(1),
    stateFIPS CHAR(2),
    countyName VARCHAR(64),
    cityType CHAR(1),
    changeDateTime TIMESTAMP(3) DEFAULT 'CURRENT_TIMESTAMP(3)' NOT NULL,
    changeUserID VARCHAR(20),
    createDateTime TIMESTAMP(3) DEFAULT '0000-00-00 00:00:00.000' NOT NULL,
    createUserID VARCHAR(20),
    updateVersion INT(10),
    CONSTRAINT `PRIMARY` PRIMARY KEY (zip, city, countyFIPS)
);
CREATE INDEX zipCodeCity ON zipcode (city);
CREATE INDEX zipCodeZip ON zipcode (zip);
CREATE INDEX zipCode_CD ON zipcode (changeDateTime);
CREATE INDEX zipcode_idx1 ON zipcode (city, state, zip);
CREATE INDEX zipCounty ON zipcode (countyFIPS);
CREATE INDEX zipMSA ON zipcode (metroServiceArea);
CREATE INDEX zipState ON zipcode (state);

Table 2 schema has 20 ecords

CREATE TABLE user_tab
(
    userGroupID VARCHAR(100) PRIMARY KEY NOT NULL,
    userGroupDesc VARCHAR(200),
    mainMenu VARCHAR(60),
    changeDateTime TIMESTAMP(3) DEFAULT 'CURRENT_TIMESTAMP(3)' NOT NULL,
    changeUserID VARCHAR(20),
    createDateTime TIMESTAMP(3) DEFAULT '0000-00-00 00:00:00.000' NOT NULL,
    createUserID VARCHAR(20),
    updateVersion INT(10)
);
CREATE INDEX user_tab_CD ON user_tab (changeDateTime);
xmlking commented 8 years ago

Anybody facing crash issues with bootstrap with Maxwell v1.1.1? it used to work fine in previous releases. CDC works fine but I cannot bootstrap tables that has timestamp :(

java.sql.SQLException: Cannot convert value '2016-05-26 08:06:40.000' from column 22 to TIMESTAMP.
osheroff commented 8 years ago

@xmlking Ok, I can repro your error if I install that precise binary. will see about a fix.

osheroff commented 8 years ago

https://github.com/zendesk/maxwell/pull/370 seems to fix it for me locally

osheroff commented 8 years ago

released in v1.1.2