Altinity / clickhouse-sink-connector

Replicate data from MySQL, Postgres and MongoDB to ClickHouse
https://www.altinity.com
Apache License 2.0
203 stars 48 forks source link

Add `driver.max_packet_size` to config.yml #562

Open subkanthi opened 2 months ago

subkanthi commented 2 months ago

Add configuration driver.max_packet_size to config.yml and document this variable in Troubleshooting.md.

aadant commented 2 months ago

Large events may lead to this message

ERROR io.debezium.pipeline.ErrorHandler - Producer failure
io.debezium.DebeziumException: log event entry exceeded max_allowed_packet; Increase max_allowed_packet on master; the first event 'binary.085408' at 388427867, the last event read from '/var/lib/mysql/logs/binary.085408' at 126, the last byte read from '/var/lib/mysql/logs/binary.085408' at 388427886. Error code: 1236; SQLSTATE: HY000.
    at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.wrap(MySqlStreamingChangeEventSource.java:1088)
    at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$ReaderThreadLifecycleListener.onCommunicationFailure(MySqlStreamingChangeEventSource.java:1133)
    at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1096)
    at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:648)
    at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:949)
    at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: com.github.shyiko.mysql.binlog.network.ServerException: log event entry exceeded max_allowed_packet; Increase max_allowed_packet on master; the first event 'binary.085408' at 388427867, the last event read from '/var/lib/mysql/logs/binary.085408' at 126, the last byte read from '/var/lib/mysql/logs/binary.085408' at 388427886.
    at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1060)
    ... 3 more

It is also recommended to set max_allowed_packet on the primary in addition to driver.max_packet_size in config.yml.