Decode Exception due to Inaccurate Length from self.query_length Leading to Byte Sequence Truncation

[zj5220924]

Version

Please specify the versions you are using. Exact version numbers are preferred.

• Pymyrepl (1.1.0):
• OS (Centos 7):
• Pymyrepl (1.0.6):
• Database and version (Remove unnecessary options):
  • MySQL: 8.0.32

System Variables

List relevant system variables using the query SHOW VARIABLES LIKE '<variable_name_here>';

• log_bin: YES
• binlog_format: ROW
• binlog_row_image: FULL
• enforce_gtid_consistency: True
• gtid_mode: YES

If you are using MySQL 8.0.14 or later, please also provide the following variables:

• binlog_row_metadata: FULL
• binlog_row_value_options: Null

error massage: Traceback (most recent call last): File "/ROOT/tmp/my2sql/my2sql.py", line 144, in binlog2sql.process_binlog() File "/ROOT/tmp/my2sql/my2sql.py", line 72, in process_binlog for binlog_event in stream: File "/usr/local/python3.11/lib/python3.11/site-packages/pymysqlreplication/binlogstream.py", line 617, in fetchone binlog_event = BinLogPacketWrapper( ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/python3.11/lib/python3.11/site-packages/pymysqlreplication/packet.py", line 115, in init self.event = event_class( ^^^^^^^^^^^^ File "/usr/local/python3.11/lib/python3.11/site-packages/pymysqlreplication/event.py", line 885, in init self.query = self.packet.read(self.query_length).decode("utf-8") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 205-206: unexpected end of data

cat /....../pymysqlreplication/event.py ...... def init(self, from_packet, event_size, table_map, ctl_connection, kwargs): super(RowsQueryLogEvent, self).init( from_packet, event_size, table_map, ctl_connection, kwargs ) self.query_length = self.packet.read_uint8() self.query = self.packet.read(self.query_length).decode("utf-8") print(self.query_length) print(self.query) ......

After testing, it was found that the length obtained from self.query_length is inaccurate, causing the byte sequence read by self.packet.read to be truncated, leading to decoding exceptions.

[dongwook-chan]

@zj5220924 Thank you for reporting this issue.

the length obtained from self.query_length is inaccurate

You're indeed right. The length must be ignored without being used in parsing the query. code from msql-server suggests so.

Other clients in different languages implement the RowsQueryEvent the way mysql-server does.

• MysqlCdc (c#)
• mysql-binlog-connector-java (java)

I'll have this fixed ASAP.

[dongwook-chan]

@zj5220924 Could you please clarify steps to reproduce so that I could add test cases? Please provide at least 1 query.

[zj5220924]

@dongwook-chan 22159.log This is my test data. thx~