julien-duponchelle / python-mysql-replication

Pure Python Implementation of MySQL replication protocol build on top of PyMYSQL
2.33k stars 679 forks source link

Can't handle super long json #343

Open skyecool opened 3 years ago

skyecool commented 3 years ago

mysql> select max(length($json)) from json_data +--------------------+ | max(length($json)) | +--------------------+ | 42864 | +--------------------+ 1 row in set (0.24 sec)

Traceback (most recent call last): File "/root/binlog2sql_json/binlog2sql/binlog2sql.py", line 150, in binlog2sql.process_binlog() File "/root/binlog2sql_json/binlog2sql/binlog2sql.py", line 110, in process_binlog for row in binlog_event.rows: File "/usr/local/lib/python3.8/site-packages/pymysqlreplication/row_event.py", line 433, in rows self._fetch_rows() File "/usr/local/lib/python3.8/site-packages/pymysqlreplication/row_event.py", line 428, in _fetch_rows self.__rows.append(self._fetch_one_row()) File "/usr/local/lib/python3.8/site-packages/pymysqlreplication/row_event.py", line 519, in _fetch_one_row row["after_values"] = self._read_column_data(self.columns_present_bitmap2) File "/usr/local/lib/python3.8/site-packages/pymysqlreplication/row_event.py", line 181, in _read_column_data values[name] = self.packet.read_binary_json(column.length_size) File "/usr/local/lib/python3.8/site-packages/pymysqlreplication/packet.py", line 352, in read_binary_json return self.read_binary_json_type(t, length) File "/usr/local/lib/python3.8/site-packages/pymysqlreplication/packet.py", line 357, in read_binary_json_type return self.read_binary_json_object(length - 1, large) File "/usr/local/lib/python3.8/site-packages/pymysqlreplication/packet.py", line 440, in read_binary_json_object data = self.read_binary_json_type(t, length) File "/usr/local/lib/python3.8/site-packages/pymysqlreplication/packet.py", line 359, in read_binary_json_type return self.read_binary_json_array(length - 1, large) File "/usr/local/lib/python3.8/site-packages/pymysqlreplication/packet.py", line 454, in read_binary_json_array raise ValueError('Json length is larger than packet length') ValueError: Json length is larger than packet length

sean-k1 commented 1 year ago

@julien-duponchelle @skyecool this Issue resolved in https://github.com/julien-duponchelle/python-mysql-replication/pull/481