fix bug signedness order change to All Column Order

[sean-k1]

I was implementing optional_metadata field in table_map_event for python-mysql-replication.

Changes

• Add convertColumnOrder function

Reason

optional metadata type SIGNEDNESS case The order of indices in the Inputstream corresponds to the order of numeric columns So we need to map the index to all columns index(include non numeric type columns)

Examples

SIGNEDNESS bit map now result is signedness={0, 2}, but we expect result should be signedness={0, 3}, Because test3 table column col1 and col4 are unsigned

AS-IS

• sql

  create table test3( col1 int unsigned , col2 int, col3 varchar(30), col4 int unsigned)
  insert into test3 values (1,2,3,4)

• application log

  Event{header=EventHeaderV4{timestamp=1692014933000, eventType=QUERY, serverId=1, headerLength=19, dataLength=193, nextPosition=2911, flags=0}, data=QueryEventData{threadId=1396, executionTime=0, errorCode=0, database='pymysqlreplication_test', sql='create table test3( col1 int unsigned , col2 int, col3 varchar(30), col4 int unsigned)'}}
  Event{header=EventHeaderV4{timestamp=1692014936000, eventType=ANONYMOUS_GTID, serverId=1, headerLength=19, dataLength=60, nextPosition=2990, flags=0}, data=null}
  Event{header=EventHeaderV4{timestamp=1692014936000, eventType=QUERY, serverId=1, headerLength=19, dataLength=75, nextPosition=3084, flags=8}, data=QueryEventData{threadId=1396, executionTime=0, errorCode=0, database='pymysqlreplication_test', sql='BEGIN'}}
  Event{header=EventHeaderV4{timestamp=1692014936000, eventType=TABLE_MAP, serverId=1, headerLength=19, dataLength=86, nextPosition=3189, flags=0}, data=TableMapEventData{tableId=4555, database='pymysqlreplication_test', table='test3', columnTypes=3, 3, 15, 3, columnMetadata=0, 0, 120, 0, columnNullability={0, 1, 2, 3}, eventMetadata=TableMapEventMetadata{signedness={0, 2}, defaultCharset=255, charsetCollations=null, columnCharsets=null, columnNames=col1, col2, col3, col4, setStrValues=null, enumStrValues=null, geometryTypes=null, simplePrimaryKeys=null, primaryKeysWithPrefix=null, enumAndSetDefaultCharset=null, enumAndSetColumnCharsets=null,visibility={0, 1, 2, 3}}}}

TO-BE

• sql

  create table test3( col1 int unsigned , col2 int, col3 varchar(30), col4 int unsigned)
  insert into test3 values (1,2,3,4)

• application log

  Event{header=EventHeaderV4{timestamp=1692014933000, eventType=QUERY, serverId=1, headerLength=19, dataLength=193, nextPosition=2911, flags=0}, data=QueryEventData{threadId=1396, executionTime=0, errorCode=0, database='pymysqlreplication_test', sql='create table test3( col1 int unsigned , col2 int, col3 varchar(30), col4 int unsigned)'}}
  Event{header=EventHeaderV4{timestamp=1692014936000, eventType=ANONYMOUS_GTID, serverId=1, headerLength=19, dataLength=60, nextPosition=2990, flags=0}, data=null}
  Event{header=EventHeaderV4{timestamp=1692014936000, eventType=QUERY, serverId=1, headerLength=19, dataLength=75, nextPosition=3084, flags=8}, data=QueryEventData{threadId=1396, executionTime=0, errorCode=0, database='pymysqlreplication_test', sql='BEGIN'}}
  Event{header=EventHeaderV4{timestamp=1692014936000, eventType=TABLE_MAP, serverId=1, headerLength=19, dataLength=86, nextPosition=3189, flags=0}, data=TableMapEventData{tableId=4555, database='pymysqlreplication_test', table='test3', columnTypes=3, 3, 15, 3, columnMetadata=0, 0, 120, 0, columnNullability={0, 1, 2, 3}, eventMetadata=TableMapEventMetadata{signedness={0, 3}, defaultCharset=255, charsetCollations=null, columnCharsets=null, columnNames=col1, col2, col3, col4, setStrValues=null, enumStrValues=null, geometryTypes=null, simplePrimaryKeys=null, primaryKeysWithPrefix=null, enumAndSetDefaultCharset=null, enumAndSetColumnCharsets=null,visibility={0, 1, 2, 3}}}}

[dongwook-chan]

PTAL, @osheroff I desperately need this feature. Optional meta data in table map events holds invaluable data and users will benefit so much from correctly parsing it. I can't wait this PR to be merged!

[osheroff]

hi, sorry for the delay, can you please write an integration test that shows the issue clearly? Your patch looks ok -- although it'd be better if we could avoid another for-all-columns loop and the ArrayList allocation -- but I'd like to be able to verify this against a running server.

thnx

[sean-k1]

@osheroff hello

If we add a function to the TableMapEventMetadataDeserializer class that takes ColumnType information as a parameter and that determines if it's numeric, we can eliminate the part where we allocate an ArrayList. But this codes.. It seems like this may not be the best structure for writing test code. If this approach is better, I'll modify code

and i added integration Test too

[sean-k1]

@osheroff Can you check my PR?

[dongwook-chan]

@osheroff PTAL

[sean-k1]

@osheroff It takes a month 🥲

[dongwook-chan]

@osheroff I know you're busy but I think it's high time you appoint additional maintainer. I think @sean-k1 is one of the most competent candidates. He has conducted a comprehensive walkthrough of the project and understands every aspect of it. And on top of everything, he is a maintainer and active contributor of python-mysql-replication. He is an expert in MySQL replication.