Shopify / ghostferry

The swiss army knife of live data migrations
https://shopify.github.io/ghostferry
MIT License
748 stars 70 forks source link

unsigned mediumint value through binlog streamer wrongfully parsed #299

Open HemeraOne opened 3 years ago

HemeraOne commented 3 years ago
table:
CREATE TABLE `customer` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `geo_city` mediumint(8) unsigned DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=7 DEFAULT CHARSET=utf8mb4

queries:
insert into customer values (null, 1);
update customer set geo_city = 8592090 where id = 1; //crashes

error:
ERRO[0009] fatal error detected                          errfrom=binlog_writer error="exec query at pos (mysql-bin.049833, 1049512401) -> (mysql-bin.049833, 1049512401) (163 bytes): Error 1264: Out of range value for column 'geo_city' at row 1" tag=error_handler

ghostferry query:
/*application:ghostferry*/ BEGIN;\n/*application:ghostferry*/ UPDATE `martijntest`.`customer` SET `id`=6,`geo_city`=4286782170 WHERE `id`=6 AND `geo_city`=1;\nCOMMIT

so somehow the unsigned value which lies outside of the signed value gets wrongfully converted: 8592090 -> 4286782170

value parsed from binlog: -8185126

should be converted to int24 which seems to be not a thing in golang

Manan007224 commented 3 years ago

There is an issue in the upstream library go-mysql regarding unsigned integer issues - https://github.com/go-mysql-org/go-mysql/issues/206

HemeraOne commented 3 years ago

as quoted from the library maintainer in the issue:

binlog event doesn't contain unsigned type, you should convert outside with your own schema.

Which is done for uInt8,uInt16,uInt32 and uInt64 but there is no native uint24 in go