go-mysql-org / go-mysql

a powerful mysql toolset with Go
MIT License
4.6k stars 985 forks source link

add table cache for table not exist #437

Open lintanghui opened 5 years ago

lintanghui commented 5 years ago

The problem: I made a ddl on a online table, in case of table lock, I use mysql gh-ost tool. when using gh-ost tool, old data would be write into a temp table and made a lot row events. after ddl finished. temp table would be deleted.

create temp table
write row event 
wrte row event
a lot of row event ...
some other event 
a log of temp table event 
delete temp table

If I run canal from pos of some other event (maybe restart). i would get a event of temp table, but can not get schema of temp table but return table not exist since temp table had been deleted. there ere a lot of row event of temp table, every time i parse a event i send a req of IsTableExist to mysql but get not exist each time. in mysql, show processlist ,binlog dump is block since canal were always doing get table schema image

how to slove this: in my opinion. we should add a table no exist cache for not exist table. when get a row event of table which not exist in cache, skip it instead of get table schema each time. if it's ok, i will make a pr. @siddontang

siddontang commented 5 years ago

@lintanghui

yes, now we haven't supported some DDLs well. seem https://github.com/siddontang/go-mysql/pull/358 can solve your problem but we have not advanced this.

PTAL @GregoryIan

lintanghui commented 5 years ago

same with #427. once canal start from a middle pos. we may get a wrong table schema or can not get table schema once table been changed or been deleted. since we get scheme by GetTable. GetTable return current table scheme but not when binlog happend.