maxwell和bireme处理时间问题

HashDataInc / bireme

Bireme is an incremental synchronization tool for the Greenplum / HashData data warehouse

https://hashdatainc.github.io/bireme/

Apache License 2.0

137 stars 53 forks source link

Closed shubifeng closed 6 years ago

shubifeng commented 6 years ago

当库表较多和kafka消息队列中消息数过于庞大时, 当maxwell.properties配置同步一个表时, bireme入库时间会很长.

wangzw commented 6 years ago

Please update your benchmark result and more details.

Usually it is because that the database cannot handle the deleting/loading quickly. Extend you database cluster and test again.

shubifeng commented 6 years ago

应该不是数据库集群问题.

所有的binlog解析到maxwell-topic, 要么是kafka队列读的慢, 因为有一部分数据还能快速更新, 有一部分表的数据很慢很慢.

没数据的时候也建立临时库? 为什么会没有自动删除?