TraceNature / redissyncer-server

RedisSyncer是一个多任务的redis数据同步工具,可灵活的满足Redis间的数据同步、迁移需求; redissyncer is a redis synchronization tool, used in redis single instance and cluster synchronization
https://github.com/TraceNature/redissyncer-server
Apache License 2.0
639 stars 104 forks source link

同步时遇到大key报错,任务退出 #91

Open pyb0815 opened 1 year ago

pyb0815 commented 1 year ago

被抛弃key数量到达阈值[1],exception reason[For input string: "QUEUED"->KeyValueCommandSyncEventFilter->KeyValueRdbSyncEventFilter->KeyValueEventDBMappingFilter->KeyValueDataAnalysisFilter->KeyValueTimeCalculationFilter]

同步时遇到大key报错,任务退出。日志信息如上

pingxingshikong commented 1 year ago

redis是哪个版本的,能否给个测试数据样本?

lyulong00 commented 4 months ago

redis:6.2.6,我们也出现了

2024-06-18T11:10:41.489884829Z 2024-06-18 11:10:41.489|[ WARN ]|[6CED8FCB5C2B461ABF34F645BB27B5BB: threadPoolTaskExecutor-10] -> KeyValueRdbSyncEventFilter#run:103 | taskId为[6CED8FCB5C2B461ABF34F645BB27B5BB]的任务全量同步结束..进入增量同步模式 time:[158:s] 2024-06-18T11:10:41.500083652Z 2024-06-18 11:10:41.499|[ INFO ]|[6CED8FCB5C2B461ABF34F645BB27B5BB: threadPoolTaskExecutor-10] -> RedisSocketReplicator#heartbeat:336 | heartbeat started. 2024-06-18T11:10:41.507800374Z 2024-06-18 11:10:41.507|[ WARN ]|[6CED8FCB5C2B461ABF34F645BB27B5BB: threadPoolTaskExecutor-10] -> KeyValueCommandSyncEventFilter#run:58 | taskId为[6CED8FCB5C2B461ABF34F645BB27B5BB]的任务增量同步开始.. 2024-06-18T11:10:41.562375280Z 2024-06-18 11:10:41.562|[ WARN ]|[6CED8FCB5C2B461ABF34F645BB27B5BB: threadPoolTaskExecutor-10] -> TaskErrorUtils#brokenStatusAndLog:54 | 任务Id【6CED8FCB5C2B461ABF34F645BB27B5BB】任务启动/运行异常停止 ,Class【class syncer.syncerservice.sync.SendCommandWithOutQueue】,异常原因【被抛弃key数量到达阈值[1],exception reason[For input string: "QUEUED"->KeyValueCommandSyncEventFilter->KeyValueRdbSyncEventFilter->KeyValueEventDBMappingFilter->KeyValueDataAnalysisFilter->KeyValueTimeCalculationFilter]】 2024-06-18T11:10:41.562508621Z 2024-06-18 11:10:41.562|[ERROR ]|[6CED8FCB5C2B461ABF34F645BB27B5BB: threadPoolTaskExecutor-10] -> SendCommandWithOutQueue#run:153 | [6CED8FCB5C2B461ABF34F645BB27B5BB]抛弃key:setmua:location:465873:1009587 ,class:[class syncer.syncerplusredis.cmd.impl.DefaultCommand]:原因[For input string: "QUEUED"->KeyValueCommandSyncEventFilter->KeyValueRdbSyncEventFilter->KeyValueEventDBMappingFilter->KeyValueDataAnalysisFilter->KeyValueTimeCalculationFilter] 2024-06-18T11:10:41.562968115Z syncer.syncerservice.exception.FilterNodeException: For input string: "QUEUED"->KeyValueCommandSyncEventFilter->KeyValueRdbSyncEventFilter->KeyValueEventDBMappingFilter->KeyValueDataAnalysisFilter->KeyValueTimeCalculationFilter 2024-06-18T11:10:41.562992311Z at syncer.syncerservice.filter.KeyValueTimeCalculationFilter.run(KeyValueTimeCalculationFilter.java:68) 2024-06-18T11:10:41.562998777Z at syncer.syncerservice.filter.KeyValueRunFilterChain.run(KeyValueRunFilterChain.java:42) 2024-06-18T11:10:41.563004338Z at syncer.syncerservice.sync.SendCommandWithOutQueue.run(SendCommandWithOutQueue.java:62) 2024-06-18T11:10:41.563009315Z at syncer.syncerservice.sync.RedisDataSyncTransmissionTask$1.onEvent(RedisDataSyncTransmissionTask.java:184) 2024-06-18T11:10:41.563014474Z at syncer.syncerplusredis.extend.replicator.listener.ValueDumpIterableEventListener.onEvent(ValueDumpIterableEventListener.java:53) 2024-06-18T11:10:41.563019721Z at syncer.syncerplusredis.replicator.AbstractReplicatorListener.doEventListener(AbstractReplicatorListener.java:92) 2024-06-18T11:10:41.563025424Z at syncer.syncerplusredis.replicator.AbstractReplicator.submitEvent(AbstractReplicator.java:105) 2024-06-18T11:10:41.563030947Z at syncer.syncerplusredis.RedisSocketReplicator$RedisSocketReplicatorRetrier.open(RedisSocketReplicator.java:532) 2024-06-18T11:10:41.563036605Z at syncer.syncerplusredis.replicator.AbstractReplicatorRetrier.retry(AbstractReplicatorRetrier.java:103) 2024-06-18T11:10:41.563041934Z at syncer.syncerplusredis.RedisSocketReplicator.open(RedisSocketReplicator.java:135) 2024-06-18T11:10:41.563046785Z at syncer.syncerplusredis.replicator.RedisReplicator.open(RedisReplicator.java:316) 2024-06-18T11:10:41.563050899Z at syncer.syncerservice.sync.RedisDataSyncTransmissionTask.run(RedisDataSyncTransmissionTask.java:194) 2024-06-18T11:10:41.563055585Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 2024-06-18T11:10:41.563065049Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 2024-06-18T11:10:41.563069482Z at java.lang.Thread.run(Thread.java:748) 2024-06-18T11:10:41.563073602Z 线程正准备关闭...6CED8FCB5C2B461ABF34F645BB27B5BB: threadPoolTaskExecutor-10