OpenAtomFoundation / pika

Pika is a Redis-Compatible database developed by Qihoo's infrastructure team.
BSD 3-Clause "New" or "Revised" License
5.83k stars 1.19k forks source link

Pika 4.0 使用 codis-fe 迁移 slot 时偶尔有报错 #2870

Open zenxme opened 1 month ago

zenxme commented 1 month ago

Is this a regression?

Yes

Description

一边写入数据, 一边迁移, 就会有概率出现这个问题

codis-fe 报错截图: image

客户端收到的错误: redis.exceptions.ResponseError: handle request, bad slotsmgrt-exec-wrapper resp: type(array[0]) = <int>, len(array[0].value) = 2

pika-server 迁移源节点日志: pika_migrate_thread.cc:662] PikaMigrateThread::ReqMigrateOne Slot : 886 is not the migrating slot:-1

pika-server 目标节点日志:

W20240816 10:16:59.247891   349 pika_migrate_thread.cc:662] PikaMigrateThread::ReqMigrateOne Slot : 886 is not the migrating slot:25
W20240816 10:16:59.247941   341 pika_slot_command.cc:1250] Slot batch migrate keys error
I20240816 10:16:59.440676   419 pika_migrate_thread.cc:943] PikaMigrateThread::ThreadMain send_num:367 response_num:367
I20240816 10:16:59.440834   419 pika_migrate_thread.cc:967] PikaMigrateThread::ThreadMain slot_size:367 moved_num:367
I20240816 10:16:59.440999   419 pika_migrate_thread.cc:755] PikaMigrateThread::DestroyThread
I20240816 10:16:59.441212   420 pika_migrate_thread.cc:519] PikaParseSendThread::ThreadMain :140736425092800 exit !!!

Please provide a link to a minimal reproduction of the bug

No response

Screenshots or videos

No response

Please provide the version you discovered this bug in (check about page for version information)

No response

Anything else?

No response

Issues-translate-bot commented 1 month ago

Bot detected the issue body's language is not English, translate it automatically.


Title: Pika 4.0 occasionally reports errors when using codis-fe to migrate slots.

chejinge commented 1 week ago

迁移数据的时候是会对slot进行加锁的,加锁的时候这个slot读写是阻塞的,所以读写请求都进不去

Issues-translate-bot commented 1 week ago

Bot detected the issue body's language is not English, translate it automatically.


When migrating data, the slot will be locked. When locked, reading and writing in this slot are blocked, so read and write requests cannot enter.

zenxme commented 1 week ago

迁移数据的时候是会对slot进行加锁的,加锁的时候这个slot读写是阻塞的,所以读写请求都进不去

如果要加锁的话怎么实现在不影响在线业务的情况下,动态迁移slot ? codis 是可以的,pika 不支持吗?

Issues-translate-bot commented 1 week ago

Bot detected the issue body's language is not English, translate it automatically.


When migrating data, the slot will be locked. When locked, the slot is blocked for reading and writing, so read and write requests cannot enter.

If locking is required, how can the slot be dynamically migrated without affecting the online business? Codis is ok, but pika doesn’t support it?

chejinge commented 5 days ago

迁移数据的时候是会对slot进行加锁的,加锁的时候这个slot读写是阻塞的,所以读写请求都进不去

如果要加锁的话怎么实现在不影响在线业务的情况下,动态迁移slot ? codis 是可以的,pika 不支持吗?

尽可能的把数据打散,把slot个数多一些 会在迁移的过程中对于用户的影响尽可能的少

Issues-translate-bot commented 5 days ago

Bot detected the issue body's language is not English, translate it automatically.


When migrating data, the slot will be locked. When locked, the slot is blocked for reading and writing, so read and write requests cannot enter.

If locking is required, how can the slot be dynamically migrated without affecting the online business? Codis is ok, but pika doesn’t support it?

Scattering the data as much as possible and increasing the number of slots will have as little impact on users as possible during the migration process.