OpenAtomFoundation / pika

Pika is a Redis-Compatible database developed by Qihoo's infrastructure team.
BSD 3-Clause "New" or "Revised" License
5.76k stars 1.19k forks source link

incr caculate error with ttl #2760

Open chejinge opened 1 week ago

chejinge commented 1 week ago

Is this a regression?

Yes

Description

1 主节点连续执行:setex key1 10 value1, (value1=100), incr key1。主节点这边的key1会变成101, 且10s后就会过期 2 假设从节点30s后才收到binlog,分别是一条pksetat key1 ....和incur key1。从节点的这条pksetat采用了主的过期时间戳,所以key1一写进去就是过期的,相当于没写这个key1。尔后执行incur key1, 就制造出了一个永远不会过期的key1。这就导致了主从不一致了。

Please provide a link to a minimal reproduction of the bug

No response

Screenshots or videos

解决办法: 和鸽姐讨论了一下,鸽姐提了一个很好的方案,而且不影响兼容: incr也写binlog的时候也用pksetat来写就行,具体地: 执行incr的时候,如果发现目标是个有ttl且还没过期的key, 将其ttl取出来,构造一个pksetat Binlog发给从消费。 举例: 1 主节点执行: 1.1 setex key1 10 1 (10s后过期,value为1) 1.2 incur key1 2 在生成incr的Binlog时构造一个pksetat Binlog,从节点Apply这个Binlog的时候相当于set key1 11 (并且给了一个和主那边一样的过期时间戳)

这样的话至少incr引起的这个问题解决了。

总的来说工作量会比较大。

1 有问题的命令不止incr,需要 1.2 梳理哪些结构支持过期时间

1.3 有哪些命令有incr这种性质(会给你无中生有)

2 刚刚发现,setex的binlog重写为了pksetat命令,但是setnx这种一样性质的命令却没有重写Binlog,这块也需要检查,所有类似性质的命令都需要check一下

Please provide the version you discovered this bug in (check about page for version information)

No response

Anything else?

No response

chejinge commented 1 day ago

详细梳理及方案

String 类型的:

SET key value: 如果键不存在,则创建该键。

INCR key: 如果键不存在,则创建该键,初始值为 0,然后递增。

APPEND key value: 如果键不存在,则创建该键并设置值为 value。

Hash数据类型的:

HSET key field value: 如果键不存在,则创建该键,并添加哈希字段和值。

HINCRBY key field increment: 如果键不存在,则创建该键,初始哈希字段值为 0,然后进行递增操作。

List数据类型的:

LPUSH key value [value ...]: 如果键不存在,则创建一个新列表,并将 value 插入到列表头部。(先不修复)

RPUSH key value [value ...]: 如果键不存在,则创建一个新列表,并将 value 插入到列表尾部。(先不修复)

Set数据类型的:

SADD key member [member ...]: 如果键不存在,则创建一个新集合,并添加成员。(先不修复)

SMOVE source destination member: 如果目标键不存在,则创建一个新集合,并将成员移动到这个新集合中。(先不修复)

Sorted Set(有序集合)数据类型:

ZADD key score member [score member ...]: 如果键不存在,则创建一个新有序集合,并添加成员和对应的分值。(先不修复)

上述这些命令会在操作过程中检测到不存在的key时候进行自动创建,从而实现无中生有

需要修改的地方:

sting命令

1.Incr命令在从节点写binlog的时候同时需要将ttl传过去

2.Apend命令在从节点写binlog的时候将该key的ttl传过去

Hash 命令

1.HINCRBY key field increme 命令在给从节点写binlog时将ttl携带上

List数据类型

LPUSH key value [value ...] 在传输的过程中将该key的ttl携带上

目前先只修改 incr 和hincr命令即可

Issues-translate-bot commented 1 day ago

Bot detected the issue body's language is not English, translate it automatically.


Detailed review and plan

String type:

SET key value: If the key does not exist, create the key.

INCR key: If the key does not exist, create the key with an initial value of 0 and then increment.

APPEND key value: If the key does not exist, create the key and set the value to value.

Hash data type:

HSET key field value: If the key does not exist, create the key and add the hash field and value.

HINCRBY key field increment: If the key does not exist, create the key with an initial hash field value of 0 and then increment it.

List data type:

LPUSH key value [value ...]: If the key does not exist, create a new list and insert value at the head of the list. (don’t fix it yet)

RPUSH key value [value ...]: If the key does not exist, create a new list and insert value at the end of the list. (don’t fix it yet)

Set data type:

SADD key member [member ...]: If the key does not exist, create a new collection and add the member. (don’t fix it yet)

SMOVE source destination member: If the target key does not exist, create a new collection and move the member to this new collection. (don’t fix it yet)

Sorted Set (ordered set) data type:

ZADD key score member [score member ...]: If the key does not exist, create a new ordered set and add members and corresponding scores. (don’t fix it yet)

The above commands will automatically create them when a non-existent key is detected during the operation, thus creating something out of nothing.

What needs to be modified:

sting command

  1. The Incr command needs to pass the ttl when writing the binlog from the node.

  2. The Apend command passes the ttl of the key when writing binlog from the slave node.

Hash command

  1. The HINCRBY key field increme command carries ttl when writing binlog to the slave node.

List data type

LPUSH key value [value ...] carries the ttl of the key during the transmission process

For now, just modify the incr and hincr commands.