search

OpenAtomFoundation / pika

Pika is a Redis-Compatible database developed by Qihoo's infrastructure team.
BSD 3-Clause "New" or "Revised" License
5.83k stars 1.19k forks source link

基于codis搭建的pika集群，pipeline导入数据时proxy报错，无法导入 #2824

Open LinzWang opened 1 month ago

LinzWang commented 1 month ago

Is this a regression?

Yes

Description

本地通过codis连接两个pika服务，目前测试时发现通过pipeline写入数据时，100个命令拼接的pipeline可以正常写入，1000个命令时proxy，初次可能成功，第二次一定会报错，且链接超时后才会退出。错误多次发生后会导致pika异常退出

codis-proxy错误信息如下：

2024/07/24 18:00:05 session.go:270: [ERROR] 2024-07-24 17:58:48 remote:127.0.0.1:59260, start_time(us):1721815128177972, duration(us): [6, 11517, 77295887], 77307410, tasksLen:997, command:[SET 998 998].
[stack]: 
    3   /Users/edy/temp/pika-4.0.0/codis/pkg/proxy/session.go:270
            pika/codis/v2/pkg/proxy.(*Session).loopWriter.func2
    2   /Users/edy/temp/pika-4.0.0/codis/pkg/proxy/request.go:156
            pika/codis/v2/pkg/proxy.(*RequestChan).PopFrontAll
    1   /Users/edy/temp/pika-4.0.0/codis/pkg/proxy/session.go:226
            pika/codis/v2/pkg/proxy.(*Session).loopWriter
    0   /Users/edy/temp/pika-4.0.0/codis/pkg/proxy/session.go:148
            pika/codis/v2/pkg/proxy.(*Session).Start.func1.3
        ... ...
2024/07/24 18:00:05 session.go:270: [ERROR] 2024-07-24 17:58:48 remote:127.0.0.1:59260, start_time(us):1721815128177972, duration(us): [15, 11508, 77295927], 77307450, tasksLen:998, command:[SET 999 999].
[stack]: 
    3   /Users/edy/temp/pika-4.0.0/codis/pkg/proxy/session.go:270
            pika/codis/v2/pkg/proxy.(*Session).loopWriter.func2
    2   /Users/edy/temp/pika-4.0.0/codis/pkg/proxy/request.go:156
            pika/codis/v2/pkg/proxy.(*RequestChan).PopFrontAll
    1   /Users/edy/temp/pika-4.0.0/codis/pkg/proxy/session.go:226
            pika/codis/v2/pkg/proxy.(*Session).loopWriter
    0   /Users/edy/temp/pika-4.0.0/codis/pkg/proxy/session.go:148
            pika/codis/v2/pkg/proxy.(*Session).Start.func1.3
        ... ...
2024/07/24 18:00:05 session.go:93: [INFO] session [0xc080b83180] closed: {"ops":1002,"create":1721815128,"lastop":1721815128,"remote":"127.0.0.1:59260"}, error: EOF
2024/07/24 18:00:10 backend.go:335: [WARN] backend conn [0xc0000dac60] to 127.0.0.1:9221, db-0 writer-[0] exit
[error]: backend conn failure, set tcp 127.0.0.1:58987: use of closed network connection
2024/07/24 18:00:10 backend.go:282: [WARN] backend conn [0xc0000dac60] to 127.0.0.1:9221, db-0 reader-[0] exit
[error]: backend conn failure, read tcp 127.0.0.1:58987->127.0.0.1:9221: read: connection reset by peer
2024/07/24 18:00:10 backend.go:261: [WARN] backend conn [0xc0000dac60] to 127.0.0.1:9221, db-0 round-[1]

pika 错误信息如下：

pika(80030,0x70000c74b000) malloc: *** error for object 0x60000106bfc0: pointer being freed was not allocated
pika(80030,0x70000c74b000) malloc: *** set a breakpoint in malloc_error_break to debug

pika配置文件只修改了thead-num 为 2，proxy配置如下


##################################################
#                                                #
#                  Codis-Proxy                   #
#                                                #
##################################################

# Set Codis Product Name/Auth.
product_name = "codis-demo"
product_auth = ""

# Set auth for client session
#   1. product_auth is used for auth validation among codis-dashboard,
#      codis-proxy and codis-server.
#   2. session_auth is different from product_auth, it requires clients
#      to issue AUTH <PASSWORD> before processing any other commands.
session_auth = ""

# Set bind address for admin(rpc), tcp only.
admin_addr = "127.0.0.1:11080"

# Set bind address for proxy, proto_type can be "tcp", "tcp4", "tcp6", "unix" or "unixpacket".
proto_type = "tcp4"
proxy_addr = "127.0.0.1:19000"

# Set jodis address & session timeout
#   1. jodis_name is short for jodis_coordinator_name, only accept "zookeeper" & "etcd".
#   2. jodis_addr is short for jodis_coordinator_addr
#   3. jodis_auth is short for jodis_coordinator_auth, for zookeeper/etcd, "user:password" is accepted.
#   4. proxy will be registered as node:
#        if jodis_compatible = true (not suggested):
#          /zk/codis/db_{PRODUCT_NAME}/proxy-{HASHID} (compatible with Codis2.0)
#        or else
#          /jodis/{PRODUCT_NAME}/proxy-{HASHID}
jodis_name = ""
jodis_addr = ""
jodis_auth = ""
jodis_timeout = "20s"
jodis_compatible = false

# Set datacenter of proxy.
proxy_datacenter = ""

# Set max number of alive sessions.
proxy_max_clients = 5000

# Set max offheap memory size. (0 to disable)
proxy_max_offheap_size = "4gb"

# Set heap placeholder to reduce GC frequency.
proxy_heap_placeholder = "2gb"

# Proxy will ping backend redis (and clear 'MASTERDOWN' state) in a predefined interval. (0 to disable)
backend_ping_period = "5s"

# Set backend recv buffer size & timeout.
backend_recv_bufsize = "128kb"
backend_recv_timeout = "3600s"

# Set backend send buffer & timeout.
backend_send_bufsize = "128kb"
backend_send_timeout = "3600s"

# Set backend pipeline buffer size.
backend_max_pipeline = 20480

# Set backend never read replica groups, default is false
backend_primary_only = false

# Set backend parallel connections per server
backend_primary_parallel = 2
backend_replica_parallel = 2
# Set quick backend parallel connections per server
backend_primary_quick = 1
backend_replica_quick = 1

# Set slot num
max_slot_num = 2

# Set backend tcp keepalive period. (0 to disable)
backend_keepalive_period = "75s"

# Set number of databases of backend.
backend_number_databases = 1

# If there is no request from client for a long time, the connection will be closed. (0 to disable)
# Set session recv buffer size & timeout.
session_recv_bufsize = "128kb"
session_recv_timeout = "30m"

# Set session send buffer size & timeout.
session_send_bufsize = "128kb"
session_send_timeout = "30m"

# Make sure this is higher than the max number of requests for each pipeline request, or your client may be blocked.
# Set session pipeline buffer size.
session_max_pipeline = 100000

# Set session tcp keepalive period. (0 to disable)
session_keepalive_period = "75s"

# Set session to be sensitive to failures. Default is false, instead of closing socket, proxy will send an error response to client.
session_break_on_failure = false

# Slowlog-log-slower-than(us), from receive command to send response, 0 is allways print slow log
slowlog_log_slower_than = 100000

# quick command list
quick_cmd_list = ""
# slow command list
slow_cmd_list = ""

# Set metrics server (such as http://localhost:28000), proxy will report json formatted metrics to specified server in a predefined period.
metrics_report_server = ""
metrics_report_period = "1s"

# Set influxdb server (such as http://localhost:8086), proxy will report metrics to influxdb.
metrics_report_influxdb_server = ""
metrics_report_influxdb_period = "1s"
metrics_report_influxdb_username = ""
metrics_report_influxdb_password = ""
metrics_report_influxdb_database = ""

# Set statsd server (such as localhost:8125), proxy will report metrics to statsd.
metrics_report_statsd_server = ""
metrics_report_statsd_period = "1s"
metrics_report_statsd_prefix = ""

# Maximum delay statistical time interval.(This value must be greater than 0.)
max_delay_refresh_time_interval = "15s"

python 测试脚本：

import redis
import time
from threading import Thread
from concurrent.futures import ProcessPoolExecutor
pool = redis.ConnectionPool(host='127.0.0.1', port=19000)
r = redis.Redis(connection_pool=pool)
def try_pipeline(count):

    start = time.time()
    with r.pipeline(transaction=False) as pipe:
        for i in range(1000):
            #value = random.choice(string.ascii_letters).zfill(100)
            pipe.set(i+count*100, i)
        pipe.execute()
    print(f'pipeline: {time.time() - start}')
if __name__ == '__main__':
    try_pipeline(0)

Please provide a link to a minimal reproduction of the bug

No response

Screenshots or videos

No response

Please provide the version you discovered this bug in (check about page for version information)

Pika 版本： 3.5.3 
Codis版本： 根据4.0.0中代码构建
OS版本： macOS  14.2.1

Anything else?

No response

Issues-translate-bot commented 1 month ago

Bot detected the issue body's language is not English, translate it automatically.

Title: In the pika cluster built based on codis, when the pipeline imports data, the proxy reports an error and cannot be imported.

LinzWang commented 1 month ago

ubuntu中可以写入，但proxy日志有大量的错误，检查pika中数据正常写入。

image

错误信息和mac中的错误类似

2024/07/25 18:08:59 session.go:270: [ERROR] 2024-07-25 18:08:57 remote:127.0.0.1:58964, start_time(us):1721902137218354, duration(us): [90, 291263, 1695292], 1986646, tasksLen:8458, command:[HSET prop-user000000000049499 list00000000000000048true].
[stack]: 
    3   /pika/codis/pkg/proxy/session.go:270
            pika/codis/v2/pkg/proxy.(*Session).loopWriter.func2
    2   /pika/codis/pkg/proxy/request.go:156
            pika/codis/v2/pkg/proxy.(*RequestChan).PopFrontAll
    1   /pika/codis/pkg/proxy/session.go:226
            pika/codis/v2/pkg/proxy.(*Session).loopWriter
    0   /pika/codis/pkg/proxy/session.go:148
            pika/codis/v2/pkg/proxy.(*Session).Start.func1.3
        ... ...
2024/07/25 18:08:59 session.go:270: [ERROR] 2024-07-25 18:08:57 remote:127.0.0.1:58964, start_time(us):1721902137218357, duration(us): [90, 291260, 1695330], 1986682, tasksLen:8459, command:[HSET prop-user000000000049499 list00000000000000049true].
[stack]: 
    3   /pika/codis/pkg/proxy/session.go:270
            pika/codis/v2/pkg/proxy.(*Session).loopWriter.func2
    2   /pika/codis/pkg/proxy/request.go:156
            pika/codis/v2/pkg/proxy.(*RequestChan).PopFrontAll
    1   /pika/codis/pkg/proxy/session.go:226
            pika/codis/v2/pkg/proxy.(*Session).loopWriter
    0   /pika/codis/pkg/proxy/session.go:148
            pika/codis/v2/pkg/proxy.(*Session).Start.func1.3
luky116 commented 1 month ago

Codis 不支持 pipline 命令，属于是正常现象

Issues-translate-bot commented 1 month ago

Bot detected the issue body's language is not English, translate it automatically.

Codis does not support the pipline command, which is normal.