dragonflydb / dragonfly

A modern replacement for Redis and Memcached
https://www.dragonflydb.io/
Other
25.37k stars 917 forks source link

Why is the free dragonfly io utilized so much? #3786

Closed boomballa closed 2 hours ago

boomballa commented 2 hours ago

Describe the bug Idle dragonfly with high physical machine IO footprint, but no high IO write and read operations.

Screenshots sudo iotop -b -n 1 |head -n 50 image

pidstat -dl -p 25966 image

lsof -p 25966 image

Environment (please complete the following information):

Physical Machine Environment CPU 40 core Memory 128GiB Disk 1.8T SAS(Not SSD)

Additional context In response to this issue, where the IO footprint is high, but there are no obvious disk write and read operations, I understand that there are no more than a few reasons that can cause high IO for cached instances:

  1. save alive bgsave operation is being carried out, resulting in the RDB file write disk is more centralized.
  2. replica instance pulling bgsave files generated by the disk to cause a large number of read operations
  3. log printing is more frequent, resulting in greater pressure on the SAS disk IO.
  4. Frequent opening and closing of file handles by the dragonfly process
  5. hard disk failure

According to the phenomenon, dragonfly opened 40 threads, almost full of all the cpu io processes. image

It's true that I didn't specify the cpu resources that dragonfly can take up, but it shouldn't produce such a large IO footprint when there are no read/write accesses, I don't know if it's a bug or if I'm just lacking understanding of dragonfly's features.

--pidfile=/home/dba/dragonfly/dragonfly7379/dragonfly.pid

--log_dir=/home/dba/dragonfly/dragonfly7379
--logtostderr=false
--logtostdout=false
--max_log_size=500

--dir=/home/dba/dragonfly/dragonfly7379
--max_log_size=20
--version_check=true
--port=7379
--list_compress_depth=0
--cluster_mode=emulated

--maxclients=64000
--maxmemory=10Gib

--bind=10.46.128.11
--admin_bind=17379
--pipeline_squash=10
--tcp_keepalive=1800
--rename_command=
--hz=80
--keys_output_limit=8192

--compression_level=0
--compression_mode=0
--list_compress_depth=0

--admin_nopass=false
--multi_eval_squash_buffer=4096

--df_snapshot_format=true
--dbfilename=dragonfly7379-dump

Looking forward to the answer, thanks!

romange commented 2 hours ago

See here: https://github.com/dragonflydb/dragonfly/issues/2444#issuecomment-1901162720

the answer was given many times