sonic-net / sonic-buildimage

Scripts which perform an installable binary image build for SONiC
Other
741 stars 1.44k forks source link

redis-server process in 202311 branch consume much more RAM comparing to 202305 #19825

Open dprital opened 3 months ago

dprital commented 3 months ago

Description

redis-server process in 202311 branch consume much more RAM comparing to 20235

Steps to reproduce the issue:

  1. Install 202305 SONiC image
  2. Get into database docker (command: 'docker exec -it database bash')
  3. Find redis-server PID (command: 'ps -a | grep redis-server)
  4. check redis-server RAM usage (command: 'cat /proc//smaps| grep Pss | awk '{Total+=$2} END {print Total/1024}'')
  5. Repeat the test with 202311 branch

Describe the results you received:

for: SONiC Software Version: SONiC.202305.610558-058b69a1d

admin@r-bulldog-03:~$ docker exec -it database bash
root@r-bulldog-03:/# ps -a | grep redis
     60 pts/0    00:00:02 redis-server
root@r-bulldog-03:/# cat /proc/60/smaps| grep Pss | awk '{Total+=$2} END {print Total/1024}'
28.6445

SONiC Software Version: SONiC.202311.610570-27b7bea6f

admin@r-bulldog-03:~$ docker exec -it database bash
root@r-bulldog-03:/# ps -a | grep redis-server
     63 pts/0    00:00:07 redis-server
root@r-bulldog-03:/# cat /proc/63/smaps| grep Pss | awk '{Total+=$2} END {print Total/1024}'
51.5

Almost twice RAM usage os redis-server process in 202311 comparing to 202305

Describe the results you expected:

Similar RAM consumption of redis-server as we see in 202305

Output of show version:

SONiC Software Version: SONiC.202311.610570-27b7bea6f
SONiC OS Version: 11
Distribution: Debian 11.10
Kernel: 5.10.0-23-2-amd64
Build commit: 27b7bea6f
Build date: Sat Aug  3 12:13:48 UTC 2024
Built by: azureuser@3ee7831cc000004

Platform: x86_64-mlnx_msn2100-r0
HwSKU: ACS-MSN2100
ASIC: mellanox
ASIC Count: 1
Serial Number: MT1752X06330
Model Number: MSN2100-CB2F
Hardware Revision: A1
Uptime: 18:22:08 up 6 min,  1 user,  load average: 0.41, 0.91, 0.57
Date: Mon 05 Aug 2024 18:22:08

Docker images:
REPOSITORY                    TAG                       IMAGE ID       SIZE
docker-syncd-mlnx             202311.610570-27b7bea6f   734eb36f7175   819MB
docker-syncd-mlnx             latest                    734eb36f7175   819MB
docker-orchagent              202311.610570-27b7bea6f   f7355e649ffa   339MB
docker-orchagent              latest                    f7355e649ffa   339MB
docker-teamd                  202311.610570-27b7bea6f   cfdefd3e624b   327MB
docker-teamd                  latest                    cfdefd3e624b   327MB
docker-sflow                  202311.610570-27b7bea6f   edcb9a54abf8   329MB
docker-sflow                  latest                    edcb9a54abf8   329MB
docker-fpm-frr                202311.610570-27b7bea6f   f2a2904e6f0e   359MB
docker-fpm-frr                latest                    f2a2904e6f0e   359MB
docker-nat                    202311.610570-27b7bea6f   d48b1f416049   330MB
docker-nat                    latest                    d48b1f416049   330MB
docker-dhcp-relay             latest                    5a2e0e8bb79b   310MB
docker-macsec                 latest                    04207bdadd76   329MB
docker-eventd                 202311.610570-27b7bea6f   02825f663993   301MB
docker-eventd                 latest                    02825f663993   301MB
docker-platform-monitor       202311.610570-27b7bea6f   c8001662a9e1   807MB
docker-platform-monitor       latest                    c8001662a9e1   807MB
docker-snmp                   202311.610570-27b7bea6f   45e65b6dcd2d   340MB
docker-snmp                   latest                    45e65b6dcd2d   340MB
docker-mux                    202311.610570-27b7bea6f   a1d0bcf57be4   349MB
docker-mux                    latest                    a1d0bcf57be4   349MB
docker-lldp                   202311.610570-27b7bea6f   5aa2f406a971   343MB
docker-lldp                   latest                    5aa2f406a971   343MB
docker-sonic-gnmi             202311.610570-27b7bea6f   732e32bbeb66   389MB
docker-sonic-gnmi             latest                    732e32bbeb66   389MB
docker-database               202311.610570-27b7bea6f   f5843d305055   301MB
docker-database               latest                    f5843d305055   301MB
docker-router-advertiser      202311.610570-27b7bea6f   e8ef5633780f   301MB
docker-router-advertiser      latest                    e8ef5633780f   301MB
docker-sonic-mgmt-framework   202311.610570-27b7bea6f   3f36ee2ab50e   431MB
docker-sonic-mgmt-framework   latest                    3f36ee2ab50e   431MB

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

zjswhhh commented 3 months ago

@qiluo-msft - please check if there is any redis-server version change or config change that can explain this? We need to know if this is expected.

zjswhhh commented 3 months ago

Also is it increasing comparing to earlier versions too?

dprital commented 3 months ago

@zjswhhh , I verified with older image. From 202205. the increase is the same Vs this version.

Version:

SONiC Software Version: SONiC.202205.570595-442fe3e7b
SONiC OS Version: 11
Distribution: Debian 11.8
Kernel: 5.10.0-23-2-amd64
Build commit: 442fe3e7b
Build date: Thu Jun 13 13:04:20 UTC 2024
Built by: cloudtest@6be35a23c000003

Platform: x86_64-mlnx_msn2100-r0
HwSKU: ACS-MSN2100
ASIC: mellanox
ASIC Count: 1
Serial Number: MT1752X06330
Model Number: MSN2100-CB2F
Hardware Revision: A1
Uptime: 19:19:01 up 3 min,  1 user,  load average: 0.94, 1.11, 0.52
Date: Thu 15 Aug 2024 19:19:01

Docker images:
REPOSITORY                    TAG                       IMAGE ID       SIZE
docker-syncd-mlnx             202205.570595-442fe3e7b   23c67a0b2558   830MB
docker-syncd-mlnx             latest                    23c67a0b2558   830MB
docker-platform-monitor       202205.570595-442fe3e7b   9e14a2b5faaf   753MB
docker-platform-monitor       latest                    9e14a2b5faaf   753MB
docker-orchagent              202205.570595-442fe3e7b   e02dc2b7a419   350MB
docker-orchagent              latest                    e02dc2b7a419   350MB
docker-fpm-frr                202205.570595-442fe3e7b   0cce06bee1a2   362MB
docker-fpm-frr                latest                    0cce06bee1a2   362MB
docker-teamd                  202205.570595-442fe3e7b   2f6e5d4f8d92   332MB
docker-teamd                  latest                    2f6e5d4f8d92   332MB
docker-macsec                 latest                    c20bd0eff299   334MB
docker-dhcp-relay             latest                    220e99fecccb   324MB
docker-sonic-telemetry        202205.570595-442fe3e7b   398d82a97614   397MB
docker-sonic-telemetry        latest                    398d82a97614   397MB
docker-snmp                   202205.570595-442fe3e7b   8c46297f89fa   366MB
docker-snmp                   latest                    8c46297f89fa   366MB
docker-router-advertiser      202205.570595-442fe3e7b   6f076238d475   316MB
docker-router-advertiser      latest                    6f076238d475   316MB
docker-mux                    202205.570595-442fe3e7b   2313b142b7c0   365MB
docker-mux                    latest                    2313b142b7c0   365MB
docker-lldp                   202205.570595-442fe3e7b   aa7dfbd14531   358MB
docker-lldp                   latest                    aa7dfbd14531   358MB
docker-database               202205.570595-442fe3e7b   aab9a27152c9   316MB
docker-database               latest                    aab9a27152c9   316MB
docker-nat                    202205.570595-442fe3e7b   7ac6f4cdbcf0   305MB
docker-nat                    latest                    7ac6f4cdbcf0   305MB
docker-sflow                  202205.570595-442fe3e7b   17ae68b5e61c   303MB
docker-sflow                  latest                    17ae68b5e61c   303MB
docker-sonic-mgmt-framework   202205.570595-442fe3e7b   d3e248dfec00   432MB
docker-sonic-mgmt-framework   latest                    d3e248dfec00   432MB

RAM Check:

admin@r-bulldog-03:~$ docker exec -it database bash
root@r-bulldog-03:/# ps -a | grep redis
     52 pts/0    00:00:03 redis-server
root@r-bulldog-03:/# cat /proc/52/smaps| grep Pss | awk '{Total+=$2} END {print Total/1024}'
28.3164