[Bug]: 1.10.2, kvsWebrtcClientMasterGstSample continually increases CPU usage

[lankahsu520]

Please confirm you have already done the following

• [X] I have searched the repository for related/existing bug reports
• [X] I have all the details the issue requires

Please answer the following prompt

• [X] This issue is replicable using the unmodified sample application

Describe the bug

No modifications have been applied to the source code

$ gcc --version
gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.6 LTS
Release:        20.04
Codename:       focal

$ cmake ..
$ make
$ export AWS_KVS_LOG_LEVEL=1
$ export DEBUG_LOG_SDP=TRUE
$ ./samples/kvsWebrtcClientMasterGstSample HelloLankaKVS

Use the "KVS WebRTC Test Page" to start the viewer and then stop it.

$ ps -p `pidof kvsWebrtcClientMasterGstSample` -o %mem,%cpu,vsz,cmd
%MEM %CPU    VSZ CMD
 0.3  7.8 1171532 ./samples/kvsWebrtcClientMasterGstSample HelloLankaKVS

$ ps -p `pidof kvsWebrtcClientMasterGstSample` -o %mem,%cpu,vsz,cmd
%MEM %CPU    VSZ CMD
 0.3 10.3 1171532 ./samples/kvsWebrtcClientMasterGstSample HelloLankaKVS

$ ps -p `pidof kvsWebrtcClientMasterGstSample` -o %mem,%cpu,vsz,cmd
%MEM %CPU    VSZ CMD
 0.3 29.3 1171532 ./samples/kvsWebrtcClientMasterGstSample HelloLankaKVS

Expected Behavior

kvsWebrtcClientMasterGstSample should release CPU usage

Current Behavior

Same as Describe the bug.

This issue is easy to reproduce. To avoid exposing Access Key and ID, I did not attach the log.

Reproduction Steps

Same as Describe the bug

WebRTC C SDK version being used

1.10.2

If it was working in a previous version, which one?

No response

Compiler and Version used

9.4.0

Operating System and version

Ubuntu 20.04.6 LTS

Platform being used

11th Gen Intel(R) Core(TM) i9-11980HK @ 2.60GHz

[niyatim23]

Hi @lankahsu520, what duration was this tested for? Do you see it stabilize after a point? We haven't seen this issue in the past

[lankahsu520]

log-1.7.2.txt log-1.9.1.txt log-1.10.2.txt

Run & Log

sequenceDiagram
    participant test as KVS WebRTC Test Page
    participant kvs as kvsWebrtcClientMasterGstSample

    loop 5 times
    test ->> kvs: View
    test ->> test: wait 30 seconds
    test ->> kvs: Stop
    test ->> test: wait 30 seconds
    end
    test ->> test: wait 10 minutes

VERSION	TIME	MEM	CPU	VSZ
1.7.2	13:17:10.742	0.3	12.4	1045276
1.7.2	13:31:14.395	0.4	16.8	1176348
1.9.1	13:36:12.513	0.3	10.1	1045432
1.9.1	13:54:14.166	0.4	12.4	1045432
1.10.2	13:58:14.532	0.3	12.5	1172764
1.10.2	14:13:06.036	0.4	15.5	1377568

export AWS_DEFAULT_REGION=ap-northeast-1
export AWS_KVS_LOG_LEVEL=5
export DEBUG_LOG_SDP=FALSE

export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...

[lankahsu520]

$ nproc
4
$ cat /proc/cpuinfo | grep "physical id" | sort | uniq|wc -l
1
$ cat /proc/cpuinfo | grep "processor" |wc -l
4
$ cat /proc/cpuinfo | grep "cores"|uniq
cpu cores       : 4

$ ps -p `pidof kvsWebrtcClientMasterGstSample` -o %mem,%cpu,vsz,cmd,time,etime,start
%MEM %CPU    VSZ CMD                             TIME     ELAPSED  STARTED
 0.4 15.5 1377568 /work/kvs-demo/amazon-kines 00:09:02      58:13 13:57:09

After 20 minutes, the %CPU usage is approximately 15.6. However, when observing CPU% using htop, it sometimes spikes up to 29.

[niyatim23]

How much RAM does your device have? Does the CPU usage taper?

[lankahsu520]

log-1.10.2-longtest.txt

After 6 hours, even without any viewers connecting, kvsWebrtcClientMasterGstSample still occupies 33% of CPU usage.

If run on modern computers, where performance is generally good, most people won't notice any lag. However, if used on an embedded system, lag might become noticeable. long-term CPU usage might also cause lag in other daemons.

In addition, consider running kvsWebrtcClientMasterGstSample more than twice on the same computer.

$ free
[Mon Jul 08 08:07:30.978 2024]               total        used        free      shared  buff/cache   available
[Mon Jul 08 08:07:30.979 2024] Mem:       16373300     1283820    11075108       29996     4014372    14724076
[Mon Jul 08 08:07:30.979 2024] Swap:       2097148           0     2097148

VERSION	TIME	MEM	CPU	VSZ
1.10.2	08:09:31.817	0.3	20.3	1174696
	09:09:32.319	0.4	36.7	1379500
	10:09:31.507	0.4	33.9	1379500
	11:09:31.270	0.4	33.2	1379500
	12:09:31.522	0.4	33.0	1379500
	13:09:31.602	0.4	33.0	1379500

[niyatim23]

Do you see the same pattern with this sample as well?

[lankahsu520]

log-1.10.2-kvsWebrtcClientMaster.txt

The same !!! Due to work commitments, I am unable to focus on tracing the code.

However, in my use case, it is possible to run over 100 instances on a single computer, which could amplify this issue. Currently, I can only use logs to inform you about this phenomenon.

This time I added nlwp to the ps command line.

We can see that at startup there are only 7 threads. But after just one connection, it increases to 10 threads.

I hope this information helps you identify the issue.

VERSION		TIME	NLWP	MEM	CPU	VSZ
1.10.2		14:23:32.623	7	0.0	0.0	262772
	view	14:24:32.732	12	0.0	0.1	1043860
	stop	14:25:12.779	10	0.0	0.4	1032844
	view	14:25:52.849	12	0.0	0.5	1032844
	stop	14:26:22.891	10	0.0	0.5	1032844
	view	14:27:02.944	12	0.0	0.6	1032844
	stop	14:27:32.988	10	0.0	0.7	1032844
	view	14:28:13.051	12	0.1	0.7	1032844
	stop	14:28:43.083	10	0.1	0.7	1032844
	view	14:29:23.138	12	0.1	0.8	1188504
	stop	14:29:53.174	10	0.1	0.8	1163916
		15:25:18.671	10	0.1	0.4	1163916