High CPU usage while in call but all are muted / hidden

[PVince81]

Steps to reproduce

1. Setup a call with two participants, one using Firefox and one using Chrome on the same machine
2. Setup an avatar for both participants (might not be required)
3. Start the call on Firefox, observe CPU usage
4. Start the call from the other participant on Chromium, observe CPU usage
5. Mute everyone and disable the camera for everyone

Expected behaviour

No or low CPU usage

Actual behaviour

Chromium uses around 30% on my machine.

Talk app

Talk app version: 11.0.0 (git master 199c7d84f720b578aa76ba9480f03058b40e1ac4)

No custom servers

Browser

Microphone available: yes

Camera available: yes

Operating system: openSUSE Tumbleweed 20201007

Browsers: Firefox 80.0, Chromium 85.0.4183.121

Browser log

Server configuration

Nextcloud 21.0.0 alpha git master b76698c547bcaf22857f8ec67a8567399520f73a

Runs on the same machine.

[PVince81]

Things I've tried so far that did not work:

• commenting out the ".speaking" pulse animation (which I didn't see appear anyway)
• commenting out the blurred background
• hiding the element "#videos" completely: so it's not a rendering problem

I ran the performance tool from Chromium but nothing special was reported there.

[PVince81]

Hmm I also see a process "Web Content" which might be Firefox, and that one has 80% CPU also.

[PVince81]

I've tried the following hack to close the audio and video tracks when muting (similar to https://github.com/nextcloud/spreed/issues/4008#issuecomment-706185774):

diff --git a/src/utils/webrtc/simplewebrtc/localmedia.js b/src/utils/webrtc/simplewebrtc/localmedia.js
index 35c1563a9..dba1ba160 100644
--- a/src/utils/webrtc/simplewebrtc/localmedia.js
+++ b/src/utils/webrtc/simplewebrtc/localmedia.js
@@ -600,7 +600,11 @@ LocalMedia.prototype._setAudioEnabled = function(bool) {

        this.localStreams.forEach(function(stream) {
                stream.getAudioTracks().forEach(function(track) {
-                       track.enabled = !!bool
+                       if (!bool) {
+                               track.stop()
+                       } else {
+                               track.enabled = !!bool
+                       }
                })
        })
 }
@@ -609,7 +613,11 @@ LocalMedia.prototype._setVideoEnabled = function(bool) {

        this.localStreams.forEach(function(stream) {
                stream.getVideoTracks().forEach(function(track) {
-                       track.enabled = !!bool
+                       if (!bool) {
+                               track.stop()
+                       } else {
+                               track.enabled = !!bool
+                       }
                })
        })
 }

And then clicked the mute buttons and video hide buttons 2-3 times to trigger the routine, and end up with the muted/hidden state.

The CPU usage has reduced below 10% now (and my laptop fans are resting).

So it seems it's the WebRTC / audio / video stream bits that are causing high cpu, even when disabled. Not sure if those can be optimized.

I'm starting to see a benefit to actually stopping those streams, not only for CPU usage but also for https://github.com/nextcloud/spreed/issues/4008

[PVince81]

Another try where I close the video stream but leave audio open but disabled: both browsers report around 30 % CPU usage.

I suspect that the browsers might still be recording audio and video but discarding the data. Pulseaudio reports that both browsers have an active RecordStream. However if I close the stream nothing appears there.

If I block camera and microphone permissions, the CPU usage drops below 5%. This confirms that the CPU usage is likely related to the audio and video streams.

[PVince81]

I see there's also a cloned audio track to detect speaking when mute, to show the hint bubble. Not sure if that cloned track is causing additional overhead.

[PVince81]

I checked Jitsi yesterday and noticed that when disabling video, the video stream is stopped completely. But for audio the red microphone icon in Firefox was still there, I guess it's for detecting if someone is speaking.

Disabling video would already significantly reduce CPU usage, so I think disabling video and keeping audio would be a good tradeoff.

I had another look at the code and tried to (ab)use _handleVideoInputIdChanged() in localmedia.js as I saw that this method is re-creating the video stream. The result only worked once. I'll need to invest more time to understand better how the stream juggling (localStreams array), etc is working in more detail.

[PVince81]

To confirm that it's not only a Linux thing, with @Ivansss we tested on a Mac and can confirm that CPU usage stays the same when disabling video in the current code. However when using the hack from https://github.com/nextcloud/spreed/issues/4355#issuecomment-707034460 (the video path), CPU usage drops by 10%.

[PVince81]

cc @danxuliu as you seem the most familiar with this code