search

jitsi / jitsi-meet

Jitsi Meet - Secure, Simple and Scalable Video Conferences that you use as a standalone app or embed in your web application.
https://jitsi.org/meet
Apache License 2.0
22.78k stars 6.67k forks source link

Website is melting CPU! #5816

Closed JosjaVanBever closed 2 years ago

JosjaVanBever commented 4 years ago

I reported this at https://github.com/jitsi/jitsi-meet/issues/2596, but discovered that this issue was considered as "resolved". No, this issue is not resolved. You can read the description at that issue. I copy here my comment and the response of somebody with the same issue. This issue prevents any decent or acceptable use of jitsi and makes the program useless for at least a lot of unix based systems.

JosjaVanBever commented 2 days ago

Hello,

I am using a brand new Dell XPS13 with Ubuntu19 installed. The jitsi web application is eating up to 250% of cpu when having a video conference. Consequently, the cpu fan is turning grazy so loudly that my microphone only hears computer noise. As a result, I cannot access the conversation anymore because my voice is not recognized within the noise of the fan. Using a headset is no option because we participate with two in the video conference and the noise of the fan keeps going anyway. This problem prohibits me of using jitsi and I will need to convince all people participating to our video conferences to not use it anymore. Could you please fix this grazy overconsumption problem asap? As a first step, you could tell how I do set the absolute minimum configuration with only camera and microphone via https using standard wifi, excluding tools as TCP harvesting (sth I've read at https://community.jitsi.org/t/jitsi-videobridge-consuming-200-cpu-utilization-constantly/19482) and so on. I am even not interested to share my screen at this point as talking is already a challenge in these times.

Thanks in advance, Josja Van Bever

up_the_irons commented 2 days ago

I used Jitsi for the first time yesterday and my CPU load average climbed so high, that eventually I was forced to reboot. Shutting down Chrome didn't solve the issue. There was a single chrome process still running (I run latest Arch Linux). I couldn't even kill -9 it, and I'm not sure why. My mouse stopped working and my Bluetooth keyboard disconnected. It was like a total meltdown.

My friend, who was on the other end of the video conference, was on a Mac and he also had to reboot his machine.

I know this is all mostly anecdotal, but I just want to bring this up because it's still an issue.

phil-flex commented 4 years ago

I am the same issue. I am using the following configured laptop,

Processor Intel(R) Core(TM) i7-7500U CPU @ 2.70GHz, 2904 Mhz, 2 Core(s), 4 Logical Processor(s) OS Name Microsoft Windows 10 Pro System SKU LENOVO_MT_20K7_BU_Think_FM_ThinkPad 25 Installed Physical Memory (RAM) 16.0 GB Available Physical Memory 7.17 GB

Running the Chrome Version 81.0.4044.122 (Official Build) (64-bit)

it used all CPU resource.

WolfganP commented 4 years ago

Did you try disableAudioLevels=true as per https://github.com/jitsi/jitsi-meet/issues/5464#issuecomment-614000758 / https://community.jitsi.org/t/host-a-meeting-with-500-people-ideas/34672/4 ?

JosjaVanBever commented 4 years ago

Does this mean that I need to insert the url https://meet.jit.si/NAME#disableAudioLevels=true instead of https://meet.jit.si/NAME?

saghul commented 4 years ago

@JosjaVanBever correct.

dybber commented 4 years ago

I also experience this. However, it's somewhat better when using tile view. It still uses more resources than what should be expected, but at least it's usable.

cweiske commented 4 years ago

We see massive CPU loads in conferences with ~10 people. Load on the Ubuntu 18.04 with Chromium 81.0.4044.122 is at around 3.5, with 4 CPU cores.

Other people in the same conference report the same issues; they use Macbooks with Chrome.

Audio levels have already been disabled.

ccapndave commented 4 years ago

Likewise - in a chat with 5 people on MacOS in Chrome the fans are constantly running at full pelt.

monnier commented 4 years ago

Does this mean that I need to insert the url https://meet.jit.si/NAME#disableAudioLevels=true instead of https://meet.jit.si/NAME?

Any hope this can be more easily available (i.e. appear in the parameters, or even come with an automatic popup suggestion when Jitsi notices that the CPU use is too high?)

luixxiul commented 4 years ago

See https://community.jitsi.org/t/reducing-resource-usage-to-improve-performance-both-client-side-and-server-side/39891 for possible improvement by modifying configuration files.

By default the resolution is set to 720p, so basically this means that if you have five pictures of the other people on the tile view or the film strip bar, it is like playing five 720p high resolution videos of YouTube at the same time 😲 It must be obvious that this should consume resource of your PC very much and you should need a nice GPU for better UX...

saghul commented 4 years ago

That is not the case.

Only the active speaker in large view mode will be using 720p. Thanks to simulcast, the filmstrip will be using 180-240p per thumbnail.

In tule view we adjust the size based on the rendered size. You can check the active resolution by hovering over the “GSM bars” icon.

luixxiul commented 4 years ago

Ach, thanks for the correction!

I'm wondering what to be fixed to improve performance 🤔

nekohayo commented 4 years ago

The audio levels suggestion mentioned above is interesting, but since I don't expect users to know/think of that, I created issue #6920 with some ideas of a less expensive (though less fancy) way of representing audio activity.

monnier commented 4 years ago

The audio levels suggestion mentioned above is interesting, but since I don't expect users to know/think of that, I created issue #6920 with some ideas of a less expensive (though less fancy) way of representing audio activity.

Actually, I think the current behavior is a plain performance bug: there's no fundamental reason why the little blue dots should require more work with 500 participants than with 10 when only 10 thumbnails are visible (and hence only those 10 really need their little blue dots to wiggle).

[ I understand that fixing this performance bug may be non-trivial, e.g. maybe the current code is not aware which thumbnails are currently visible and which aren't (they're all in the DOM and the rest is handled by the browser's HTML rendering engine), but it's a performance bug nevertheless. ]

timaschew commented 4 years ago

Is it possible to configure this globally via the server: disableAudioLevels=true?

danwalmsley commented 4 years ago

Can confirm I am also experiencing this. You can be in a meeting, everything fine, then 10 or 15 minutes in, might be after you switch to another tab or app to look at someting... cpu goes into meltdown, and the whole machine is unusable, audio jitters and cuts out... it was taking me like 5 minutes just to be able to kill the browser in the task manager and then get back to the meeting.

even mouse cursor basically grinds to a halt.

Windows 10, Edge (chrome version)

paweldomas commented 4 years ago

Thanks for the report! Just wanted to let you know that it's on our radar. Maybe there's some unnecessary work happening or a backlog of work builds up, either when the tab is in the background or when the CPU is overloaded.

paweldomas commented 4 years ago

There's also a plan to add sort of advanced options menu where you would be able to disable features like audio levels.

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

afranke commented 3 years ago

Still relevant, go away stalebot.

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

nekohayo commented 3 years ago

Begone, foul stale robot!

nekohayo commented 3 years ago

One of the best ways to experience the full extent of all the combined performance issues is to test meet.jit.si with a 10-20 participants video meeting on older computers, running for example a Core2 Duo or a Pentium D CPU, on Firefox on Linux. The result is that if you only have two CPU cores, both cores get pegged at 100% and once you reach 100%:

From my testing tonight, this is caused by three things:

On a laptop with a dual core Intel T3200 CPU and Intel graphics tonight, the only way I was able to get good audio was to switch to low-bandwidth mode (audio only) AND to switch to another (blank) tab, which made the CPU usage go from 2x100% (where everything was unusable) to 2x85% (where audio quality was perfect);

A rule of thumb I have when testing software for performance bottlenecks: always assume your users will be using potato computers. I suspect most developers in general are testing with powerful modern machines, with at least a 2nd-generation Intel Core i7 CPU with 4 to 8 threads (or more), good GPUs, and SSDs, which is totally understandable because they need to be productive in the day-to-day; my recommendation however would be: please smoke-test with 10-years-old machines, because that's what many users out there will be using in practice (seriously) and it makes it much easier to find performance issues that benefit everybody anyway.

monnier commented 3 years ago

Indeed, I use a Core 2 Duo on some of my machines and the performance limitations tend to be problematic there. Low bandwidth connections can also be problematic. I can understand that this may not be the main target, but I think it does point to some real improvements: it's OK if a Core 2 Duo with a slow connection can't provide the "full" experience, but when for some reason the Jitsi client "can't keep up" (either because of the CPU or because of the network), it should be able to degrade more gracefully by:

as much as needed to keep the audio from lagging or breaking up and to keep the UI responsive. [ Oh, and I wish I could sometimes choose between reducing resolution and refresh rate. E.g. when the video feed comes from a camera, it's OK to reduce the resolution, but when it mostly comes from PDF slides (maybe with a camera in the corner), then I'd rather have full resolution even if that requires going down to 0.1 frames/s refresh rate. ]

saghul commented 3 years ago

Thanks for providing feedback folks. Performance is an area we are currently spending a lot of time on.

running for example a Core2 Duo or a Pentium D CPU, on Firefox on Linux.

Have you tried Chromium on the same hardware?

From my testing tonight, this is caused by three things:

* Receiving videos of all the participants

Were you using tile view or stage view? You can try to limit how many videos you receive with the following URL parameter: #config.channelLastN=X and see if that helps.

This is a hard problem to solve because if you receive many videos it means you have the bandwidth, and detecting CPU overuse is not straightforward.

* Encoding/sending your own video (probably; but I have no way to test this independently because low-bandwidth mode doesn't let me send my video;

We have recently added "layer suspension", so the sender will only encode the video at the resolution being received, so 720p layers will be dropped if nobody is receiving you in HD for example.

* on that note, I would like the ability to turn off others' video while continuing to send my own video feed, see issue #8457)

You can do that with the config option I mentioned earlier. We currently have no plans to make that something you can tweak in the UI.

* If you were to get rid of vumeter animations in the UI, you would gain a lot of performance that stacks with turning off video thumbnails and such.

I've tried to measure this by taking multiple profiling samples but haven't been able to prove it. I don't see many re-layouts being triggered, but I might have missed something. Do you have performance measurements that back this up?

A rule of thumb I have when testing software for performance bottlenecks: always assume your users will be using potato computers.

Absolutely. As I mentioned earlier, however, it's not straightforward to detect CPU overuse (and recover from it!) within the web application itself.

[ Oh, and I wish I could sometimes choose between reducing resolution and refresh rate. E.g. when the video feed comes from a camera, it's OK to reduce the resolution, but when it mostly comes from PDF slides (maybe with a camera in the corner), then I'd rather have full resolution even if that requires going down to 0.1 frames/s refresh rate. ]

This should already be happening automatically. Screen sharing does not run at 30fps. Are you observing otherwise? What browser are you using?

nekohayo commented 3 years ago

What browser are you using?

Tested with Firefox 84.x and Firefox 78 ESR, same kind of performance between the two Firefox versions, from what I can tell.

Have you tried Chromium on the same hardware?

Also tested with Chromium now, it might have slightly better performance, but is still sluggish: even with only two participants (both having webcams) in the meeting, if muted then chromium sits around 60-70% CPU usage, but as soon as you unmute the other participant and have it tap its microphone constantly with a finger (to generate sound) then you see CPU usage climb to 100% in Chromium. So apparently even with not-too-many participants I can reproduce the issue.

I don't know how to properly do performance measurements (please advise), so I just opened up each browser's inspector, went into the performance tab, hit record, and saved the resulting json files. You can see them here and import them in your browser, crossing fingers that you can see something meaningful in there(?) : https://fortintam.com/public/jitsi-meet-perf-profiles/

Were you using tile view or stage view?

Both had identical (or seemingly identical) horrible performance last night, i.e. so sluggish I couldn't successfully click on UI elements. Trying to hide the thumbnails sidebar in stage view didn't help either.

In the profile outputs linked above, I was using tiled view. I didn't have that much patience to test out many combinations because the UI was so sluggish it was painful to get those profiles already, but if you have specific things you want me to test I can try to do so.

saghul commented 3 years ago

Thanks, I'll take a look at the profiles.

nekohayo commented 3 years ago

For fun, I also tried the (electron?) desktop app from https://flathub.org/apps/details/org.jitsi.jitsi-meet and on that old laptop, with both participants (two computers on the same network, connecting to meet.jit.si) muted but emitting video, and watching the CPU usage with htop:

An Intel Sandybridge i5 CPU with the same app doesn't experience as many problems in this two-participants test, as it has twice the number of logical CPUs and it "only" eats 50% of each... and on that computer, the performance difference between Firefox and the electron app doesn't seem very noticeable, Firefox uses maybe 0 to 10% more CPU I think.

saghul commented 3 years ago

The desktop app uses Electron, so it should have the same performance characteristics as Chromium.

gtsop commented 3 years ago

Struggling with this myself, I compiled a list of optimizations I gathered by visiting multiple threads (some of them listed here as well). I have successfully ran meetings with ~25 participants all using video/audio and ~100 participants (post says 75 but I got a more recent one) that involved a single talker (video and audio) and the rest were muted/novideo, just listening and chatting.

I would be very grateful if a member/author could check the validity of these config files since I have the feeling some options are outdated.

https://blog.gtsop.me/jitsi-meet-performance-tuning.html

saghul commented 3 years ago

Having a 5-10 fps will give you low CPU usage, but also a very bad user experience.

gtsop commented 3 years ago

Having a 5-10 fps will give you low CPU usage, but also a very bad user experience.

It depends. For my use case that involves a few people talking to the camera this is a perfectly acceptable experience. I haven't had a single person complain about low frame rate and I don't even notice it as a problem. Of course it will differ for others if they wish to share high framerate video.

monnier commented 3 years ago

Having a 5-10 fps will give you low CPU usage, but also a very bad user experience.

It gives a much better user experience than a higher frame rate with audio drops.

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

nekohayo commented 3 years ago

Hi @saghul, by any chance had you had time to look at the performance profiles mentioned in https://github.com/jitsi/jitsi-meet/issues/5816#issuecomment-765514781, did anything stand out to you?

Werkov commented 3 years ago

FWIW, I profiled Jitsi in FF and found out audio processing is not very efficient. Read: this would tackle only the audio part and it won't make FF less CPU demanding than Chromium that already uses the updated libwebrtc.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

dsvk commented 2 years ago

Regarding the audio processing overhead pointed out by @Werkov above, the Bugzilla bug #1702781 mentioned has been marked as a duplicate of now-resolved #1654112 (libwebrtc has been updated in FF96) and there appears to be some resulting improvement for Jitsi Meet.

(This is not a call to close, there are probably other optimizations in scope of this issue that could still be done as well.)

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

maymage commented 1 year ago

@saghul

I really think, this issue should be reopened. I am too on Linux, Current Debian Sid + Gnome with a N5101 and all four cores are >95% in htop.. the experience is jittery.

saghul commented 1 year ago

There is just too much variance on Linux. Wayland vs Xorg, Nvidia drivers vs other vs nvidia open source, etc.

At this point we'd need more concrete evidence that the source for that high CPU usage is Jitsi Meet itself.

As an example, macOS stopped advertising MJPEG as a supported capture format for webcams which results in VideoToolbox based conversions which use more CPU. This is, however, not something we can fix at all.

maymage commented 1 year ago

There is just too much variance on Linux. Wayland vs Xorg, Nvidia drivers vs other vs nvidia open source, etc.

The future of Linux desktop seems to be Wayland+Pipewire+{Flatpak,Snap} and to quote: f**k Nvidia. So that's probably the situation to optimize for.

At this point we'd need more concrete evidence that the source for that high CPU usage is Jitsi Meet itself.

Jitsi seems to melt the CPU and this is that.. So how to to analyse the problem to go up the hierarchy and possibly exclude, that it's caused by jitsi itself. What other application video chat system would you recommend to compare against? @nekohayo gave some profiling, so if that is not sufficient, do you have further instructions how we can test this?

med-zz-eis commented 1 year ago

Struggling with this myself, I compiled a list of optimizations I gathered by visiting multiple threads (some of them listed here as well). I have successfully ran meetings with ~25 participants all using video/audio and ~100 participants (post says 75 but I got a more recent one) that involved a single talker (video and audio) and the rest were muted/novideo, just listening and chatting.

https://blog.gtsop.me/jitsi-meet-performance-tuning.html

I do not see this link works anymore, might you be able to provide the guide maybe here as well?