search

livekit / client-sdk-flutter

Flutter Client SDK for LiveKit
https://docs.livekit.io
Apache License 2.0
229 stars 108 forks source link

[bug] Livekit's CPU usage #364

Open furkanKotic opened 9 months ago

furkanKotic commented 9 months ago

Describe the bug When I test my application on Flutter iPhone, livekit's CPU usage is as follows:

Viewer CPU usage when there is a streamer in the room and his cameras are off: 1

Viewer CPU usage when there is a streamer in the room and his cameras are on: 2

Viewer CPU usage when there are two streamers in the room and their cameras are on: 3 4

Viewer CPU usage when there are three streamers in the room and their cameras are off: 5

Viewer CPU usage when two streamers in the room with their cameras on and one streamer with his camera off: 6

When I test without a livekit connection in the same application, I observe 6% CPU usage. In this case, we observe 50-60% CPU usage per broadcaster in Livekit. These processor uses cause the phone to overheat and increase battery usage. I think Livekit needs optimization.

Platform information

Parfyonator commented 8 months ago

Hi. Users of my app that has video call integrated by means of this plugin also experience overheating. For some of them it is moderate and for the others it is pretty high.

cloudwebrtc commented 8 months ago

hey @furkanKotic @Parfyonator, Hey, can you guys confirm what video code was being used when this issue occurred, and what phone model you were using? I think the video code that consumes the most power in libwebrtc is usually the video codec.

Parfyonator commented 8 months ago

Hi @cloudwebrtc. In my case it is 14 pro max. One of the users reported that after 30 min of call it became so hot that the app became very laggy due to the very high temperature of the phone.

furkanKotic commented 8 months ago

Hi @cloudwebrtc. The scenario of @Parfyonator is also valid for me. My users report that the phone overheats after 30 minutes, the app becomes extremely slow, and the phone gives a high temperature warning. We are using H264 codec. We know that it is available in all recent iPhone models.

Some tested models: iPhone 15 pro max iPhone 14 pro max iPhone 14 iPhone X

I have all of these devices. These devices experience overheating when broadcasting or just participating as a viewer. But there is no problem on Android devices.

Resolution: h540_43 Bitrate: 1000 Framerate: 15

dragontran commented 8 months ago

Heyo! I wanted to point out that I've observed high CPU usage and high temps during extended use in my project, the LiveKit example project, and the Flutter WebRTC example project. I tested these projects on my iPhone XR, using the H264 encoder at a 540_43 resolution and 20 fps.

I also experimented with the example project for the camera package. It appears to utilize approximately 30% CPU when the camera is active. Is it typical for the camera in Flutter for iOS to demand such high CPU consumption?

Also, a big thanks for everything you do. @cloudwebrtc

EqualCHZH commented 7 months ago

I have the same heating problem. iPhone users complain about the application overheating and overheating after 30 minutes of use.

@cloudwebrtc @davidzhao

domvie commented 5 months ago

Can confirm, same thing happens to our iOS devices (12, 14 Pro, iPad Pro 11"). Very high energy + CPU usage. Perhaps this is a flutter-webrtc issue?

elonmaph commented 2 months ago

high cpu ,high energy and high temperature, Is there any official plan to solve this performance problem?

cloudwebrtc commented 2 months ago

Hey all, we will investigate this issue this week.

cloudwebrtc commented 2 months ago

Can confirm, same thing happens to our iOS devices (12, 14 Pro, iPad Pro 11"). Very high energy + CPU usage. Perhaps this is a flutter-webrtc issue?

hi @domvie Could you provide some reference for cpu usage? For example, open the camera on 14 pro, publish a 720p video track with using h264, I have a 14 pro, and iPad air M1, I will do a comparison test

domvie commented 2 months ago

Can confirm, same thing happens to our iOS devices (12, 14 Pro, iPad Pro 11"). Very high energy + CPU usage. Perhaps this is a flutter-webrtc issue?

hi @domvie Could you provide some reference for cpu usage? For example, open the camera on 14 pro, publish a 720p video track with using h264, I have a 14 pro, and iPad air M1, I will do a comparison test

So I just re-tested this using the livekit client-sdk-flutter example (I believe its version 2.1.0 as of right now). I connected with two real clients (iPhone 12, iPad Pro 11"). At times I've played around with adding a couple of video publishers using livekit-cli but that never seemed to make a difference.

Some stats I'm getting:

At this point I have stopped testing as it is quite time consuming.

All in all, in this session I could unfortunately not reproduce the overheating issue I encountered a while back. Especially when using hardware accelerated H264 the devices seemed to run just fine, other codecs increased CPU and heat quite a bit (although that is to be expected). Energy consumption was always very high.

I'm guessing this could be due to multiple reasons:

In the current form of our app it also does not overheat most of the time. We enable only dynacast, everything else off, with a resolution of 1440p @ 30fps and max bitrate of 15mbps using H264. These are the settings we had the most success with for our requirements. So it can definitely handle quite a bit of data without causing issues. I also remember the problem happening independent of res/fps, so it would also overheat at say 540p and a low frame rate. The weirdest thing to me is that it's not easily reproducible, or why it would sometimes happen with h264 and sometimes not. Other codecs make more sense I guess but even then it was never obvious.

Hopefully this was somehow helpful, sorry I couldn't reproduce it this time. Perhaps @furkanKotic still can.

cloudwebrtc commented 2 months ago

findings

  1. Flutter’s own texture rendering will cause basic CPU usage, take iPhone 14 pro max as an example When the camera is turned on but video rendering is not performed, the CPU usage is about 3~4%.

Turn on the camera but return a nil pointer from copyPixelBuffer, which will trigger flutter UI Widgets to refresh. At this time, the CPU usage will reach 18~22%

Normally rendering the camera stream, due to YUV rotation and conversion to RGBA format texture data, the CPU usage will reach 28~34%

  1. The same VideoTrack uses PlatformView + RTCMTLVideoView to render and the CPU usage is 10%.

loopback test 1280x720 30fps Using Flutter Texture to render CPU usage is about 47~55%

Using Flutter PlatformView + RTCMTLVideoView, the rendering CPU usage is about 30～35%

Related PR drafts https://github.com/flutter-webrtc/flutter-webrtc/pull/1569

elonmaph commented 4 weeks ago

should I just upgrade the sdk to 2.1.5 to solve this issue?

cloudwebrtc commented 4 weeks ago

@elonmaph you can try this branch https://github.com/livekit/client-sdk-flutter/pull/509, and change VideoTrackRenderer like this.

VideoTrackRenderer(
    _videoTrack!,
    fit: RTCVideoViewObjectFit.RTCVideoViewObjectFitContain,
    renderMode: VideoRenderMode.platformView,
  )
dragontran commented 1 week ago

I've been testing the platform view branch and wanted to share some feedback. The CPU performance improvements are definitely noticeable and my iphone hasn’t overheated, which is fantastic!

However, I’ve encountered a couple of issues:

I’m really excited about the platform view and performance updates for iOS! Thanks so much for your help!

furkanKotic commented 1 week ago
  • The platform views do not respect the RTCVideoViewObjectFit.RTCVideoViewObjectFitCover fit option and extend beyond the widget frame.

I noticed this problem too. It's like it doesn't care about the height of the Sizedbox and the image is created with a size higher than the value given to the Sizedbox.