j1elo
commented
4 years ago For 6.14.0 we worked on a performance issue that would cause CPU exhaustion when an Endpoint tried to disconnect from N Endpoints, for somewhat big N (depends on the machine, but somewhere around N > 20 we could observe the issues in our case).
With some optimizations the CPU usage decreased significantly, we talked a bit about this in the 6.14.0 blog post.
Your issue seems very similar to the one we solved. Maybe there are still some cases when CPU exhaustion occurs... Can you observe if the 200% CPU spikes are happening when one sender PlayerEndpoint is being disconnected from a big number of receiver WebRtcEndpoints?
Other idea is to avoid delete-and-create of new PlayerEndpoints each time the test video is repeated. What happens if, instead, you call setPosition(0) before the file ends playing? (e.g. if the video file length is 30 seconds, call player.setPosition(0) every 25 seconds).
PS. the -asan images are made to run Kurento with AddressSanitizer, to find memory corruption bugs. This doesn't look like a memory corruption issue, so I don't think we'll get anything useful from it. A better tool to use would be Callgrind, to generate profiling output and then inspect it with KCacheGrind, but we'll leave that for later if we don't find a solution.
MaxIasat
micaelgallego
Prerequisites
[X] I have read the SUPPORT document
[X] I have checked the Troubleshooting Guide
[X] I have tested with the latest version of Kurento
Issue description
Application: My application is similar to “one-to-many” tutorial, but instead of the presenter i have used PlayerEndpoint to stream the video files (mkv files). As soon as one client is connected to the application server the streaming will be started. Other clients are connected to the running stream. Some kind of business logic selects in the infinite loop the video media files and triggers the creation of the new PlayerEndpoint and then the playing the new video file. Before it, the old one PlayerEndpoint has been released. The existing WebRtcEndpoints will be connected to the new PlayerEndpoint. All the clients show the same video file.
So, i have one pipeline, one player endpoint and many WebRtcEndpoints connected (in my case i have only max 5 clients connected). The files are provided with file:// attribute to the PlayerEndpoint.
The video files have duration of 30 seconds. Every 30 s the business logic triggers the new video file. I have tested kurento 6.14.0 and 6.14.0-asan versions installed in docker on Debian 8 machine with 4 cores CPU and 4 GB RAM. Mkv video files have 720p resolution and coded as H.264 with 25fps. I have disabled the using of VP8 codec in kurento. So, only H.264 codec can be chosen. In this case the video files are not transcoded. I log this information also in my application by subscribing on MediaTranscodingStateChange event. The audio is disabled. The video files don’t have any audio tracks.
General behaviour: The general behaviour is as expected. As soon one client connects to the server the streaming is started and each 30 seconds, i see new video file. Other connected clients see immediately the running stream. The stream runs synchronously. From functionality point of view, it works great.
Long term behaviour kurento 6.14.0 In the above-mentioned configuration, the system runs stable couple of hours (ca. 8h). At the beginning I noticed cpu usage for kurento-media-server pending about 90% (with top tool). After some time I observed some kind of CPU spikes (about 200%). Finally, after ca. 8h the cpu usage was huge and the system didn’t respond. Only reboot could help. I noticed also the memory increasing over the time (with top tool).
Long term behaviour kurento 6.14.0-asan So, for further analysing (maybe memory leaks) I used the asan version of the kurento as docker with the same configuration described above. I was very surprising, but this asan version were running more than 3 days on my system. But the kurento media server crashed! Here are the last log lines:
This behaviour I could reproduce twice. The container with the crashed kurento inside I can provide as a tar file. Can you please help with analysing of this issue? Thank you and best regards
Context
I try to implement a video player which stream different videos every 30 seconds.
How to reproduce?
Do long term test with PlayerEndpoint and many clients by streaming different files in the infinite loop. See description above.
Expected & current behavior
Stable system without crashes.
(Optional) Possible solution
I suppose maybe some kind of race condition blocks the released version of kurento. In the asan version i think the timing is a little bit different and this may be the cause why kurento runs longer.
INFO about Kurento Media Server
INFO about your Application Server
INFO about end-user clients
INFO about your environment
No STUN or TURN server are configured. The system is running in the local network with dhcp.
I have used the following docker run command:
Commands output