search

AirenSoft / OvenMediaEngine

OvenMediaEngine (OME) is a Sub-Second Latency Live Streaming Server with Large-Scale and High-Definition. #WebRTC #LLHLS
https://OvenMediaEngine.com/ome
GNU Affero General Public License v3.0
2.5k stars 1.05k forks source link

No RTSP Timeout on v0.10.4 #150

Closed nangavar closed 4 years ago

nangavar commented 4 years ago

Hi All!

During some tests running v0.10.4 I found that if there is a connection drop in the RTSP provider, the stream gets halt. No timeout working. I have to restart OME.

In v0.10.3 after a connection drop in RTSP provider, OME waits 60sec and stops the stream ("No incoming packets" message). Im experiencing some problems connection the RTSP provider trough internet, no problems in LAN.

BTW, is there anyway to reduce this timeout??

Many thanks in advance!

getroot commented 4 years ago

The ability to disconnect after 60 seconds is to clean up the pull provider when 60 seconds have passed without viewers. This is a function that prevents wasting resources by running the Pull provider even when there are no viewers. This is because when the viewer requests the stream again, it will request a connection to the provider.

This function works in the WhiteElephantStreamCollector function in the /src/project/base/provider/pull_provider/application.cpp file. The stream is cleaned up after 60 seconds because the MAX_UNUSED_STREAM_AVAILABLE_TIME_SEC value is 60. This will allow you to set it in your config file in the future.

In version 0.10.3, it was a bug that the provider continued to operate even if the connection to the RTSP server was lost. The RTSP disconnection is obviously broken, so we have to clean up the stream immediately. After that, when a request comes in from Publisher (when the viewer tries to connect to the stream), the Origins setting is used to request a connection back to the Provider. (Does this function not work?)

nangavar commented 4 years ago

Hi getroot! Many thanks for your quick response! I think this function is not working. In 0.10.4 after a RTSP disconnection the stream becomes unavailable. When I try to get the stream again OME does not try to pull the RTSP source back. I have to restart OME to recover this stream.

getroot commented 4 years ago

It is normal to delete the stream when the connection to the RTSP server is lost. However, it is a bug not to try to connect to the RTSP server again when a player requests to play. I will reproduce this and fix it.

getroot commented 4 years ago

If there is a log of when that happens (the connection to the RTSP server is lost, the player requests it again, but does not request a reconnection to the RTSP server), please upload it, which helps me fix it quickly.

nangavar commented 4 years ago

This is what I get in master branch after a RTSP disconnection. I stopped playing after the disconnection, and when I try to play again I get a FFMPEG error. Then the stream becomes unavailable

[2020-07-15 10:35:07.056] I 40010 Monitor | stream_metrics.cpp:144 | A session has been stopped playing #default#app2/stream6 on the WebRTC publihser. Concurrent Viewers[WebRTC(0)/Stream total(0)/App total(0)] [2020-07-15 10:35:07.056] I 40010 Signalling | rtc_signalling_server.cpp:237 | Client is disconnected: <WebSocketClient: 0x7fb0039ec430, <ClientSocket: 0x7fb0039d8010, #4, state: 4, TCP, 181.167.1.77:62093>> (#default#app2 / stream6, ufrag: local: VlvBDL, remote: GvlK) [2020-07-15 10:35:07.656] I 40010 Signalling | rtc_signalling_server.cpp:99 | New client is connected: <ClientSocket: 0x7fb0039d8010, #4, state: 4, TCP, 181.167.1.77:62109> [2020-07-15 10:35:07.725] I 40010 Monitor | stream_metrics.cpp:119 | A new session has started playing #default#app2/stream6 on the WebRTC publihser. WebRTC(1)/Stream total(1)/App total(1) [2020-07-15 10:35:07.728] W 40010 Socket | socket_address.cpp:277 | An error occurred while resolve DNS for host [f3526761-c5ef-4a69-90e6-096276127404.local] [2020-07-15 10:35:07.728] W 40010 Socket | socket_address.cpp:84 | An error occurred: f3526761-c5ef-4a69-90e6-096276127404.local:65529 [2020-07-15 10:35:16.863] E 40552 FFmpeg | third_parties.cpp:111 | [AVCodecContext] missing picture in access unit with size 45 [2020-07-15 10:35:25.693] I 40010 Monitor | stream_metrics.cpp:144 | A session has been stopped playing #default#app2/stream6 on the WebRTC publihser. Concurrent Viewers[WebRTC(0)/Stream total(0)/App total(0)]

nangavar commented 4 years ago

Seems that the stream is never deleted by OME after the RTSP disconnection. 10 minutes later the stream stills unavailable.

nangavar commented 4 years ago

Normally I don't get the FFMPEG error, OME just stop playing and never deletes the stream.

getroot commented 4 years ago

Is there a log when the connection to the RTSP server is lost? I would like to see all logs down there from there. This is because I need to check the process of OME trying to delete the stream after disconnecting from the RTSP server.

nangavar commented 4 years ago

Im getting no disconnection message on master branch. On v0.10.3 yes, but in master I don't get any log of the disconnection. OME just stop playing when I have connection issues (ping delays). v0.10.3 starts messaging "no incoming packets" when I experience those ping delays.

getroot commented 4 years ago

I think I solved this problem (I couldn't test it because it was difficult to build the same environment. Please help me with the test.) I found a mistake in the code refactoring process. Let me know if this solved your problem.

6e68607bbba6b07d9ee91648cff6d5fe1f44d63c

nangavar commented 4 years ago

Hi getroot! Im sorry but since my cloud-server provider improved the networking and I tuned the kernel, I'm not able to reproduce the disconnections anymore. :( I tried turning off the encoder, but it has another behaviour. In fact something similar happens, once I turn off and on the encoder OME does not try to pull it again and the stream becomes unavailable.

In logs you can see when the encoder is turned off (Provider: has been stopped playing....), and then my attempts to start playing again, but I'm not able to get the stream again.

Remember my configuration ENCODER -> ORIGIN OME -> EDGE OME -> PLAYER. Logs are from EDGE.

Logs:


[2020-07-16 18:12:11.746] I 24744 Provider | stream.cpp:55   | #default#app2/stream1(100) has been stopped playing stream
[2020-07-16 18:12:18.525] I 24645 Monitor | stream_metrics.cpp:144  | A session has been stopped playing #default#app2/stream1 on the WebRTC publihser. Concurrent Viewers[WebRTC(0)/Stream total(0)/App total(0)]
[2020-07-16 18:12:18.525] I 24645 Signalling | rtc_signalling_server.cpp:237  | Client is disconnected: <WebSocketClient: 0x7efc1ee06280, <ClientSocket: 0x7efc203d8010, #4, state: 4, TCP, 181.167.1.77:62588>> (#default#app2 / stream1, ufrag: local: nojDO5, remote: ih8h)
[2020-07-16 18:12:18.559] I 24645 Signalling | rtc_signalling_server.cpp:99   | New client is connected: <ClientSocket: 0x7efc203d8290, #4, state: 4, TCP, 181.167.1.77:62600>
[2020-07-16 18:12:18.611] I 24645 Monitor | stream_metrics.cpp:119  | A new session has started playing #default#app2/stream1 on the WebRTC publihser. WebRTC(1)/Stream total(1)/App total(1)
[2020-07-16 18:12:18.612] W 24645 Socket | socket_address.cpp:277  | An error occurred while resolve DNS for host [d2e1fb2e-6355-414e-a387-3be693056d55.local]
[2020-07-16 18:12:18.612] W 24645 Socket | socket_address.cpp:84   | An error occurred: d2e1fb2e-6355-414e-a387-3be693056d55.local:51355
[2020-07-16 18:12:23.894] I 24645 Monitor | stream_metrics.cpp:144  | A session has been stopped playing #default#app2/stream1 on the WebRTC publihser. Concurrent Viewers[WebRTC(0)/Stream total(0)/App total(0)]
[2020-07-16 18:12:23.894] I 24645 Signalling | rtc_signalling_server.cpp:237  | Client is disconnected: <WebSocketClient: 0x7efc1ee06280, <ClientSocket: 0x7efc203d8290, #4, state: 4, TCP, 181.167.1.77:62600>> (#default#app2 / stream1, ufrag: local: dB8eCg, remote: i5RO)
getroot commented 4 years ago

OK, I'm going to create the same environment next week to test and reproduce the problem.

nangavar commented 4 years ago

I will try too and let you know. In the last test I wrote to you I was running master branch at the edge but 0.10.4 at origin (sorry) Now I have both in the same version. I will keep trying to reproduce the problem and let you know.

Many thanks for everything! You're the bests guys!

nangavar commented 4 years ago

Please, check #148. A stream created by edge OVT is deleted after 60 secs by origin. Only will persist if there is a viewer connected to same stream at origin.

nangavar commented 4 years ago

Running master on edge and 0.10.5 at origin the problem is solved, but the stream is never deleted by origin, it stills always available, even if edge stopped the stream. Origin is not aware that edge has stopped the stream.

Abstract:

Remember my configuration ENCODER (RTSP) -> ORIGIN OME (OVT) -> EDGE OME (WEBRTC) -> PLAYER

nangavar commented 4 years ago

Hi! I found in v0.10.5 that OME never deletes the stream, not only if it was created through a OVT request as I said before. RTSP-Pulling stream is never deleted, even if no viewers are connected by WebRTC or OVT.

getroot commented 4 years ago

I fixed this. (a3f7824554f741a4e75b6ac5a1c83c7f6eb7e5ec)

Test with the master branch.

When playing from edge to "(rtsp)-origin-(ovt)-edge" structure, origin pulls rtsp and provides it as ovt. The stream is not deleted while playing continues at the edge. If there are 0 viewers on the edge, the stream is deleted from the edge after 60 seconds. And again, after 60 seconds, the stream is deleted from origin.

nangavar commented 4 years ago

Hello Getroot! Sorry for my delay. Now it is OK, both are closing connections well. And the origin relay edge is working fine too.

Many thanks!