search

AirenSoft / OvenMediaEngine

OvenMediaEngine (OME) is a Sub-Second Latency Live Streaming Server with Large-Scale and High-Definition. #WebRTC #LLHLS
https://airensoft.com/ome.html
GNU Affero General Public License v3.0
2.61k stars 1.06k forks source link

Crash when rtmp stream is closed if a websocket client connection is open #56

Closed rubu closed 4 years ago

rubu commented 5 years ago

I have a setup where i publish rtmp feed with ffmpeg and occasionally I encountered segmentation faults when killing ffmpeg.

I traced down the issue to the following - when a stream is closed the RtcApplication::DeleteStream iterates through the session map from the stream:

    for(auto const &x : stream->GetAllSessions())
    {
        auto session = std::static_pointer_cast<RtcSession>(x.second);

        // IcePort에 모든 Session 삭제
        _ice_port->RemoveSession(session);

        // Signalling에 모든 Session 삭제
        _rtc_signalling->Disconnect(GetName(), stream->GetName(), session->GetPeerSDP());
    }

But calling RtcSignallingServer::Disconnect can boil down to calling WebRtcPublisher::OnStopCommand which will call Stream::RemoveSession(int) (see stack trace below), thus the stream map in the stream is touched while the for loop is still in progress thus invalidating the underlying iterators in the loop.

Consider this stack trace:

Stream::RemoveSession(int) (/Users/rudolfs/Documents/GitHub/OvenMediaEngine/src/projects/base/publisher/stream.cpp:233)
WebRtcPublisher::OnStopCommand(ov::String const&, ov::String const&, std::__1::shared_ptr<SessionDescription> const&, std::__1::shared_ptr<SessionDescription> const&) (/Users/rudolfs/Documents/GitHub/OvenMediaEngine/src/projects/webrtc/webrtc_publisher.cpp:183)
RtcSignallingServer::DispatchStop(std::__1::shared_ptr<RtcSignallingServer::RtcSignallingInfo>&) (/Users/rudolfs/Documents/GitHub/OvenMediaEngine/src/projects/rtc_signalling/rtc_signalling_server.cpp:904)
RtcSignallingServer::InitializeWebSocketServer()::$_3::operator()(std::__1::shared_ptr<WebSocketClient> const&) const (/Users/rudolfs/Documents/GitHub/OvenMediaEngine/src/projects/rtc_signalling/rtc_signalling_server.cpp:234)
decltype(std::__1::forward<RtcSignallingServer::InitializeWebSocketServer()::$_3&>(fp)(std::__1::forward<std::__1::shared_ptr<WebSocketClient> const&>(fp0))) std::__1::__invoke<RtcSignallingServer::InitializeWebSocketServer()::$_3&, std::__1::shared_ptr<WebSocketClient> const&>(RtcSignallingServer::InitializeWebSocketServer()::$_3&, std::__1::shared_ptr<WebSocketClient> const&) (/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/type_traits:4361)
void std::__1::__invoke_void_return_wrapper<void>::__call<RtcSignallingServer::InitializeWebSocketServer()::$_3&, std::__1::shared_ptr<WebSocketClient> const&>(RtcSignallingServer::InitializeWebSocketServer()::$_3&, std::__1::shared_ptr<WebSocketClient> const&) (/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/__functional_base:349)
std::__1::__function::__alloc_func<RtcSignallingServer::InitializeWebSocketServer()::$_3, std::__1::allocator<RtcSignallingServer::InitializeWebSocketServer()::$_3>, void (std::__1::shared_ptr<WebSocketClient> const&)>::operator()(std::__1::shared_ptr<WebSocketClient> const&) (/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/functional:1527)
std::__1::__function::__func<RtcSignallingServer::InitializeWebSocketServer()::$_3, std::__1::allocator<RtcSignallingServer::InitializeWebSocketServer()::$_3>, void (std::__1::shared_ptr<WebSocketClient> const&)>::operator()(std::__1::shared_ptr<WebSocketClient> const&) (/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/functional:1651)
std::__1::__function::__value_func<void (std::__1::shared_ptr<WebSocketClient> const&)>::operator()(std::__1::shared_ptr<WebSocketClient> const&) const (/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/functional:1799)
std::__1::function<void (std::__1::shared_ptr<WebSocketClient> const&)>::operator()(std::__1::shared_ptr<WebSocketClient> const&) const (/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/functional:2347)
WebSocketInterceptor::OnHttpClosed(std::__1::shared_ptr<HttpRequest> const&, std::__1::shared_ptr<HttpResponse> const&) (/Users/rudolfs/Documents/GitHub/OvenMediaEngine/src/projects/http_server/interceptors/web_socket/web_socket_interceptor.cpp:240)
HttpServer::DisconnectInternal(std::__1::shared_ptr<HttpClient>) (/Users/rudolfs/Documents/GitHub/OvenMediaEngine/src/projects/http_server/http_server.cpp:362)
HttpServer::Disconnect(std::__1::function<bool (std::__1::shared_ptr<HttpClient> const&)>) (/Users/rudolfs/Documents/GitHub/OvenMediaEngine/src/projects/http_server/http_server.cpp:309)
RtcSignallingServer::Disconnect(ov::String const&, ov::String const&, std::__1::shared_ptr<SessionDescription> const&) (/Users/rudolfs/Documents/GitHub/OvenMediaEngine/src/projects/rtc_signalling/rtc_signalling_server.cpp:292)
RtcApplication::DeleteStream(std::__1::shared_ptr<StreamInfo>) (/Users/rudolfs/Documents/GitHub/OvenMediaEngine/src/projects/webrtc/rtc_application.cpp:72)
Application::OnDeleteStream(std::__1::shared_ptr<StreamInfo>) (/Users/rudolfs/Documents/GitHub/OvenMediaEngine/src/projects/base/publisher/application.cpp:69)
MediaRouteApplication::OnDeleteStream(std::__1::shared_ptr<MediaRouteApplicationConnector>, std::__1::shared_ptr<StreamInfo>) (/Users/rudolfs/Documents/GitHub/OvenMediaEngine/src/projects/media_router/media_route_application.cpp:303)
MediaRouteApplicationConnector::DeleteStream(std::__1::shared_ptr<StreamInfo>) (/Users/rudolfs/Documents/GitHub/OvenMediaEngine/src/projects/base/media_route/media_route_application_connector.h:45)
TranscodeStream::DeleteStreams() (/Users/rudolfs/Documents/GitHub/OvenMediaEngine/src/projects/transcode/transcode_stream.cpp:664)
TranscodeStream::DecodeTask() (/Users/rudolfs/Documents/GitHub/OvenMediaEngine/src/projects/transcode/transcode_stream.cpp:577)
decltype(*(std::__1::forward<TranscodeStream*>(fp0)).*fp()) std::__1::__invoke<void (TranscodeStream::*)(), TranscodeStream*, void>(void (TranscodeStream::*&&)(), TranscodeStream*&&) (/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/type_traits:4302)
void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (TranscodeStream::*)(), TranscodeStream*, 2ul>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (TranscodeStream::*)(), TranscodeStream*>&, std::__1::__tuple_indices<2ul>) (/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/thread:342)
void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (TranscodeStream::*)(), TranscodeStream*> >(void*) (/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/thread:352)
_pthread_body (@_pthread_body:34)
_pthread_start (@_pthread_start:23)
thread_start (@thread_start:7)

On the top of the stack we are in tream::RemoveSession(session_id_t id), which erases the stream, while RtcApplication::DeleteStream() is still in the loop for(auto it = sessions.begin(); it != sessions.end();).

I'll try to provide a PR for this:)

dimiden commented 5 years ago

@rubu Thank you very much! Since we are currently doing a major modification, the PR you sent will be applied later to avoid merge-conflict as much as possible.

Thank you again for your contribution.

rubu commented 5 years ago

Ok, I'd really like to get in touch to understand more what is going on in the dev branch and what changes will happen, since we have done a lot of features on the master that we could contribute (like stream bypass, reworked the fragmentation header to allow more than 3 fragments in a packet etc).

dimiden commented 5 years ago

@rubu There is a lot of work on the OME development team, including MPEG-TS input, LL-DASH stabilization, and Origin-Edge structure rewriting. Also, our next goal is to merge all of these into the master quickly. Therefore, if you send PR before that, we will merge together later.

Thank you very much for your attention!

getroot commented 4 years ago

This issue has been closed since it has been inactive for quite some time. If you want to continue discussing this issue, please feel free to reopen it.