"double free or corruption" error with "Aborted (core dumped)" on multiple stream hangup

[abhikandoi]

On concurrently hanging up 8 streams (all of them have the same streamLabel), I get the following error:

... more lines omitted ...
[033:316] [343] (PeerConnectionManager.h:188): virtual PeerConnectionManager::PeerConnectionObserver::~PeerConnectionObserver()
[033:316] [342] (PeerConnectionManager.cpp:656): herehere123
[033:316] [341] (PeerConnectionManager.cpp:656): herehere123
[033:316] [340] (PeerConnectionManager.cpp:658): herehere1233124
[033:316] [343] (PeerConnectionManager.cpp:637): done peerConnection->Close()
[033:317] [340] (PeerConnectionManager.cpp:661): herehere3
[033:317] [343] (PeerConnectionManager.cpp:646): Close PeerConnection no more used ipc:///tmp/jpg-1.ipc
[033:317] [343] (PeerConnectionManager.cpp:648): herehere
[033:317] [343] (PeerConnectionManager.cpp:652): herehere2
[033:317] [343] (PeerConnectionManager.cpp:661): herehere3
[033:317] [343] (PeerConnectionManager.cpp:668): herehere4
[033:317] [343] (PeerConnectionManager.cpp:671): herehere5
[033:317] [343] (PeerConnectionManager.cpp:673): herehere6
[033:317] [343] (PeerConnectionManager.cpp:676): herehere7
[033:317] [343] (PeerConnectionManager.cpp:679): herehere8
[033:317] [343] (PeerConnectionManager.cpp:682): herehere9
[033:317] [343] (PeerConnectionManager.cpp:686): herehere10
[033:317] [332] (peerconnection.cc:738): Session: 3538888922215514852 is destroyed.
[033:317] [343] (PeerConnectionManager.cpp:690): herehere11
[033:317] [343] (PeerConnectionManager.cpp:695): herehere12
answer:true

[033:317] [340] (PeerConnectionManager.cpp:668): herehere4
[033:317] [340] (PeerConnectionManager.cpp:671): herehere5
*** Error in ./webrtc_streamer: double free or corruption (fasttop): 0x00007ff9b8006f30 ***
terminate called after throwing an instance of Aborted (core dumped)

I've added log statements to help with debugging. This is how the function hangUp looks like:

const Json::Value PeerConnectionManager::hangUp(const std::string &peerid)
{
    bool result = false;
    RTC_LOG(INFO) << __FUNCTION__ << " " << peerid;

    std::map<std::string, PeerConnectionObserver* >::iterator  it = peer_connectionobs_map_.find(peerid);
    if (it != peer_connectionobs_map_.end())
    {
        RTC_LOG(LS_ERROR) << "Close PeerConnection";
        PeerConnectionObserver* pcObserver = it->second;
        rtc::scoped_refptr<webrtc::PeerConnectionInterface> peerConnection = pcObserver->getPeerConnection();
        peer_connectionobs_map_.erase(it);

        rtc::scoped_refptr<webrtc::StreamCollectionInterface> localstreams (peerConnection->local_streams());
        Json::Value streams;

        RTC_LOG(LS_ERROR) << "peerConnection->Close()";
        peerConnection->Close();
        delete pcObserver;
        RTC_LOG(LS_ERROR) << "done peerConnection->Close()";

        for (unsigned int i = 0; i<localstreams->count(); i++)
        {
            std::string streamLabel = localstreams->at(i)->label();

            bool stillUsed = this->streamStillUsed(streamLabel);
            if (!stillUsed)
            {
                RTC_LOG(LS_ERROR) << "Close PeerConnection no more used " << streamLabel;
                std::map<std::string, rtc::scoped_refptr<webrtc::MediaStreamInterface> >::iterator it = stream_map_.find(streamLabel);
                RTC_LOG(LS_ERROR) << "herehere";
                if (it != stream_map_.end())
                {

                    RTC_LOG(LS_ERROR) << "herehere2";
                    // remove video tracks
                    while (it->second->GetVideoTracks().size() > 0)
                    {
                        RTC_LOG(LS_ERROR) << "herehere123";
                        it->second->RemoveTrack(it->second->GetVideoTracks().at(0));
                        RTC_LOG(LS_ERROR) << "herehere1233124";
                    }

                    RTC_LOG(LS_ERROR) << "herehere3";
                    // remove audio tracks
                    while (it->second->GetAudioTracks().size() > 0)
                    {
                        it->second->RemoveTrack(it->second->GetAudioTracks().at(0));
                    }

                    RTC_LOG(LS_ERROR) << "herehere4";

                    it->second.release();
                    RTC_LOG(LS_ERROR) << "herehere5";
                    stream_map_.erase(it);
                    RTC_LOG(LS_ERROR) << "herehere6";
                }

                RTC_LOG(LS_ERROR) << "herehere7";
            }

            RTC_LOG(LS_ERROR) << "herehere8";
        }

        RTC_LOG(LS_ERROR) << "herehere9";

        result = true;

        RTC_LOG(LS_ERROR) << "herehere10";
    }
    Json::Value answer;

    RTC_LOG(LS_ERROR) << "herehere11";
    if (result) {
        answer = result;
    }

    RTC_LOG(LS_ERROR) << "herehere12";

    return answer;
}

[abhikandoi]

@mpromonet any idea about this?

[mpromonet]

Hi Abhishek, Probably a concurrent access to the map. You can try reducing the number of HTTP threads of civet, adding :

        options.push_back("num_threads");
        options.push_back("1");

This should slow down the HTTP server request processing, but it should avoid concurrent access. Best Regards, Michel.

[mpromonet]

Since introducing mutex to protect the map, I did not reproduce this crash. I close the issue, you could open a new one if you still have the problem after this fix Best Regards Michel