Copy and Paste troubles

[GerbenHettinga]

I'm trying to use this library in a simple copy-paste situation such as set out in the example of https://github.com/paullouisageneau/libdatachannel/tree/master/examples/copy-paste

Instead of reading from std::cin I copy paste offers and answers into html text boxes. However I'm unable to setup a connection in this manner. I always get a "WebRTC: ICE failed, add a TURN server and see about:webrtc for more details" on Firefox or Chrome never seems to progress further than the [State: Connecting] state.

Here is the main code adapted from the libdatachannel example:

std::shared_ptr<rtc::PeerConnection> pc; 
std::shared_ptr<rtc::DataChannel> dc;

EM_JS(void, putDescriptionInOfferBox, (const char* string_pointer), {
    document.getElementById('offer').value = UTF8ToString(string_pointer);  
});

EM_JS(void, putCandidateInCandidateBox, (const char* string_pointer), {
    document.getElementById('candidate').value = UTF8ToString(string_pointer);  
});

EM_JS(void, addChatMessage, (const char* string_pointer), {
    var out = document.getElementById('output');
    out.value = UTF8ToString(string_pointer);  
});

extern "C" {
    void createOffer() {
        dc = pc->createDataChannel("test2");

        dc->onOpen([&]() { std::cout << "[DataChannel open: " << dc->label() << "]" << std::endl; });

        dc->onClosed([&]() { std::cout << "[DataChannel closed: " << dc->label() << "]" << std::endl; });

        dc->onMessage([](std::variant<rtc::binary, rtc::string> message) {
            if (std::holds_alternative<rtc::string>(message)) {
                rtc::string sMessage = std::get<rtc::string>(message);
                std::cout << "[Received message: " << sMessage << "]" << std::endl;
                addChatMessage(sMessage.c_str());
            }
        });
    }
}

extern "C" {
    void addRemoteDescriptionOffer(char* cc) {
        std::string sdp = std::string(cc);
        pc->setRemoteDescription(rtc::Description(sdp, rtc::Description::Type::Offer));
    }
}

extern "C" {
    void addRemoteDescriptionAnswer(char* cc) {
        std::string sdp = std::string(cc);
        pc->setRemoteDescription(rtc::Description(sdp, rtc::Description::Type::Answer));
    }
}

extern "C" {
    void addRemoteCandidate(char* cc) {
        std::string candidate = std::string(cc);
        std::cout << candidate << std::endl;
        pc->addRemoteCandidate(rtc::Candidate(candidate, "0"));
    }
}

extern "C" {
    void sendMessage(char* cc) {
        if (!dc->isOpen()) {
            std::cout << "** Channel is not Open ** " << std::endl;
        } else {
            std::cout << "[Message]: ";
            std::string message(cc);
            dc->send(message);
        }
    }   
}

inline const char* toString(rtc::PeerConnection::State s)
{
    switch (s)
    {
        case rtc::PeerConnection::State::New:   return "New";
        case rtc::PeerConnection::State::Connecting:   return "Connecting";
        case rtc::PeerConnection::State::Disconnected: return "Disconnected";
        case rtc::PeerConnection::State::Failed: return "Failed";
        case rtc::PeerConnection::State::Closed: return "Closed";
        default:      return "Disconnected";
    }
}

inline const char* toString(rtc::PeerConnection::GatheringState s)
{
    switch (s)
    {
        case rtc::PeerConnection::GatheringState::New:   return "New";
        case rtc::PeerConnection::GatheringState::InProgress:   return "InProgress";
        case rtc::PeerConnection::GatheringState::Complete: return "Complete";
        default:      return "Disconnected";
    }
}

int main(int, char**)
{
    rtc::Configuration config;
    config.iceServers.emplace_back("stun:stun.1.google.com:19302");

    pc = std::make_shared<rtc::PeerConnection>(config);

    pc->onLocalDescription([](rtc::Description description) {
        putDescriptionInOfferBox(rtc::string(description).c_str());
    });

    pc->onLocalCandidate([](rtc::Candidate candidate) {
        putCandidateInCandidateBox(rtc::string(candidate).c_str());
    });

    pc->onStateChange(
        [](rtc::PeerConnection::State state) { std::cout << "[State: " << toString(state) << "]" << std::endl;});

    pc->onGatheringStateChange([](rtc::PeerConnection::GatheringState state) {
        std::cout << "[Gathering State: " << toString(state) << "]" << std::endl;
    });

    pc->onDataChannel([&](std::shared_ptr<rtc::DataChannel> _dc) {
        if(!dc) { //only the answerer has to create a new dc object
            std::cout << "[Got a DataChannel with label: " << _dc->label() << "]" << std::endl;
            dc = _dc;

            dc->onClosed([&]() { std::cout << "[DataChannel closed: " << dc->label() << "]" << std::endl; });

            dc->onMessage([](std::variant<rtc::binary, rtc::string> message) {
                if (std::holds_alternative<rtc::string>(message)) {
                    rtc::string sMessage = std::get<rtc::string>(message);
                    std::cout << "[Received message: " << sMessage << "]" << std::endl;
                    addChatMessage(sMessage.c_str());
                }
            });
        }
    });

    emscripten_set_main_loop_arg(main_loop, nullptr, 0, true);
}

static void main_loop(void* arg)
{

}

and the html front end

<html lang="en-us">
  <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, minimum-scale=1, user-scalable=no"/>
    <title>WebRTC - Copy and Paste</title>
    <style>
        body { margin: 0; background-color: powderblue; }
        .emscripten {
            position: relative;
            top: 0px;
            left: 0px;
            margin: 0px;
            border: 0;
            /*width: 100%; //this will set canvas size full screen
            height: 100%;*/
            overflow: hidden;
            display: block;
            image-rendering: optimizeSpeed;
            image-rendering: -moz-crisp-edges;
            image-rendering: -o-crisp-edges;
            image-rendering: -webkit-optimize-contrast;
            image-rendering: optimize-contrast;
            image-rendering: crisp-edges;
            image-rendering: pixelated;
            -ms-interpolation-mode: nearest-neighbor;
        }
    </style>
  </head>
  <body>
    <script>
      function setDescriptionOffer() {
        var s = document.getElementById('answerdescription').value;
        var ptr = allocate(intArrayFromString(s), ALLOC_NORMAL);

        _addRemoteDescriptionOffer(ptr);

        _free(ptr)
      }

      function setCandidate() {
        var s = document.getElementById('answercandidate').value;
        var ptr = allocate(intArrayFromString(s), ALLOC_NORMAL);

        _addRemoteCandidate(ptr);

        _free(ptr)
      }
    </script>

    <p id=stat></p>
    <button id="button" onclick="_createOffer()">Offer:</button>
    <textarea id="offer"></textarea> 
    <textarea id="candidate"></textarea><br> 

    <button id="buttonOffer" onclick="setDescriptionOffer()">set Offer:</button>
    <textarea id="answerdescription" placeholder="Paste Description. And press Enter"></textarea>

    <button id="buttonCandidate" onclick="setCandidate()">set Candidate:</button> 
    <textarea id="answercandidate" placeholder="Paste Candidate. And press Enter"></textarea><br>

    <br>
    <br>
    Chat: <input id="chat"><br>
      <pre id="output">Chat: </pre>

    <script>
      chat.onkeypress = function(e) {
        if (e.keyCode != 13) return;
        var s = chat.value;
        var ptr = allocate(intArrayFromString(s), ALLOC_NORMAL);

        _sendMessage(ptr);

        _free(ptr)
        console.log(chat.value);
        chat.value = "";
      };
    </script>
    {{{ SCRIPT }}}
  </body>
</html>

I use it in the following manner. I have two tabs. The 'offerer' tab creates the initial data connection and offer, which are copy pasted to the 'answerer'. Which in turn generates an answer that gets pasted to the offerer. The same thing for the candidates. I built and ran the libdatachannel offerer and answerer example and it seems to be the exact same process and in that case I was able to set up a connection.

One thing that does allow me to make a connection is when I copy the answer from the "about:webrtc" page after pasting in the offer from the offerer. This immediately allows me to setup a connection and creates a functioning datachannel between the tabs without having to specify the candidates manually. This seems to be because the ICE candidates are already enumerated in the localdescription SDP. When I call WEBRTC.peerConnection[1].localDescription.sdp from the console I also get this version of the answer. However, I am unable to obtain the same localdescription using this library. Is there a way this is easily achieved? Why is the call to WEBRTC.peerConnection[1].localDescription.sdp in the javascript portion of the library giving a different result as when doing it in the console?

Any assistance would be appreciated.

[paullouisageneau]

I think the issue comes from the way you handle candidates: each side outputs one description but multiple candidates. It seems your code will override each candidate by the next in the candidate box and as a result only display the last one.

There is a semantic subtlety in the API: the methods are called setRemoteDescription (as there is only one so it will replace the remote description) and addRemoteCandidate (as it will add one more candidate to the list, not replace existing candidates).

This seems to be because the ICE candidates are already enumerated in the localdescription SDP. When I call WEBRTC.peerConnection[1].localDescription.sdp from the console I also get this version of the answer. However, I am unable to obtain the same localdescription using this library. Is there a way this is easily achieved?

This can be done. You just need to wait for GatheringState to be Complete, then you can retrieve the local description with all candidates by calling pc->localDescription(). This method, called non-trickle ICE, allows for an simpler signaling as it does not require exchanging candidates, only the descriptions, but it could introduce delays if a STUN/TURN server is unreachable.

[GerbenHettinga]

Indeed by appending the candidates to the textbox I was able to set up a working datachannel.

I also now check the GatheringState if it is complete after which the description is automatically put in the textbox.

pc->onGatheringStateChange([](rtc::PeerConnection::GatheringState state) {
std::cout << "[Gathering State: " << toString(state) << "]" << std::endl;

if(state == rtc::PeerConnection::GatheringState::Complete) {
    std::optional<rtc::Description> desc = pc->localDescription();
    if(desc.has_value()) {
        putDescriptionInOfferBox(rtc::string(desc.value()).c_str());
    }
}
});

[guest271314]

Did you publish the working example?

[paullouisageneau]

@paullouisageneau could you explain that into a little more detail?

The original issue was that only the last candidate was displayed in the candidate textbox, other candidates were dropped. This is unrelated to what you experience here as your example is not copy paste, and you don't trickle candidates, you wait for gathering complete to send descriptions with candidates.

I tried to make a similar connection establishment based on your copy-paste example, with the only difference that I use a WebSocket server and no copy/paste. My connection works on Firefox, but not on Chrome (same issue as above).

My code in a simplified version: [...]

The code has a bunch of issues but I don't know if it's because it's "simplified" or not: pc is a local object so it will be destroyed as soon as you leave the init_peer_connection() function, the type is not sent over the websocket but it is received it somehow, anser is written instead of anwser, a callback is set on dc instead of _dc...

You have to debug your code and check you transmit the descriptions back and forth correctly. Note it is not good practice to assume and hardcode description type, instead you should get it from description.typeString() and transmit it.

[tbuchs]

Hi, I am doing it in a very similar way (also tried the exact same program) and it works perfectly fine in Firefox. However in Chrome, it does not output any Candidates and also if setting a candidate from firefox it says: OperationError: Failed to execute 'addIceCandidate' on 'RTCPeerConnection': Error processing ICE candidate.

Is this an error on my side? I found a Stackoverflow discussion saying, that you need to call setLocalDescription(), but this isn't implemented afaik

[paullouisageneau]

Does Chrome generate a description? The error indeed means setLocalDescription() was not called so you can't add candidates. The wrapper calls it on both createDataChannel() and setRemoteDescription(), you should check they are indeed called and don't fail.

[tbuchs]

Interesting, didn't see that. Yes, it seems to be working, as if i log in the function handleDescription the variable peerConnection.localDescription.sdp it outputs 116600

[paullouisageneau]

Yes, it seems to be working, as if i log in the function handleDescription the variable peerConnection.localDescription.sdp it outputs 116600

This doesn't look right. If localDescription is set, sdp should contain the SDP description.

[tbuchs]

You are right, it contains the SDP description - this was just me looking at the wrong value. But I now found out that the function peerConnection.onicecandidate get called, however, the candidates of if(evt.candidate && evt.candidate.candidate) equal null and therefore the function handleCandidate doesn't get called

[guest271314]

@tbuchs FWIW here are a few ways I have implemented "copy/paste"/"signaling server" programmatically, and programmatically with user action

• https://gist.github.com/guest271314/04a539c00926e15905b86d05138c113c
• https://github.com/guest271314/offscreen-webrtc/blob/main/background.js
• https://github.com/guest271314/telnet-client/blob/user-defined-tcpsocket-controller-web-api/direct-sockets/direct-socket-controller.js with https://github.com/guest271314/telnet-client/blob/user-defined-tcpsocket-controller-web-api/assets/script.js

and the last time I did this with libdatachannel

• https://github.com/paullouisageneau/libdatachannel/issues/544#issuecomment-1027570821 I don't think I've tried with libdatachannel-wasm, yet.

[paullouisageneau]

You are right, it contains the SDP description - this was just me looking at the wrong value. But I now found out that the function peerConnection.onicecandidate get called, however, the candidates of if(evt.candidate && evt.candidate.candidate) equal null and therefore the function handleCandidate doesn't get called

Null candidate means all candidates have been gathered. This would mean that the browser fails to gather any candidates, not even host ones. Does WebRTC work as expected in the browser? There could be a privacy extension interfering with it.

also if setting a candidate from firefox it says: OperationError: Failed to execute 'addIceCandidate' on 'RTCPeerConnection': Error processing ICE candidate.

Actually there are two different issues as this means the remote description is not set. Is setRemoteDescription() called before attempting to add candidates?

[tbuchs]

You are right, it contains the SDP description - this was just me looking at the wrong value. But I now found out that the function peerConnection.onicecandidate get called, however, the candidates of if(evt.candidate && evt.candidate.candidate) equal null and therefore the function handleCandidate doesn't get called

Null candidate means all candidates have been gathered. This would mean that the browser fails to gather any candidates, not even host ones. Does WebRTC work as expected in the browser? There could be a privacy extension interfering with it.

That was a very good hint, thanks. It is indeed working with chrome in incognito mode - even though, I don't know the problem, since I have disabled all extensions and it is still not working

also if setting a candidate from firefox it says: OperationError: Failed to execute 'addIceCandidate' on 'RTCPeerConnection': Error processing ICE candidate.

Actually there are two different issues as this means the remote description is not set. Is setRemoteDescription() called before attempting to add candidates?

Yes, it gets called before