[REQUEST] Tutorial on how to connect two instances over IP

[ZeoWorks]

HI, it would be amazing if you could a sample and tutorial as how to connect two instances through public IP. Thanks.

[karasusan]

@ZeoWorks Is this document helpful for you? https://docs.unity3d.com/Packages/com.unity.renderstreaming@3.1/manual/turnserver.html

Some users posted questions that how to connect peers over public network. We are considering the good way to tell them.

[ZeoWorks]

Thanks, this isn't a helpful document unfortunately.

I seen #539 also is trying to set SDP over a RPC call. I'd love to learn the method as to how to make a direct connection between two IP's.

[karasusan]

@ZeoWorks Basically, it works only exchanging SDPs between peers in local network. However, in the public network, we need to take an action in accordance with network environments.

For example, we might make the docker container image for sample and write document how to deploy the image to famous cloud platform like GCP and AWS. Is that same you are expecting?

[ZeoWorks]

@ZeoWorks Basically, it works only exchanging SDPs between peers in local network. However, in the public network, we need to take an action in accordance with network environments.

For example, we might make the docker container image for sample and write document how to deploy the image to famous cloud platform like GCP and AWS. Is that same you are expecting?

What I'm hoping for is to exchange the SDP's over a UDP call (With UNET or Mirror for instance) so machines can connect over the internet.

I've attached the code thus far. It's fairly basic at the moment, I'm basically storing the SDP offer/answer data into an input field and copy/pasting the SDP offer/answer data from one instance to another. One instance has "isHost" bool enabled, the other does not.

It does not transfer the video feed successfully. Any help would be appreciated. :)

`using System.Collections; using System.Collections.Generic; using UnityEngine; using Unity.WebRTC; using UnityEngine.UI;

public class WebRTCTest : MonoBehaviour {

public InputField inputField;

public delegate void onRTCSessionDescriptionCallback(RTCSessionDescription sdp);

[SerializeField] private RawImage RtImage;
[SerializeField] private Vector2Int streamSize = new Vector2Int(640, 480);

private RTCPeerConnection peerConnection;

private DelegateOnTrack DelegateOnTrack;
public onRTCSessionDescriptionCallback OnOffer;

private RTCSessionDescription answerSdp;
private RTCSessionDescription offerSdp;
private MediaStream sourceVideoStream, receiveVideoStream;

private RTCConfiguration configuration = new RTCConfiguration
{
    iceServers = new[] { new RTCIceServer { urls = new[] { "stun:stun.l.google.com:19302" } } }
};

private void OnEnable()
{
    WebRTC.Initialize(EncoderType.Software);
}

public bool isHost;
private void Start()
{
    peerConnection = new RTCPeerConnection(ref configuration);
    peerConnection.OnConnectionStateChange = state =>
    {
        Debug.Log($"Connection state change {state}");
    };

    peerConnection.OnIceCandidate = candidate =>
    {
        Debug.Log($"CANDIDATE: {candidate}");
    };

    OnOffer = desc =>
    {
        offerSdp = desc;
    };

    if (isHost)
    {
        sourceVideoStream = Camera.main.CaptureStream(1280, 720, 1000000);

        foreach (var track in sourceVideoStream.GetTracks())
        {
            peerConnection.AddTrack(track, sourceVideoStream);
        }

        StartCoroutine(CreateOfferAsync());
    }
    else
    {
        receiveVideoStream = new MediaStream();
        receiveVideoStream.OnAddTrack = e =>
        {
            if (e.Track is VideoStreamTrack track)
            {
                RtImage.texture = track.InitializeReceiver(1280, 720);
            }
        };
        peerConnection.OnTrack = e => receiveVideoStream.AddTrack(e.Track);
    }
    StartCoroutine(WebRTC.Update());

}

//When pressing a button in the scene, this is activated.
public void TestAnswer()
{
    if (!isHost)
        StartCoroutine(CreateAnswer());
    else StartCoroutine(SetAnswer());
}

private IEnumerator CreateOfferAsync()
{
    Debug.Log("Create offer");
    var opCreateOffer = peerConnection.CreateOffer();
    yield return opCreateOffer;

    if (opCreateOffer.IsError)
    {
        yield break;
    }

    inputField.text = opCreateOffer.Desc.sdp;
    var offer = new RTCSessionDescription { type = RTCSdpType.Offer, sdp = opCreateOffer.Desc.sdp };
    Debug.Log($"Modified Offer from LocalPeerConnection\n{offer.sdp}");

    var opLocal = peerConnection.SetLocalDescription(ref offer);
    yield return opLocal;

    if (opLocal.IsError)
    {
        print("Error");
        //OnSetSessionDescriptionError(opLocal.Error);
        yield break;
    }
}

private IEnumerator CreateAnswer()
{
    var offer = new RTCSessionDescription { type = RTCSdpType.Offer, sdp = inputField.text };

    var opRemote = peerConnection.SetRemoteDescription(ref offer);
    yield return opRemote;

    if (opRemote.IsError)
    {
        print("Error!");
        yield break;
    }

    Debug.Log("Set Remote session description success on RemotePeerConnection");

    var op = peerConnection.CreateAnswer();
    yield return op;

    if (op.IsError)
    {
        print("Error2");
        yield break;
    }

    inputField.text = op.Desc.sdp;

    var answer = new RTCSessionDescription { type = RTCSdpType.Answer, sdp = inputField.text };
    Debug.Log($"Modified Answer from RemotePeerConnection\n{answer.sdp}");

    var opLocal = peerConnection.SetLocalDescription(ref answer);
    yield return opLocal;

    if (opLocal.IsError)
    {
        print("Error 2");
        //OnSetSessionDescriptionError(opLocal.Error);
        yield break;
    }
    //answerSdpInput.text = op.Desc.sdp;
    //  answerSdpInput.interactable = true;
}

IEnumerator SetAnswer()
{
    var answer = new RTCSessionDescription { type = RTCSdpType.Answer, sdp = inputField.text };
    Debug.Log($"Modified Answer from RemotePeerConnection\n{answer.sdp}");

    var opLocal = peerConnection.SetRemoteDescription(ref answer);
    yield return opLocal;

    if (opLocal.IsError)
    {
        print("Error!");
        yield break;
    }

    Debug.Log("Set Remote session description success on LocalPeerConnection");
}

// Update is called once per frame
void Update()
{

}

void handleSendChannelStatusChange()
{
    print("EEE");
}

} `

[unnanego]

erm, I've spent days trying to make it work over the internet and only here I find out that it only works over local network?! WHY DOESN'T THE INSTRUCTION STATE THIS ANYWHERE?!

[maeln]

Hey, hello, I am the one who created #539 . I cannot really share my code but I can tell you how we did it : Firstly, you not going to be able to avoid the whole SDP exchange and candidate exchange using trickle ice. In our case the two client where in different tech, one was in Unity and the other a custom peer made with Go and the lib Pion. They both communicate using JSON-RPC over Websockets. But this doesn't really matter, it can be udp socket, it can be http, as long as you got a way of sending messages back and forth.

The first thing you want to do is create the offer SDP, which you do in your CreateOfferAsync function. Get the SDP text (opCreateOffer.Desc.sdp;) and send it to your other client via your choosen method of communication (in my case, again, json-rpc over websocket). You other client you take this sdp, create the answer and send it to your client. You will need to create a RTCSessionDescription from the SDP answer text and add it to your RTCPeerConnection. For reference, here how we dot it (never mind the deserialization code) :

bool receiveOffer(JsonRpcReturnData ret)
{
    Debug.Log("Received Offer");
    string offerString = Encoding.UTF8.GetString(Convert.FromBase64String((string)ret.result));
    SdpOffer answer = JsonConvert.DeserializeObject<SdpOffer>(offerString);
    Debug.Log(answer.type);
    Debug.Log(answer.sdp);

    // This is the important part, we create the RTCSessionDescription object with the SDP text we received, i.e answer.sdp
    SdpAnswer = new RTCSessionDescription();
    SdpAnswer.type = RTCSdpType.Answer;
    SdpAnswer.sdp = answer.sdp;

    // You NEED to set the remote description on your peer with this!
    peerConnection.SetRemoteDescription(ref SdpAnswer);
    return true;
}

Important : I wasn't really able to exchange candidate directly in the SDP, in anycase, this is now deprecated and pretty much everything is moving to trickle ice. So my suggestion is, whatever client is used to generate the answer, it should send an answer with no candidate and let the candidate exchange to another part.

Which is exactly what we do. Once our Go client sent the sdp answer to unity, it start sending SDP candidate through the rpc system to the unity client. Our job is easier here because the Go client run on a server with a stable IP address opened to the internet, therefore we can send host candidate which mean we don't have to setup any stun, nat punching and other stuff. So you get those candidate from your rpc system, and then you need to add them manually to your peer connection, and thats pretty much it, the rest should be handle by the library itself. Again for reference, here is the code that is called whenever we receive a candidate through our RPC system :

 bool receiveWebRtcCandidate(JsonRpcRequestData rq)
{
    Debug.Log("Received candidate");

    WebRtcCandidate candidate = rq.parameters.ToObject<WebRtcCandidate>();
    RTCIceCandidateInit init = new RTCIceCandidateInit();
    // candidate.candidate here is a string RFC5245 like : candidate:1467250027 1 udp 2122260223 192.168.0.196 46243 typ host generation 0
    init.candidate = candidate.candidate;
    // Note: I am not sure that sdpMid and MLineIndex are essential and might be left unititialised
    init.sdpMid = candidate.sdpMid;
    init.sdpMLineIndex = int.Parse(candidate.sdpMLineIndex);
    RTCIceCandidate iceCandidate = new RTCIceCandidate(init);
    peerConnection.AddIceCandidate(iceCandidate);
    return true;
}

TL;DR: You can absolutely connect two client "over ip" using your own protocol, you just need to handle the SDP exchange and the candidate exchange (and if you want to do "over IP" you can probably just use host candidate) manually through your own protocol.

Thats it, if you have any question I can try answering them.

[karasusan]

@maeln @unnanego @ZeoWorks

Hi, I am not sure how can we help you. What do we do for solving your problems?

For example, we can make docker image contains Unity runtime to stream video/audio. Do you think It it solve your issue?

Or, should we provide services for traversing NAT using TURN/STUN server?

[unnanego]

@karasusan sorry for late reply, I'm working on a different project now, so can't work on this yet. I managed to implement it using the provided examples in the editor, but when I tried the same code, but on different clients (I'm sending the SDP data using a web server and using the same Google stun server, the clients receive the descriptions, but the status of the connection freezes at initializing. And I don't understand what to do next - the only difference with it running on the same client is that it's different clients, running on the same machine, but still no connection.

[karasusan]

@unnanego Sure. Could you share me your sample to reproduce issues?

[unnanego]

I will as soon as I get to it, sorry, too busy now. Thank you for helping anyway! Merry Christmas! =)

[unnanego]

After sitting on it for a few hours, I made it work! At least on the same wifi...

[karasusan]

I am waiting for your reply for this comment and we want to close this if there is no answer.