Creating my own WebRTC service

[meeeee12]

I am trying to use this library to create a minimalistic Java WebRTC implementation, however I am stuck at displaying the local video feed. It seems that I need to create a VideoView and use that as a sink for the VideoTrack but I can’t seem to figure out how to do it. I also don't know how to set the local view to mirror mode. Here is what I have so far:

` package WebRTC;

import dev.onvoid.webrtc.*; import dev.onvoid.webrtc.demo.javafx.control.VideoView;

import dev.onvoid.webrtc.media.MediaDevices; import dev.onvoid.webrtc.media.audio.AudioOptions; import dev.onvoid.webrtc.media.audio.AudioSource; import dev.onvoid.webrtc.media.audio.AudioTrack; import dev.onvoid.webrtc.media.video.*;

import javafx.application.Application; import javafx.geometry.Insets; import javafx.scene.Scene; import javafx.scene.layout.VBox; import javafx.stage.Stage;

import java.util.ArrayList; import java.util.List; import java.util.function.Consumer;

import static java.util.Objects.isNull;

public class Main extends Application implements PeerConnectionObserver {

private PeerConnectionFactory factory;

@Override
public void start(Stage primaryStage) throws Exception {

    VBox root = new VBox();
    root.setPadding(new Insets(10));
    root.setSpacing(8);

    VideoView remoteVideoView = new VideoView();
    remoteVideoView.setLayoutX(100);
    remoteVideoView.setLayoutY(100);

    root.getChildren().add(remoteVideoView);

    VideoView localVideoView = new VideoView();
    localVideoView.setLayoutX(100);
    localVideoView.setLayoutY(100);

    root.getChildren().add(localVideoView);

    factory = new PeerConnectionFactory();

    VideoDeviceSource videoSource = new VideoDeviceSource();
    videoSource.setVideoCaptureDevice(MediaDevices.getVideoCaptureDevices().get(0));
    VideoTrack videoTrack = factory.createVideoTrack("videoTrack", videoSource);

    // Don't know how to do this properly or set local view in mirror mode:
    videoTrack.addSink(localVideoView);

    AudioSource audioSource = factory.createAudioSource(new AudioOptions());
    AudioTrack audioTrack = factory.createAudioTrack("audioTrack", audioSource);

    RTCIceServer iceServer = new RTCIceServer();
    iceServer.urls.add("stun:stun.l.google.com:19302");

    RTCConfiguration config = new RTCConfiguration();
    config.iceServers.add(iceServer);
    config.bundlePolicy = RTCBundlePolicy.MAX_BUNDLE;
    config.rtcpMuxPolicy = RTCRtcpMuxPolicy.NEGOTIATE;
    config.iceTransportPolicy = RTCIceTransportPolicy.ALL;

    RTCPeerConnection peerConnection = factory.createPeerConnection(config, this);

    List<String> streamIds = new ArrayList<>();
    streamIds.add("stream-0");

    RTCRtpSender audioSender = peerConnection.addTrack(audioTrack, streamIds);
    RTCRtpSender videoSender = peerConnection.addTrack(videoTrack, streamIds);

    primaryStage.setTitle("WebRTC");
    primaryStage.setScene(new Scene(root, 300, 275));

    primaryStage.show();
}

@Override
public void onIceCandidate(RTCIceCandidate candidate) {

}

@Override
public void onDataChannel(RTCDataChannel dataChannel) {

}

@Override
public void onConnectionChange(RTCPeerConnectionState state) {
    if (state == RTCPeerConnectionState.CONNECTED) {

    }
}

public static void main(String[] args) {
    launch(args);
}

} `

[devopvoid]

Hi @meeeee12

your code requires a bit more work to function properly. But let's start with the video source. To receive video frames from the VideoDeviceSource, you need to start it first, e.g.,

try {
    videoSource.start();
}
catch (Exception e) {
    e.printStackTrace();
}

To use the peer connection, you need to create an offer with it, if you want to initiate the connection setup, e.g,

RTCOfferOptions options = new RTCOfferOptions();

peerConnection.createOffer(options, new CreateSessionDescriptionObserver() {

    @Override
    public void onSuccess(RTCSessionDescription description) {
        peerConnection.setLocalDescription(description, new SetSessionDescriptionObserver() {

            @Override
            public void onSuccess() {
                // Send description as offer.
            }

            @Override
            public void onFailure(String error) {
                System.out.println("Set local RTCSessionDescription failed: " + error);
            }
        });
    }

    @Override
    public void onFailure(String error) {
        System.out.println("Create RTCSessionDescription failed: " + error);
    }
});

Setting the local session description will also trigger void onIceCandidate(RTCIceCandidate candidate). These candidates must also be sent to the remote peer. Eventually, you receive an answer (RTCSessionDescription) from the remote peer which can be set with peerConnection.setRemoteDescription()...

[meeeee12]

Thank you for your help, however I'm still having problems setting up the local video sink. The following line does not compile:

videoTrack.addSink(localVideoView);

[devopvoid]

That's because localVideoView as VideoView is not a VideoTrackSink. Please see the definition of VideoTrack.addSink. You should replace the line in question with:

videoTrack.addSink(localVideoView::setVideoFrame);

[meeeee12]

When I try to run it I receive:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_UNCAUGHT_CXX_EXCEPTION (0xe06d7363) at pc=0x00007ffaf2a84ed9, pid=10400, tid=11364
#
# JRE version: Java(TM) SE Runtime Environment (15.0.1+9) (build 15.0.1+9-18)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (15.0.1+9-18, mixed mode, sharing, tiered, compressed oops, g1 gc, windows-amd64)
# Problematic frame:
# C  [KERNELBASE.dll+0x34ed9]
#
# No core dump will be written. Minidumps are not enabled by default on client versions of Windows
#
# An error report file with more information is saved as:
# C:\Users\User\Desktop\WebRTC Simple JavaFX\hs_err_pid10400.log
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

I tried running on a different computer with java 11, with the same result. My current code (stripped down to only display the local video feed):

package WebRTC;

import dev.onvoid.webrtc.*;
import dev.onvoid.webrtc.demo.javafx.control.VideoView;

import dev.onvoid.webrtc.media.MediaDevices;
import dev.onvoid.webrtc.media.video.*;

import javafx.application.Application;
import javafx.geometry.Insets;
import javafx.scene.Scene;
import javafx.scene.layout.VBox;
import javafx.stage.Stage;

public class Main extends Application {

    private PeerConnectionFactory factory;

    @Override
    public void start(Stage primaryStage) {

        VBox root = new VBox();
        root.setPadding(new Insets(10));
        root.setSpacing(8);

        VideoView localVideoView = new VideoView();
        localVideoView.setLayoutX(100);
        localVideoView.setLayoutY(100);

        root.getChildren().add(localVideoView);

        primaryStage.setTitle("WebRTC");
        primaryStage.setScene(new Scene(root, 300, 275));
        primaryStage.show();

        factory = new PeerConnectionFactory();

        VideoDeviceSource videoSource = new VideoDeviceSource();
        videoSource.setVideoCaptureDevice(MediaDevices.getVideoCaptureDevices().get(0));
        VideoTrack videoTrack = factory.createVideoTrack("videoTrack", videoSource);

        videoTrack.addSink(localVideoView::setVideoFrame);

        try {
            videoSource.start();
        }
        catch (Exception e) {
            e.printStackTrace();
        }
    }

    public static void main(String[] args) {
        launch(args);
    }
}

The line videoSource.start(); causes the crash.

For my setup I downloaded the github project, run mvn install and imported 4 jar files created as libraries into a new JavaFX project. Imported jar files: webrtc-java-0.3.0-SNAPSHOT.jar webrtc-java-0.3.0-SNAPSHOT-windows-x86_64.jar webrtc-java-demo-api-0.3.0-SNAPSHOT.jar webrtc-java-demo-javafx-0.3.0-SNAPSHOT.jar

[devopvoid]

For debugging purposes, could you please ommit this line and start the app again:

videoSource.setVideoCaptureDevice(MediaDevices.getVideoCaptureDevices().get(0));

Could something be wrong with the camera device selection. If you do not provide a video device, one will be selected by default.

[meeeee12]

Omitting that line does indeed prevent the crash, but the local video feed still isn't being displayed.

[meeeee12]

There was something wrong with my IntelliJ project. I created a new project, copied all my code and now it works.

[kinsleykajiva]

Hi @meeeee12

your code requires a bit more work to function properly. But let's start with the video source. To receive video frames from the VideoDeviceSource, you need to start it first, e.g.,

try {
  videoSource.start();
}
catch (Exception e) {
  e.printStackTrace();
}

To use the peer connection, you need to create an offer with it, if you want to initiate the connection setup, e.g,

RTCOfferOptions options = new RTCOfferOptions();

peerConnection.createOffer(options, new CreateSessionDescriptionObserver() {

  @Override
  public void onSuccess(RTCSessionDescription description) {
      peerConnection.setLocalDescription(description, new SetSessionDescriptionObserver() {

          @Override
          public void onSuccess() {
              // Send description as offer.
          }

          @Override
          public void onFailure(String error) {
              System.out.println("Set local RTCSessionDescription failed: " + error);
          }
      });
  }

  @Override
  public void onFailure(String error) {
      System.out.println("Create RTCSessionDescription failed: " + error);
  }
});

Setting the local session description will also trigger void onIceCandidate(RTCIceCandidate candidate). These candidates must also be sent to the remote peer. Eventually, you receive an answer (RTCSessionDescription) from the remote peer which can be set with peerConnection.setRemoteDescription()...

Hi , I have been looking at this sample but I seem not to trigger the void onIceCandidate(RTCIceCandidate candidate) function I have tried a lot of different implementations, I wish to use this with Janus server, please can you help?