WebRTC FEC configuration support

[svnhub]

Recently GStreamer added ways to control the forward error encoding of WebRTC streams, which greatly improves stream quality over network links with random package loss at the cost of a (configurable) increase in bandwidth used.

The settings are located on the transceivers which are accessible through the WebRTCbin's on-new-transceiver signal (when they are added) or get-transceivers/get-transceiver(index) afterwards.

It would be nice if it was possible to do something similar to the native code variant like:

static void
_on_new_transceiver (GstElement * webrtc, GstWebRTCRTPTransceiver * trans) {
  g_object_set (trans, "fec-type", GST_WEBRTC_FEC_TYPE_ULP_RED, NULL);
}

or

  g_signal_emit_by_name (webrtc, "get-transceivers", &transceivers);
  trans = g_array_index (transceivers, GstWebRTCRTPTransceiver *, 0);
  g_object_set (trans, "fec-type", GST_WEBRTC_FEC_TYPE_ULP_RED, "fec-percentage", 100, NULL);

This feature is quite useful for streaming to clients on wireless or mobile networks.

[neilcsmith-net]

Agreed, and see discussion in #168 - PR welcomed if anyone wants to contribute.

[neilcsmith-net]

It's possible the get-transceiver(index) action signal might already work with the bindings using GObject::emit Have you tried something like -

GObject trans = webtrc.emit(GObject.class, "get-transceiver", index);

[svnhub]

Someone tried that in #168 seconds after you wrote it. Unfortunately it returns a WebRTCRTPTransceiver which isn't mapped yet.

Another thing that isn't mapped yet is the native enum for GST_WEBRTC_FEC_TYPE which would look something like this:

/*.
 * Copyright (c) 2019 Janus Kristensen
 *.
 * This file is part of gstreamer-java.
 *
 * This code is free software: you can redistribute it and/or modify it under.
 * the terms of the GNU Lesser General Public License version 3 only, as
 * published by the Free Software Foundation.
 *
 * This code is distributed in the hope that it will be useful, but WITHOUT.
 * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or.
 * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU Lesser General Public License.
 * version 3 for more details.
 *
 * You should have received a copy of the GNU Lesser General Public License
 * version 3 along with this work.  If not, see <http://www.gnu.org/licenses/>.
 */
package org.freedesktop.gstreamer.webrtc;

import org.freedesktop.gstreamer.Gst;
import org.freedesktop.gstreamer.glib.NativeEnum;

/**
 * The type of forward error coding used by a {@link WebRTCRTPTransceiver}.
 * <p>
 * @see https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/blob/b6411ae74cc7319ef368f5b4a6a872231b49620a/gst-libs/gst/webrtc/webrtc_fwd.h
 * Available since GStreamer 1.14.1
 */
@Gst.Since(minor = 14)
public enum WebRTCFECType implements NativeEnum<WebRTCFECType> {
    NONE(0),// Disable FEC (default)
    ULP_RED(1); // Uneven Level Protection Forward Error Coding + RED

    private final int value;

    private WebRTCFECType(int value) {
        this.value = value;
    }

    /**
     * Gets the integer value of the enum
     * @return the integer value for this enum.
     */
    @Override
    public int intValue() {
        return value;
    }
}

[neilyoung]

@svnhub Yepp, that crashes

            Integer index = 0;
            GObject trans = webRTCBin.emit(GObject.class, "get-transceiver",new Object[]{index});

Jul 13, 2019 2:05:54 PM com.sun.jna.Native$1 uncaughtException
WARNUNG: JNA: Callback org.freedesktop.gstreamer.Element$1@7462a821 threw the following exception
java.lang.RuntimeException: java.lang.InstantiationException
    at org.freedesktop.gstreamer.glib.NativeObject.objectFor(NativeObject.java:193)
    at org.freedesktop.gstreamer.glib.Natives.objectFor(Natives.java:163)
    at org.freedesktop.gstreamer.glib.Natives.objectFor(Natives.java:103)
    at org.freedesktop.gstreamer.glib.GObject.emit(GObject.java:146)

[neilcsmith-net]

Unfortunately it returns a WebRTCRTPTransceiver which isn't mapped yet.

@svnhub not a problem, assuming your example C code is showing that this is a GObject. GObject and MiniObject subclasses do not need to be mapped to get a reference as a parent type.

@neilyoung I assume you found the right method now?! :smile: Are you sure that the version of GStreamer you're using supports this action signal? If so, please provide a full MCVE and I'll take a look - you shouldn't be getting that exception if the right type is being returned here as far as I can tell. But I'd need to step through to be sure.

[neilyoung]

@neilcsmith-net No it is still not working for me. GStreamer version is 1.16.0, java library is 1.1.0. Not sure if it provides the signal. I suggest to use https://github.com/centricular/gstwebrtc-demos and simply just add the two lines I provided above somewhere in onIncomingDecodebinStream

[neilcsmith-net]

@neilyoung OK, will take a look at some point, but don't have a machine with 1.16 on it at the moment, so won't be for a while. Suggest you also debug and step through emit(..) and see what exactly its failing on - I wonder if at the stage you're doing it it isn't getting a valid pointer to an object back. Otherwise, step through until https://github.com/gstreamer-java/gst1-java-core/blob/master/src/org/freedesktop/gstreamer/glib/NativeObject.java#L167

Judging from your stack trace, it's not finding a registration it should be. I also realise that if it doesn't, GObject is not the right type, because it's abstract (I may make it concrete in 1.2)

What about ?

GstObject trans = webRTCBin.emit(GstObject.class, "get-transceiver",new Object[]{index});

[neilyoung]

@neilcsmith-net He, good catch. GstObject seems to match.

[neilyoung]

Now the question is: What to do with the result :) ? Let's see

[neilcsmith-net]

Well, from the original issue from @svnhub you should be able to call

GstObject trans = webRTCBin.emit(GstObject.class, "get-transceiver",new Object[]{index});
trans.set("fec-type", 1);
trans.set("fec-percentage", 100);

I have no idea what you need to call on it. :smile:

Also, I would have thought you could pass in index directly rather than wrapping in an Object[]?!

[neilyoung]

I would like to enable “do-nack”. Will do that later the evening

[neilyoung]

I was thinking, that setting do-nack to true at the transeiver(s) is making the receiving Java app sending NACK in case of detected packet losses. But I don't have one single NACK counted at the sending side. Even though there is a significant (and detected) packet loss.

It seems, that just my mechanism of sending FIR in case of detected packet losses does work.

But I could have misunderstood this do-nack thing entirely....

[neilyoung]

Sorry, realised that I was hijacking this thread. Will continue in "mine"

[svnhub]

The suggestion of using the super object works for both setting ULP+FEC/RED (via fec-type) and RTX (via do-nack), thanks!

[neilyoung]

@svnhub Wait. I agree, I can set those properties. But do you also see an effect, e.g. do you see NACKs at sending side? I don't

[svnhub]

For FEC I see ulp and red present in the SDP and a matching bandwidth increase, although I'm not sure the browser is making use of it. For RTX I see it in the SDP and NACKs from the browser but no re-transmissions, but I suspect an upstream problem, not related to the Java-wrapper.

[neilyoung]

@svnhub My setup is: The browser (or a raspberry PI) are sending video (no audio). The Java app is receiving only. I have applied disturbances to the downlink of the Java app. I would expect to see the Java app sending NACKs, since there must be some. I don't see these NACKs coming from the Java app. Will check the RTP statistics of the Java app for incoming NACKs.

But my expectation was, that do-nack just enables the Java app to send NACKs, is this wrong?

[neilyoung]

@svnhub Agreed, NACKs are sent. My bad. Found the problem. SDP negotiation did not cover the RTX part