search

gstreamer-java / gst1-java-core

Java bindings for GStreamer 1.x
GNU Lesser General Public License v3.0
194 stars 72 forks source link

GArray binding missing, unable to enable RTX #168

Closed neilyoung closed 5 years ago

neilyoung commented 5 years ago

Hi,

I need to enable RTX like shown here

GstWebRTCRTPTransceiver *trans;
GArray *transceivers;  
g_signal_emit_by_name (t->webrtc1, "get-transceivers", &transceivers);
fail_unless_equals_int (transceivers->len, 2);
trans = g_array_index (transceivers, GstWebRTCRTPTransceiver *, 1);
g_object_set (trans, "fec-type", GST_WEBRTC_FEC_TYPE_ULP_RED, "do-nack", TRUE, NULL);
g_array_unref (transceivers);

Even though I'm able to retrieve the transceivers, I'm unable to use them, since the Java binding for GArray is missing. What options do I have?

neilcsmith-net commented 5 years ago

Immediate options are -

neilyoung commented 5 years ago

:) Thanks. What specifically do you mean by your first point?

neilcsmith-net commented 5 years ago

See https://gstreamer.freedesktop.org/documentation/tools/gst-launch.html?gi-language=c You can use the same String description you would use on the command line to create pipelines, partial pipelines, bins and elements (see Gst::parseLaunch and Gst::parseBinFromDescription). This is sometimes a useful workaround where a mapping is missing. See also #166 for another use of this.

neilyoung commented 5 years ago

Ah this part. Yes. I'm aware of, but I never could make a command line run using webrtcbin. Most likely due to signalling issues

neilyoung commented 5 years ago

Similar to this, I guess.... https://gstreamer.freedesktop.org/documentation/rtpmanager/rtprtxreceive.html?gi-language=c

neilyoung commented 5 years ago

Maybe I don't need that at all. In my use case the video is just fetched from the PI to the java app. As far as I can see enabling RTX would be something for the Java side upstream.

Then I'm wondering, why webrtcbin does not recover at packet loss and I'm forced to send a FIR request upstream.

Both parties declare support for nack pli, as far as I can see...

PI's offer:

v=0 o=- 213190082869872455 2 IN IP4 127.0.0.1 s=- t=0 0 a=group:BUNDLE video a=msid-semantic: WMS stream_label m=video 39749 UDP/TLS/RTP/SAVPF 100 c=IN IP4 192.168.189.30 a=rtcp:9 IN IP4 0.0.0.0 generation 0 network-id 4 network-cost 50 a=candidate:2411936722 1 udp 2122194687 192.168.189.30 39749 typ host generation 0 network-id 3 network-cost 50 a=ice-ufrag:Y0Tm a=ice-pwd:8RKsBzMrY8FHJUdrNGXsqQzF a=ice-options:trickle a=fingerprint:sha-256 42:05:8E:16:9E:D7:23:3F:8A:BC:19:6B:F5:A4:BC:44:A6:5E:E2:80:AC:9A:9F:E3:73:1C:95:BA:B4:13:23:B5 a=setup:actpass a=mid:video a=extmap:2 urn:ietf:params:rtp-hdrext:toffset a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time a=extmap:4 urn:3gpp:video-orientation a=extmap:5 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01 a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/video-timing a=extmap:10 http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07 a=extmap:12 http://www.webrtc.org/experiments/rtp-hdrext/color-space a=sendonly a=rtcp-mux a=rtcp-rsize a=rtpmap:100 H264/90000 a=rtcp-fb:100 goog-remb a=rtcp-fb:100 transport-cc a=rtcp-fb:100 ccm fir a=rtcp-fb:100 nack a=rtcp-fb:100 nack pli a=fmtp:100 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e028 a=ssrc-group:FID 1771540765 2920726280 a=ssrc:1771540765 cname:Il5CtWFfEbHGSg7D a=ssrc:1771540765 msid:stream_label video_label a=ssrc:1771540765 mslabel:stream_label a=ssrc:1771540765 label:video_label a=ssrc:2920726280 cname:Il5CtWFfEbHGSg7D a=ssrc:2920726280 msid:stream_label video_label a=ssrc:2920726280 mslabel:stream_label a=ssrc:2920726280 label:video_label

webrtcbins' answer:

v=0 o=- 213190082869872455 0 IN IP4 0.0.0.0 s=- t=0 0 m=video 9 UDP/TLS/RTP/SAVPF 100 c=IN IP4 0.0.0.0 a=ice-ufrag:fJbD465eUcHEU73w9gfPKHfjmqa96IVB a=ice-pwd:6CZpCiitVn0ZZNg+nbgj0JZZhGkyKFno a=mid:video a=rtcp-mux a=setup:active a=rtpmap:100 H264/90000 a=rtcp-fb:100 nack pli a=rtcp-fb:100 ccm fir a=fmtp:100 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e028 a=recvonly a=fingerprint:sha-256 6F:1A:3D:01:0E:DF:B7:CA:70:8D:20:CF:BF:C2:58:08:E7:F5:96:CA:48:A5:D0:41:EE:FD:2C:38:DD:9B:F5:EF

It should recover after packet loss, but it doesn't....

svnhub commented 5 years ago

There is now a get-transceiver signal (without the 's') with an index that returns the GstWebRTCRTPTransceiver directly to avoid GArary, but this result has no mapping either.

After reading your issue it seems very similar to #169

neilyoung commented 5 years ago

Doesn't work. Excepts.

Integer index = 0;
GObject trans = webRTCBin.emit(GObject.class, "get-transceiver",new Object[]{index});

Jul 13, 2019 2:05:54 PM com.sun.jna.Native$1 uncaughtException
WARNUNG: JNA: Callback org.freedesktop.gstreamer.Element$1@7462a821 threw the following exception
java.lang.RuntimeException: java.lang.InstantiationException
    at org.freedesktop.gstreamer.glib.NativeObject.objectFor(NativeObject.java:193)
    at org.freedesktop.gstreamer.glib.Natives.objectFor(Natives.java:163)
    at org.freedesktop.gstreamer.glib.Natives.objectFor(Natives.java:103)
    at org.freedesktop.gstreamer.glib.GObject.emit(GObject.java:146)
neilyoung commented 5 years ago

I was thinking, that setting do-nack to true at the transeiver(s) is making the receiving Java app sending NACK in case of detected packet losses. But I don't have one single NACK counted at the sending side. Even though there is a significant (and detected) packet loss.

It seems, that just my mechanism of sending FIR in case of detected packet losses does work.

But I could have misunderstood this do-nack thing entirely....

Since I'm not sure, which transceiver is the right one, I have set it to true for both...


            Integer index = 0;
            GstObject trans = webRTCBin.emit(GstObject.class, "get-transceiver",new Object[]{index});
            trans.set("do-nack", true);

            index+=1;
            GstObject trans1 = webRTCBin.emit(GstObject.class, "get-transceiver",new Object[]{index});
            trans1.set("do-nack", true);
neilyoung commented 5 years ago

@neilcsmith-net That being said: I see NO visible effect of the "do-nack" setting. I believe this is not a Java issue, since I can set and retrieve the setting. Who would have to be bothered, according to your esteemed opinion?

neilcsmith-net commented 5 years ago

:laughing: Upstream problem!

neilyoung commented 5 years ago

Why this? Neither a browser nor a PI counts incoming NACKS, even though there should be, since the downstream is heavily distorted. Would there be a chance to see such NACKs, e.g. in Wireshark?

neilcsmith-net commented 5 years ago

I meant upstream as in ask the GStreamer people. :smile: Doesn't seem to be anything specific to the Java bindings.

neilyoung commented 5 years ago

Oops, got that wrong... Sorry. OK, will try to do that.

neilyoung commented 5 years ago

Found the problem. My bad. SDP negotiation did not cover the RTX part