Support GStreamer for H.264

[GoogleCodeExporter]

Mozilla is now directly supporting this, so we will follow suit. It is not 
enabled right now.

https://bugzilla.mozilla.org/show_bug.cgi?id=422540

Once somebody creates PPC libraries for GStreamer, this should "just work." We 
still do not have licensing clarity from Mozilla, so this is partially 
policy-blocked.

Original issue reported on code.google.com by classi...@floodgap.com on 5 Jun 2012 at 9:21

[GoogleCodeExporter]

http://www.tribler.org/trac/wiki/GstreamerMac

These are old, but should still generally be relevant.

Original comment by classi...@floodgap.com on 5 Jun 2012 at 9:25

[GoogleCodeExporter]

I tried that with Aurora 16.

Building gstreamer worked without problems using MacPorts.
A slight tweak had to be made to configure.in so that the test could find the 
gstreamer libraries. Building gstreamer support does also work.

But it doesn't work at all and hangs at exit after trying to play ogg vorbis 
videos. Some threading issue?

Original comment by Tobias.N...@gmail.com on 23 Jul 2012 at 8:37

[GoogleCodeExporter]

Original comment by Tobias.N...@gmail.com on 23 Jul 2012 at 8:37

• Changed state: Started

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Songbird does use this for Intel OS X 10.5 - but as far as I could see they 
base on mozilla 1.9.2 .

Original comment by Tobias.N...@gmail.com on 26 Jul 2012 at 8:38

[GoogleCodeExporter]

GStreamer also has a QuickTime wrapper (which Songbird also uses); it seems to 
accomplish exactly what QuickTime Enabler is doing.

Original comment by Tobias.N...@gmail.com on 26 Jul 2012 at 10:06

[GoogleCodeExporter]

nsGStreamerReader is waiting infinitely for GST_MESSAGE_ASYNC_DONE.

Original comment by to...@jesus.de on 27 Jul 2012 at 10:03

[GoogleCodeExporter]

That was me once more.
And GStreamer is "waiting in preroll for flush or PLAYING".

Original comment by Tobias.N...@gmail.com on 27 Jul 2012 at 10:17

[GoogleCodeExporter]

Unmuxed video only h.264 streams DO play. The audio is causing problems!

Original comment by Tobias.N...@gmail.com on 27 Jul 2012 at 10:56

[GoogleCodeExporter]

Turns out to also be an endianness issue.

Original comment by Tobias.N...@gmail.com on 27 Jul 2012 at 1:46

[GoogleCodeExporter]

The following patch is needed:

diff -r 2887b2e42a95 configure.in
--- a/configure.in  Mon Jul 23 13:19:43 2012 +0200
+++ b/configure.in  Fri Jul 27 16:11:15 2012 +0200
@@ -5639,7 +5639,7 @@
                       gstreamer-plugins-base-$GST_API_VERSION)
     if test -n "$GSTREAMER_LIBS"; then
        _SAVE_LDFLAGS=$LDFLAGS
-       LDFLAGS="$LDFLAGS -lgstvideo-$GST_API_VERSION"
+       LDFLAGS="$LDFLAGS $GSTREAMER_LIBS -lgstvideo-$GST_API_VERSION"
        AC_TRY_LINK(,[return 0;],_HAVE_LIBGSTVIDEO=1,_HAVE_LIBGSTVIDEO=)
        if test -n "$_HAVE_LIBGSTVIDEO" ; then
           GSTREAMER_LIBS="$GSTREAMER_LIBS -lgstvideo-$GST_API_VERSION"
diff -r 2887b2e42a95 content/media/gstreamer/nsGStreamerReader.cpp
--- a/content/media/gstreamer/nsGStreamerReader.cpp Mon Jul 23 13:19:43 2012 
+0200
+++ b/content/media/gstreamer/nsGStreamerReader.cpp Fri Jul 27 16:11:15 2012 
+0200
@@ -131,7 +131,11 @@

   mAudioSink = gst_parse_bin_from_description("capsfilter name=filter ! "
         "appsink name=audiosink sync=true caps=audio/x-raw-float,"
+#ifdef __BIG_ENDIAN__
+        "channels={1,2},rate=44100,width=32,endianness=4321", TRUE, NULL);
+#else
         "channels={1,2},rate=44100,width=32,endianness=1234", TRUE, NULL);
+#endif
   mAudioAppSink = GST_APP_SINK(gst_bin_get_by_name(GST_BIN(mAudioSink),
         "audiosink"));
   gst_app_sink_set_callbacks(mAudioAppSink, &mSinkCallbacks,

Performance is bad but h.264 works.

Original comment by Tobias.N...@gmail.com on 27 Jul 2012 at 2:13

[GoogleCodeExporter]

And there are situations where firefox hangs upon quitting.

Original comment by Tobias.N...@gmail.com on 27 Jul 2012 at 2:15

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

YouTube asks for WebM support before asking for h.264 support. For that reason 
WebKit treats youtube.com differently from other URLs by not stating WebM as 
supported even if it indeed is (be means of Perian).
You could work around this by setting "media.webm.enabled" (and maybe also 
"media.ogg.enabled"?) to false.
Audio output is broken, isn't it?

Original comment by Tobias.N...@gmail.com on 29 Jul 2012 at 12:12

[GoogleCodeExporter]

Audio output using QuickTime codec wrapper was broken because of the same 
endianness issue as in libcube. That's a bug in GStreamer itself. A patch is 
attached.

Now using GStreamer for h264 works without any licensing problems since it's 
using QuickTime codecs exclusively.

In order to ship it in an application bundle some work has to be done, like 
having firefox launched by a shell script that sets some GStreamer environment 
variables and supply more than 10 shared libraries, each of them treated with 
install_name_tool.

Original comment by Tobias.N...@gmail.com on 29 Jul 2012 at 4:54

Attachments:

• audiodecoders.patch

[GoogleCodeExporter]

It seems youtube can't be forced into providing h264 instead of WebM by 
disabling WebM in firefox. It will fallback to flash instead; probably youtube 
does browser detection.

Original comment by to...@jesus.de on 29 Jul 2012 at 5:17

[GoogleCodeExporter]

That was me once again.

Original comment by Tobias.N...@gmail.com on 29 Jul 2012 at 5:18

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Setting environment variables is pretty easily done from nsBrowserApp.cpp.

Original comment by classi...@floodgap.com on 30 Jul 2012 at 1:28

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

MPAPI (see https://bugzilla.mozilla.org/show_bug.cgi?id=749897 and the wiki 
page linked in that meta bug) is actually much more interesting than GStreamer.
So improving the current GStreamer support any further will not make much 
sense. GStreamer could even be integrated as MPAPI plugin - when using 
"osxvideosink" as output this would provide performance near to directly using 
OS X API.

Original comment by Tobias.N...@gmail.com on 30 Jul 2012 at 3:21

[GoogleCodeExporter]

The gstreamer bug was filed to the gnome team as:
https://bugzilla.gnome.org/show_bug.cgi?id=680923

Original comment by Tobias.N...@gmail.com on 31 Jul 2012 at 4:27

[GoogleCodeExporter]

The mozilla gstreamer backend bug has already been fixed in mozilla-central:
https://bugzilla.mozilla.org/show_bug.cgi?id=777696

Original comment by Tobias.N...@gmail.com on 31 Jul 2012 at 4:58

[GoogleCodeExporter]

So I'm thinking this will be good to schedule for 18, since we'll be shipping 
our own new dylibs for libc anyway. This fits in pretty well with that. What 
are all the libs that we need to include?

Original comment by classi...@floodgap.com on 2 Aug 2012 at 3:14

[GoogleCodeExporter]

So far I added the following libraries:
libffi.6.dylib
libgio-2.0.0.dylib
libglib-2.0.0.dylib
libgmodule-2.0.0.dylib
libgobject-2.0.0.dylib
libgstapp-0.10.0.dylib
libgstaudio-0.10.0.dylib
libgstbase-0.10.0.dylib
libgstcodecparsers-0.10.23.dylib
libgstcontroller-0.10.0.dylib
libgstinterfaces-0.10.0.dylib
libgstpbutils-0.10.0.dylib
libgstreamer-0.10.0.dylib
libgstriff-0.10.0.dylib
libgstrtp-0.10.0.dylib
libgsttag-0.10.0.dylib
libgstvideo-0.10.0.dylib
libgthread-2.0.0.dylib
libiconv.2.dylib
libintl.8.dylib
liborc-0.4.0.dylib
liborc-test-0.4.0.dylib
libxml2.2.dylib (for 10.5 the system installed version could be used)
libz.1.dylib (for 10.5 the system installed version could be used)

Plus one executable:
gst-plugin-scanner

Plus the gstreamer plugins (can be stripped down a little more as not all of 
them are necessary):
libgstapp.so
libgstaudioconvert.so
libgstaudioparsers.so
libgstaudioresample.so
libgstauparse.so
libgstcdxaparse.so
libgstcoreelements.so
libgstcoreindexers.so
libgstdecodebin.so
libgstdecodebin2.so
libgstffmpegcolorspace.so
libgsth264parse.so
libgsthdvparse.so
libgstisomp4.so
libgstivfparse.so
libgstmpegaudioparse.so
libgstmpegvideoparse.so
libgstpcapparse.so
libgstplaybin.so
libgstqtwrapper.so
libgstrawparse.so
libgstsubparse.so
libgsttypefindfunctions.so
libgstvideofilter.so
libgstvideofiltersbad.so
libgstvideoparsersbad.so
libgstvideoscale.so
libgstvolume.so
libgstwavparse.so

I attached two script files that invoke install_name_tool for all those 
libraries, one for the ".dylib"s (ist.sh) and one for the ".so"s (ist_2.sh). 
Those scripts are more "intelligent" than the one for libgcc and libstdc++.

The third script is to set up the environment variables for GStreamer to work 
within the application bundle.

Original comment by Tobias.N...@gmail.com on 2 Aug 2012 at 7:42

Attachments:

• ist.sh
• ist_2.sh
• firefox.command

[GoogleCodeExporter]

However I'd consider this type of GStreamer integration as a fallback as long 
as there isn't an OS X MPAPI plugin.
You need a *fast* dual G4 or a (maybe also dual core?) G5 for fluid playback.

Original comment by Tobias.N...@gmail.com on 2 Aug 2012 at 7:49

[GoogleCodeExporter]

Mozilla bug 760140 is about extending the use of gstreamer codecs.

Original comment by Tobias.N...@gmail.com on 13 Oct 2012 at 6:51

[GoogleCodeExporter]

Been thinking. It seems like we could make a limited "phony GStreamer" that 
just provides the functions needed as shims to QuickTime, since GStreamer is 
just running in that capacity. It would then eliminate the need to build 
GStreamer and include all the libs.

Original comment by classi...@floodgap.com on 27 Oct 2012 at 10:41

[GoogleCodeExporter]

I'm also thinking about resurrecting M435298 ( 
https://bugzilla.mozilla.org/show_bug.cgi?id=435298 ). Although Mozilla 
abandoned it, it's new enough it probably will still work with 18+ with some 
modification, and though it uses APIs Apple apparently doesn't support, that's 
not our problem for 10.4 and 10.5. Not having seeking sucks, but it's workable.

Original comment by classi...@floodgap.com on 30 Oct 2012 at 4:08

[GoogleCodeExporter]

(If we did that, we would need to make similar changes based on M513144 and 
M736400, and maybe M503989 and M481057.)

Original comment by classi...@floodgap.com on 30 Oct 2012 at 4:13

[GoogleCodeExporter]

At least for 10.5 there's another option. VLC is coming with a new Mac OS X 
NPAPI plugin. It will use the supported Cocoa event model and both CoreGraphics 
and CoreAnimation drawing models. Cocoa events and CoreGraphics drawing model 
should be fine with 10.4 .

Original comment by Tobias.N...@gmail.com on 19 Nov 2012 at 9:49

[GoogleCodeExporter]

Using GStreamer ffmpeg and mad (MP3) codecs instead of GStreamer QuickTime 
wrappers reduces the number of needed shared libraries and improves h264 
performance a lot so that 360p can be played on fast single core G4s and 480p 
on single core G5s. That way the video from issue 181 does also play well.
I also found out that GStreamer uses liborc as SIMD JIT library that does also 
generate Altivec code. It does indeed generate the code on the fly.

Original comment by Tobias.N...@gmail.com on 28 Dec 2012 at 3:47

[GoogleCodeExporter]

Aurora 20 has generally improved video rendering perfomance noticeably, 
affecting both GStreamer playback as well as WebRTC getUserMedia.
Performance of GStreamer with ffmpeg and mad codecs is acceptable while 
getUserMedia is tolerable.
Indeed nice to see how mozilla is working towards speed on lower end devices.

Original comment by Tobias.N...@gmail.com on 14 Jan 2013 at 11:29

[GoogleCodeExporter]

Is WebM better as well?

Original comment by classi...@floodgap.com on 14 Jan 2013 at 11:42

[GoogleCodeExporter]

Yes, a lot. Stuttering from time to time in 19 and smooth in 20.

Original comment by Tobias.N...@gmail.com on 14 Jan 2013 at 11:59

[GoogleCodeExporter]

The iMac G5 2.1 GHz now scores 553 in peacekeeper which is faster than 3rd 
generation iPad. I didn't ever get a score higher than 500. Video playback in 
peacekeeper is noticeably smoother while WebM is still performing worse than 
any other codecs (yes, Ogg Theora is faster and H.264 using GStreamer is much 
better).

Original comment by Tobias.N...@gmail.com on 15 Jan 2013 at 12:10

[GoogleCodeExporter]

Looks like comment 36 is a reality per 
https://bugzilla.mozilla.org/show_bug.cgi?id=851290 -- problem is, VideoToolbox 
doesn't exist on 10.4, I don't think.

Original comment by classi...@floodgap.com on 28 Jun 2013 at 5:06

[GoogleCodeExporter]

(The other option is to simply use GStreamer and to say to hell with the 
licensing. I'm starting to lean that way.)

Original comment by classi...@floodgap.com on 28 Jun 2013 at 5:18

[GoogleCodeExporter]

This is imminent: https://bugzilla.mozilla.org/show_bug.cgi?id=806917

--enable-gstreamer=1.0

Original comment by classi...@floodgap.com on 31 Jan 2014 at 4:10

[GoogleCodeExporter]

Mozilla has also added FFmpeg support. I'm strongly considering it, though 
probably not for 38.

https://bugzilla.mozilla.org/show_bug.cgi?id=1107264

Original comment by classi...@floodgap.com on 25 Feb 2015 at 5:48