Direct3D Video Sink

[GoogleCodeExporter]

I open this issue to track the development of the plugin.

This morning I have some spare time at work and I'll try to start playing with 
the code. For instance I'd like to make use of the VideoBaseSink and implement 
prerolling.

Original issue reported on code.google.com by ylatuya on 22 Jul 2010 at 9:57

[GoogleCodeExporter]

Original comment by david.g.hoyt on 20 Jan 2011 at 12:34

• Changed state: Started

[GoogleCodeExporter]

Original comment by david.g.hoyt on 20 Jan 2011 at 12:35

[GoogleCodeExporter]

Original comment by david.g.hoyt on 20 Jan 2011 at 12:35

• Added labels: Type-Other
• Removed labels: Type-Defect

[GoogleCodeExporter]

There was one gcc error (using 32 bit mingw-w64 in Cygwin), see 
d3d-error.patch. The patch fixes calling DIRECTX_DEBUG and other macros with 
only one parameter, see:

http://gcc.gnu.org/onlinedocs/gcc/Variadic-Macros.html
http://gcc.gnu.org/onlinedocs/cpp/Variadic-Macros.html

The new code also produces some gcc warnings, I'll check them later.

Attached is Makefile.am that I used.

Original comment by raimo.ja...@gmail.com on 20 Jan 2011 at 5:08

Attachments:

• d3d-error.patch
• Makefile.am

[GoogleCodeExporter]

Calling gst_d3dvideosink_initialize_d3d_device with a NULL parameter causes a 
crash, see the patch.

Original comment by raimo.ja...@gmail.com on 23 Jan 2011 at 9:42

Attachments:

• d3d-crash.patch

[GoogleCodeExporter]

raimo.jarvi: I addressed some of the warnings and the crash in r969. Please let 
me know if everything's resolved now.

Original comment by david.g.hoyt on 25 Jan 2011 at 10:51

[GoogleCodeExporter]

Code compiles with gcc now. There's warnings about pointer types in the new 
code introduced in r959. For example, it looks like arguments of 
DirectXComponentInitializeFunction don't match arguments of dx9_d3d_init et al.

Attached is a patch that fixes log formatting and unhandled enumeration value 
warnings in d3dvideosink.c. You might also want to fix these:

d3dvideosink.c:1543:11: warning: suggest parentheses around `&&' within `||'

d3dvideosink.c:2028:15: warning: `d3dfourcc' may be used uninitialized in this 
function

Original comment by raimo.ja...@gmail.com on 26 Jan 2011 at 8:31

Attachments:

• d3d-warning.patch

[GoogleCodeExporter]

Applied your patch in r971.

Original comment by david.g.hoyt on 27 Jan 2011 at 1:20

[GoogleCodeExporter]

Time to prepare this for submission upstream. Thoughts?

Original comment by david.g.hoyt on 3 Mar 2011 at 10:08

[GoogleCodeExporter]

I forgot to post a patch for the remaining gcc warnings, here it is. There are 
still some warnings about unused stuff:

directx/directx9/dx9_d3d.h:69:1: warning: 
`init_directx_api_component_d3d_DIRECTX_9_' defined but not used
directx/directx9/dx9.h:36:1: warning: `init_directx_DIRECTX_9_supported_api' 
defined but not used
directx/directx10/dx10_d3d.h:70:1: warning: 
`init_directx_api_component_d3d_DIRECTX_10_' defined but not used
directx/directx10/dx10.h:37:1: warning: `init_directx_DIRECTX_10_supported_api' 
defined but not used
directx/directx11/dx11_d3d.h:70:1: warning: 
`init_directx_api_component_d3d_DIRECTX_11_' defined but not used
directx/directx11/dx11.h:37:1: warning: `init_directx_DIRECTX_11_supported_api' 
defined but not used

Original comment by raimo.ja...@gmail.com on 4 Mar 2011 at 8:09

Attachments:

• d3d.patch

[GoogleCodeExporter]

i'm just realize while directdrawsink has a field as "handle" which we use for 
overlay, but d3dvideosink has not. is there any field in d3dvideosink which can 
be used for overlay or we'd have to revert to directdrawsink in gstreamer-java?
thanks

Original comment by lfar...@gmail.com on 21 Mar 2011 at 1:30

[GoogleCodeExporter]

I don't think I follow... the window handle is emitted as a message by the sink 
which should be read synchronously by the application and use the xoverlay 
interface to set it before the sink creates its own window. You shouldn't be 
accessing it directly. Anyway you have 'window_handle' in case you need it: 
http://code.google.com/p/ossbuild/source/browse/trunk/Main/GStreamer/Source/gst-
plugins-bad/sys/d3dvideosink/d3dvideosink.c#64

Original comment by ylatuya on 21 Mar 2011 at 2:24

[GoogleCodeExporter]

See the documentation for the XOverlay interface: 
http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-libs/h
tml/gst-plugins-base-libs-gstxoverlay.html

You shouldn't be relying on a handle property -- xoverlay is the best and most 
compatible way of doing what you need.

Original comment by david.g.hoyt on 21 Mar 2011 at 4:01

[GoogleCodeExporter]

sorry, it was my fault. anyway we need handle directly in gstreamer-java in 
order to be able to use overlay we need the native handle. in SWT it's not the 
d3dvideosink's field, but the SWT Control's windows implementation's handle. in 
c it's easy to get the native handle, but from java it's not so easy.
anyway when when we use d3dvideosink on windows (from gstreamer-java for 
overlay) we always got this error, but the d3dvideosink seems to working:
----------------------------------
0:00:00.484375000  3304   03512A28 ERROR           d3dvideosink
d3dvideosink.c:2204:gst_d3dvideosink_release_swap_chain: Direct3D
device has not been initialized
----------------------------------

Original comment by lfar...@gmail.com on 22 Mar 2011 at 10:28

[GoogleCodeExporter]

I'm a bit confused -- I've used d3dvideosink w/ gstreamer-java (and with SWT) 
before with no issues. Are you trying to get the handle for the window that 
gstreamer creates if xoverlay isn't used?

If you're using it normally, you should already know the window handle and be 
able to access it since you had to provide it to gstreamer in the first place 
for xoverlay to work (otherwise a separate window will pop up).

SWT makes it relatively easy to get the window handle. With AWT/Swing, use 
JNA's Native.getComponentID() or Native.getComponentPointer().

If by overlay you mean you want to draw on top of what's already being drawn to 
screen, then d3dvideosink doesn't support that. You can modify the buffer by 
creating your own plugin before it gets to d3dvideosink (e.g. the timeoverlay 
plugin). I'd imagine that's more a cross-platform way to go. However it 
wouldn't look as nice if you were forced to draw in a small frame that was then 
scaled up...

Original comment by david.g.hoyt on 22 Mar 2011 at 10:50

[GoogleCodeExporter]

These days, you might want to consider using autovideosink. d3dvideosink should 
gracefully downgrade to dshowvideosink/directdrawsink if it can't run correctly.

Original comment by david.g.hoyt on 22 Mar 2011 at 10:51

[GoogleCodeExporter]

I attach a very simple example which works both on windows and linux with 
gstreamer-java. if you run it you'll get the above error.
anyway how did you get window handle easily in SWT? we use this code in 
gstreamer-java:
http://code.google.com/p/gstreamer-java/source/browse/trunk/gstreamer-java/src/o
rg/gstreamer/swt/overlay/SWTOverlay.java#91
but if you know any portable and easier way it'd be useful...

Original comment by lfar...@gmail.com on 22 Mar 2011 at 11:55

Attachments:

• SWTVideoTestSrc.java

[GoogleCodeExporter]

I said "relatively easy" (c: -- IOW, you don't have to resort to JNI to get it 
(which is what JNA does under the covers for Native.getComponentID()). Using 
reflection is rather nasty, however -- and if my memory serves me, you only 
need to do that if you're using an older version of SWT.

What you're doing in SWTOverlay is pretty much what I've done -- I've used that 
very same class. But you need to add a synchronous bus listener, look for the 
xoverlay signals, and set the window handle at the appropriate time. I haven't 
peeked under the covers to see what VideoComponent does, but I assume it's 
doing that...

So is the problem that you're receiving an error message when it's unexpected? 
The one you posted is rather serious -- the video sink shouldn't work at all 
without an initialized device. The error is occurring in 
gst_d3dvideosink_release_swap_chain() which is presumably being called from 
gst_d3dvideosink_set_window_handle(). That function is meant to be called from 
the asynchronous bus handler. If you're doing that and still getting this 
error, then we need to reevaluate.

Original comment by david.g.hoyt on 23 Mar 2011 at 12:14

[GoogleCodeExporter]

could you show me how do you do it? and you solution is platform independent?
what do you mean by "older version of SWT"? how can i do it in newer SWT?

Original comment by lfar...@gmail.com on 23 Mar 2011 at 9:47

[GoogleCodeExporter]

IDirect3DDevice9_StretchRect uses D3DTEXF_NONE as rescaling function, it looks 
way better with D3DTEXF_LINEAR

Original comment by wintermu...@gmail.com on 23 Mar 2011 at 2:22

[GoogleCodeExporter]

also video caps have only YUV enabled and rgb/bgr is commented out

GST_VIDEO_CAPS_YUV("{ YUY2, UYVY, YV12, I420 }")
";" GST_VIDEO_CAPS_BGRx)

I can confirm it works ok with this settings
it can save one color conversion before renderer (we use BGR in our filter)

Original comment by wintermu...@gmail.com on 23 Mar 2011 at 2:26

[GoogleCodeExporter]

The problem is that the RGB/BGR doesn't work as expected with some video cards. 
In my machine xRGB was actually BGRx so we disabled it until we now how to 
interpret the endiannes properly or we get enough feedback

Original comment by ylatuya on 23 Mar 2011 at 2:46

[GoogleCodeExporter]

wintermute101: your change for using D3DTEXF_LINEAR was made in r988 (with some 
modifications to determine the device caps first). The RGB video caps won't be 
done for the reasons ylatuya stated. I've personally witnessed the problems he 
described.

Original comment by david.g.hoyt on 23 Mar 2011 at 8:08

[GoogleCodeExporter]

raimo.jarvi: applied your patch in r990.

Original comment by david.g.hoyt on 23 Mar 2011 at 8:22

[GoogleCodeExporter]

We are currently using Direct3D Video Sink with BGR enabled as we need it. We 
can provide you some feedback from intel and nvidia video cards (where it woks 
correctly). Just say what feedback you need.

Original comment by wintermu...@gmail.com on 24 Mar 2011 at 12:36

[GoogleCodeExporter]

The fact that it's actually working shows the issue itself :)
The caps are defined as GST_VIDEO_CAPS_BGRx, but the surface is initiate as 
D3DFMT_X8R8G8B8 which is xRGB. For me this combination worked, but for David 
GST_VIDEO_CAPS_xRGB worked instead.

Original comment by ylatuya on 24 Mar 2011 at 5:41

[GoogleCodeExporter]

unfortunately autovideosink can't be used in autolink environment:-( eg you 
can't use this (or at least won't work) on linux:
-------------------------------
Pipeline pipe = new Pipeline("Test");
Element src = ElementFactory.make("videotestsrc", "videotest");
Element sink = ElementFactory.make("autovideosink", "videosink");
pipe.addMany(src, sink);
Element.linkMany(src, sink);
pipe.play();
-------------------------------
while if you change from autovideosink to xvimagesink then it's works:-(
and that's why we can't use it as an overlay backend.

Original comment by lfar...@gmail.com on 25 Mar 2011 at 2:11

[GoogleCodeExporter]

Throw in an ffmpegcolorspace element and try again. Playbin2 should work fine 
-- in fact, by default I believe it uses autovideosink.

Original comment by david.g.hoyt on 25 Mar 2011 at 6:25

[GoogleCodeExporter]

If it works w/ xvimagesink as opposed to autovideosink, then it makes me wonder 
what video sink gstreamer is selecting for autovideosink. It sounds like it's 
not actually selecting xvimagesink. gst-inspect autovideosink shows that its 
caps are "ANY" which shouldn't limit its autoselection capabilities. What are 
the caps for xvimagesink? gst-inspect videotestsrc shows a wide number of caps, 
so it's likely that it's compatible with most video sinks directly. Honestly, 
something smells fishy...Then again, I'm not as well versed on the Linux side 
of the house...

Use GST_DEBUG=auto*:4 and then run gst-launch videotestsrc ! autovideosink and 
paste your output. That ought to tell you which sink has been selected.

Original comment by david.g.hoyt on 25 Mar 2011 at 6:44

[GoogleCodeExporter]

gst-launch videotestsrc ! autovideosink 
works and choose xvimagesinkg:-) that's what i try to debug today, but if i use 
the above simple example it's not working:-(
and in gstreamer-java if i try to choose autovideosink as a backend for 
VideoComponent (which could be useful on all platform) than the above simple 
example won't working with VideoComponent either.
the reason for this is that autovideosink hierarchy is (it's parent is Bin): 
GObject
 +----GstObject
       +----GstElement
             +----GstBin
                   +----GstAutoVideoSink
while xvimagesink is from GstBaseSink (which is good):
GObject
 +----GstObject
       +----GstElement
             +----GstBaseSink
                   +----GstVideoSink
                         +----GstXvImageSink
anyway i don't know the reason for this.

Original comment by vitorl...@gmail.com on 25 Mar 2011 at 8:09

[GoogleCodeExporter]

autovideosink as a bin makes sense since it detects at runtime which sink to 
use and then populates itself with it. Perhaps you need to add a ghostpad to 
autovideosink first 
(http://gstreamer.freedesktop.org/data/doc/gstreamer/head/manual/html/section-pa
ds-ghost.html).

Try using the gstreamer-java equivalent to gst_parse_bin_from_description() 
using "videotestsrc ! autovideosink" and see if that works.

Original comment by david.g.hoyt on 26 Mar 2011 at 12:07

[GoogleCodeExporter]

What's the best video sink on Linux? I take issue with xvimagesink because 
there's a limited number of simultaneous xvimagesinks you can use. With my 
video card, the maximum was 4 -- meaning you could only have 4 videos playing 
back w/ xvimagesink. Beyond that and it would either degrade to ximagesink or 
fail to function. I think, in terms of rank (disregarding availability), it 
should be glimagesink > xvimagesink > ximagesink. I'm not sure where something 
like dfbvideosink belongs.

Original comment by david.g.hoyt on 26 Mar 2011 at 12:14

[GoogleCodeExporter]

Compiling d3dvideosink with 64 bit mingw-w64 produces the following errors and 
warnings, the attached patch should fix these:

d3dvideosink.c: In function 'SharedHiddenWndProc':
d3dvideosink.c:585:44: warning: cast from pointer to integer of different size
d3dvideosink.c: In function 'WndProc':
d3dvideosink.c:722:44: warning: cast from pointer to integer of different size
d3dvideosink.c: In function 'gst_d3dvideosink_wnd_proc':
d3dvideosink.c:766:82: error: 'GWL_WNDPROC' undeclared (first use in this 
function)
d3dvideosink.c:766:82: note: each undeclared identifier is reported only once 
for each function it appears in
d3dvideosink.c: In function 'gst_d3dvideosink_window_thread':
d3dvideosink.c:915:41: warning: cast from pointer to integer of different size
d3dvideosink.c: In function 'gst_d3dvideosink_set_window_for_renderer':
d3dvideosink.c:1077:64: error: 'GWL_WNDPROC' undeclared (first use in this 
function)
d3dvideosink.c: In function 'gst_d3dvideosink_remove_window_for_renderer':
d3dvideosink.c:1268:68: error: 'GWL_WNDPROC' undeclared (first use in this 
function)

Original comment by raimo.ja...@gmail.com on 26 May 2011 at 9:06

Attachments:

• d3d.diff

[GoogleCodeExporter]

does the d3d-error.patch in #54 needed or not? since it's still not applied...

Original comment by lfar...@lfarkas.org on 3 Jun 2011 at 1:04

[GoogleCodeExporter]

As David writes in #56, the error was fixed in r969.

Original comment by raimo.ja...@gmail.com on 3 Jun 2011 at 1:27