please support this

[GoogleCodeExporter]

I'm seeing this project starting and I am pretty pleased that someone is trying 
to do it using DX.

I already have a feature request which will seem a little strange.

Please make this:

- filter attachable to the splitter subtitle output only
- allow filter to accept an interface on which to raise callbacks
- callbacks take as parameter the DX surface that the filter is using
- callbacks are raised for each frame the filter renders

This will make it possible to use the filter as an overlay image inside a WPF 
application instead of passing through the graph to the renderer. 
It is then possible to apply additional transformations on the subtitles / or 
not make them depending on the application settings.

For example:
- attach the subtitles to the video frame / or to the window
- scale / rotate subtitles with the video frame
- adjust contrast / brightness / opacity of subtitles

All this in real time, depending on user settings.

This is not possible in any player up to date with any existing combination of 
filters.

I've managed to do the image transformations by retaining video acceleration 
and I'm looking for a way to incorporate subtitles. Right now I can draw simple 
textual SRT subtitles (must be in same folder as file), but implementing 
ASS/SSA is miraculously hard, and it would be cool to have such a callback with 
a new surface.

You can see a demonstration here:
https://www.dropbox.com/s/7kvdd4wt18mkwpx/MediaPoint%202012.1%20x64.zip
Or attachment

Try:
- go fullscreen
- use mouse scroll to zoom
- press right mouse button and continue with scroll to rotate image

I can provide you with the interface implementation which I already have for 
the custom presenter that sends frames to the WPF application.

Needs .NET 3.5 and Windows 7 64

Original issue reported on code.google.com by marino.simic@gmail.com on 8 May 2012 at 1:08

Attachments:

• [MediaPoint 2012.1 x64.zip](https://storage.googleapis.com/google-code-attachments/dxsubfilter/issue-1/comment-0/MediaPoint 2012.1 x64.zip)

[GoogleCodeExporter]

Uh, thanks for the interest in the project I suppose. We can take your requests 
into consideration, but we first need to actually get this project working. 
Everything else comes later :)

Original comment by utopiafa...@gmail.com on 11 May 2012 at 6:22

[GoogleCodeExporter]

Of course that is out of question :)
First make it work :)

I can participate at some point when you manage to have something working :)

Original comment by marino.simic@gmail.com on 14 May 2012 at 2:05

[GoogleCodeExporter]

I've seen you made some progress and still have not initiated work on 
subtitlecore.
Please try to setup the subtitlecore to use a D3DSurface as input.
Have a DXEngine create yourdevice copy the data from the IMediaSample 
into/create a surface and make subtitlecore work only with DX.

Original comment by marino.simic@gmail.com on 20 May 2012 at 8:35

[GoogleCodeExporter]

This will make the subtitlecore reusable. I have a custom presenter which could 
then use just subtitlecore on its surfaces, instead of adding the filte rto the 
filtergraph. 

Original comment by marino.simic@gmail.com on 20 May 2012 at 8:38

[GoogleCodeExporter]

or at least support the subtitle core to take external surfaces with just the 
SubtitleInputPin connected. so that the video has not to come from the Filter 
itself but a surface being set each frame from outside.

Original comment by marino.simic@gmail.com on 20 May 2012 at 8:51

[GoogleCodeExporter]

Look, let us finish our work to a point that we're comfortable with and then 
you can start worrying about whether or not we should expose additional 
functionality that improve re-usage of our library. The DirectShow 
functionality is absolutely our highest priority and our design of subtitlecore 
will be beholden to the needs of our filter and our performance requirements.

Original comment by utopiafa...@gmail.com on 20 May 2012 at 8:55

[GoogleCodeExporter]

Well, finish it by all means. However I'm not interested much anymore in the 
project since it is not using Direct X. the name of the library mislead me into 
thinking that. It should be called dssubfilter or dwsubfilter.

I now see that the subtitle rendering is not done with Direct X on the GPU but 
it is filling memory buffers and then blending on the CPU with the IMEdiaSample 
buffer.

That wont provide any benefit for blending FullHD+ images to consumers, the CPU 
will never be idling.

Just my 2c.

Cheers

Original comment by marino.simic@gmail.com on 31 May 2012 at 2:25

[GoogleCodeExporter]

I'm not sure what you mean when you say it's not done with DirectX. DirectX is 
a suite of Microsoft libraries, of which DirectWrite and Direct2D are a part of 
(the X is just in there as a variable to denote that it can be replaced with 
more specific names). DirectWrite can interface with several rasterization 
systems, one of which is Direct2D, and since we use two DirectX technologies, I 
chose to just name our project dxsubfilter.

I'm also not sure you actually understand how Direct2D works. Yes, we do 
currently use a WICBitmapRenderTarget (and consequently the high performance 
software rasterizer of D2D) and copy out rectangles from the bitmap because 
that is what we need for our DirectShow filter and because that's pretty 
flexible in terms of reuse even outside of DirectShow. 

Your usage case isn't unreasonable either, albeit just more niche right now, 
and would only require 1 major change for us: use a DXGISurfaceRenderTarget. 
Everything else will just work and you'll get the GPU acceleration you're so 
concerned about. However, like I said earlier, we're not going to build our 
system around what your needs are in your specific project, but we are open to 
extending the interface we provide once we actually get everything done. (Also, 
in case you weren't aware, D2D/D3D interop requires D3D10.1 API usage with a 
D3D10 GPU and B8G8R8A8_UNORM DXGI surface format).

Anyway, if you feel like this project won't be able to assist with your 
project, then that's regrettable and I wish you luck in finding/building a 
library that meets your needs.

(P.S.: I will be on vacation for the next two weeks so if you make a reply 
after today, I won't be able to respond until after June 17.)

Original comment by utopiafa...@gmail.com on 31 May 2012 at 6:06

[GoogleCodeExporter]

I am more concerned about supporting IMFSample as input (you are now using
IMediaSample which is required to make it a filter).
But that wont work for a custom presenter (allocator) used by EVR.
IMFSample as part of Media Foundation already has a D3D surface inside it
and surely does not need DX10. As it is a D3DSurface9 and it works on a
patched Win XP.

What I though the project will be is that you will be doing this the other
way around:
- creating a D3D surface from the input frame of transform
- use D3D operations to blend subtitles, doing it on the GPU in the
subtitlecore
- send the resulting surface by rendering it as a bitmap on the output of
the transform

You are right, that I'm wrong when saying it is not done in DirectX. It is
not done on the GPU, while it uses the APIs that DirectX provides, just not
the D3D ().

I just had a very quick look at the source on googcode browser but if I
understand correctly the "Transform" method is asking subtitlecore to give
it the subtitle bitmap, then it is blending the input bitmap with the
subtitle bitmap and sending that to the output.

This is basically what any subtitle filter does, but just by using probably
a better API - but still on the GPU.

My idea when I first read the name "dxsubfilter" was something that will do
Video overlay usign the GPU functionalities of DirectX.
This article is nice to understand what I was thinking:
http://www.codeproject.com/Articles/207642/Video-Shadering-with-Direct3D

If this was the case and if the input was first converted to a surface,
then all the subtitling operations and blending, overlay, animations, color
convertion, stretching ... could be done using the gpu, and would still
support most of the output formats required by the filter, by rendering the
surface to a bitmap.

If the input was a surface then my use case would be feasible without much
modifications to subtitle core (one would be reusing the surface Dx device
from IMFSample for operations for example), while all the work would remain
on the GPU, and I would not require rendering the surface to a bitmap like
the Transform would need (but that would probably be faster than the
blending is right now anyway).

This would mean I can reuse the subtitle core INSIDE my EVR custom
presenter, without attaching it to the filter graph, and feed it with
IMFSample-s (which contain the media time etc). This process would be very
fast. However as it is now, I would need first to render the IMFSample to a
bitmap, then use subtitle core to get the bitmap for the subtitles, then
bledn those two, then create a new surface from the resulting bitmap, then
blend the new surface with the input surface.

That is not something you want to do with DXVA video accelerated custom
presenter :D

BTW, Have a nice holiday man.
I like your coding, you comment well and write nicely.

Original comment by marino.simic@gmail.com on 1 Jun 2012 at 11:12

[GoogleCodeExporter]

Sorry it's taken me so long to reply. I'm currently in the midst of moving to a 
new place for my new job so I haven't really had a chance to give this much 
thought. On that note, my ability to work on this project will be severely 
limited once I start work so I hope you're not relying on this project to reach 
a usable state anytime soon.

There are several issues at hand that I'll try to touch upon:

1. As it is now, the subtitlecore interface is basically designed for reuse 
with DShow Filters or Media Foundation Transforms. As you pointed out, you 
could also feasibly use the current interface with D3D surfaces, but it would 
be slow and ugly. That is correct because I have not built out the GPU 
accelerated interface yet - see next point.

2. Because subtitlecore relies on Direct2D for actual text rasterization, 
interop with D3D must follow the requirements Microsoft set out, namely D3D10.1 
API usage with D3D10 GPU hardware. More specifically, you must supply Direct2D 
with a DXGI surface in the B8G8R8A8_UNORM format. This is something that is out 
of my hands. See this link for more details: 
http://msdn.microsoft.com/en-us/library/windows/desktop/dd370966(v=vs.85).aspx

3. If you are able to extract an IDXGISurface or create one from the IMFSample, 
then I can extend subtitlecore to accept the IDXGISurface and draw subtitles 
directly to the video frame using the GPU. At this point, the process flow 
becomes like this:
  a. Your renderer uses subtitlecore internally and obtains the necessary renderers from the factory.
  b. Perform all necessary color space conversion and video processing until you have a B8G8R8A8_UNORM DXGI surface containing the video data.
  c. Pass this surface into the renderer and it will draw all the subtitles onto the video frame.
  d. Present the video frame.

4. However, this means you have to use the D3D10.1 API to process and present 
the video frames, which in turn means that you can't support Windows XP at all. 
In fact, just wanting to use subtitlecore means you need to be willing to not 
support Windows XP as DirectWrite/Direct2D requires Vista SP2 with Platform 
Update at a minimum.

Hopefully, this illuminates my reasoning and motivation behind not targeting 
GPU accelerated subtitle rendering right from the start (plus that wouldn't 
play nice with a DShow filter since the D3D surface map/unmap would decimate 
performance). It's not entirely infeasible to do, but it's very restrictive and 
probably not what you were hoping for. Do let me know your thoughts though.

Original comment by utopiafa...@gmail.com on 2 Jul 2012 at 2:53

[GoogleCodeExporter]

I was not aware of the DirectWrite requirements and DXGI. 

That puts everything in another perspective.

But then again who cares about WinXP, it is almost defunct.

The problem with the approach would still be present however because drawing is 
done on CPU side. And Map/Unmap is actually copying the buffer from the GPU to 
the CPU and viceversa, right ?

I was searching for a way to draw subtitles purely by using the data buffer on 
the GPU and GPU methods where the buffer is never swapped between CPU and GPU.

This seems not possible as of now :(

But it does not matter, we must use what we have been given and make the best 
of it :)

Original comment by marino.simic@gmail.com on 3 Jul 2012 at 10:14

[GoogleCodeExporter]

"The problem with the approach would still be present however because drawing 
is done on CPU side. And Map/Unmap is actually copying the buffer from the GPU 
to the CPU and viceversa, right ?"

No, Direct2D will use the GPU to accelerate glyph rasterization if I can use a 
DXGI render target. The only time it falls back to its software rasterizer is 
if I'm using the WICBitmapRenderTarget like I am now, or if it was unable to 
create the render target on the hardware device (e.g. if you were using the 
D3D10.1 API on a D3D9 GPU). Otherwise, it works exactly as you were hoping: 
font setup and text layout is still handled CPU side, but rasterization of 
glyph shapes, filling, and rotation is all done on the GPU. 

My comment about needing to map/unmap only applies if you were trying to do GPU 
accelerated text rendering in the context of a filter, which is basically 
infeasible because of the copy latency. However, if you do subtitle rendering 
as the last step in the video renderer before frame presentation like I 
outlined above, you can do everything on the GPU without needing to copy 
anything back CPU side.

All that being said, ASS is rather complex and Direct2D might not provide the 
necessary flexibility to support all its features. We'll see as I'm able to 
complete more of the SRT renderer.

Original comment by utopiafa...@gmail.com on 3 Jul 2012 at 4:00