Open GoogleCodeExporter opened 9 years ago
Uh, thanks for the interest in the project I suppose. We can take your requests
into consideration, but we first need to actually get this project working.
Everything else comes later :)
Original comment by utopiafa...@gmail.com
on 11 May 2012 at 6:22
Of course that is out of question :)
First make it work :)
I can participate at some point when you manage to have something working :)
Original comment by marino.simic@gmail.com
on 14 May 2012 at 2:05
I've seen you made some progress and still have not initiated work on
subtitlecore.
Please try to setup the subtitlecore to use a D3DSurface as input.
Have a DXEngine create yourdevice copy the data from the IMediaSample
into/create a surface and make subtitlecore work only with DX.
Original comment by marino.simic@gmail.com
on 20 May 2012 at 8:35
This will make the subtitlecore reusable. I have a custom presenter which could
then use just subtitlecore on its surfaces, instead of adding the filte rto the
filtergraph.
Original comment by marino.simic@gmail.com
on 20 May 2012 at 8:38
or at least support the subtitle core to take external surfaces with just the
SubtitleInputPin connected. so that the video has not to come from the Filter
itself but a surface being set each frame from outside.
Original comment by marino.simic@gmail.com
on 20 May 2012 at 8:51
Look, let us finish our work to a point that we're comfortable with and then
you can start worrying about whether or not we should expose additional
functionality that improve re-usage of our library. The DirectShow
functionality is absolutely our highest priority and our design of subtitlecore
will be beholden to the needs of our filter and our performance requirements.
Original comment by utopiafa...@gmail.com
on 20 May 2012 at 8:55
Well, finish it by all means. However I'm not interested much anymore in the
project since it is not using Direct X. the name of the library mislead me into
thinking that. It should be called dssubfilter or dwsubfilter.
I now see that the subtitle rendering is not done with Direct X on the GPU but
it is filling memory buffers and then blending on the CPU with the IMEdiaSample
buffer.
That wont provide any benefit for blending FullHD+ images to consumers, the CPU
will never be idling.
Just my 2c.
Cheers
Original comment by marino.simic@gmail.com
on 31 May 2012 at 2:25
I'm not sure what you mean when you say it's not done with DirectX. DirectX is
a suite of Microsoft libraries, of which DirectWrite and Direct2D are a part of
(the X is just in there as a variable to denote that it can be replaced with
more specific names). DirectWrite can interface with several rasterization
systems, one of which is Direct2D, and since we use two DirectX technologies, I
chose to just name our project dxsubfilter.
I'm also not sure you actually understand how Direct2D works. Yes, we do
currently use a WICBitmapRenderTarget (and consequently the high performance
software rasterizer of D2D) and copy out rectangles from the bitmap because
that is what we need for our DirectShow filter and because that's pretty
flexible in terms of reuse even outside of DirectShow.
Your usage case isn't unreasonable either, albeit just more niche right now,
and would only require 1 major change for us: use a DXGISurfaceRenderTarget.
Everything else will just work and you'll get the GPU acceleration you're so
concerned about. However, like I said earlier, we're not going to build our
system around what your needs are in your specific project, but we are open to
extending the interface we provide once we actually get everything done. (Also,
in case you weren't aware, D2D/D3D interop requires D3D10.1 API usage with a
D3D10 GPU and B8G8R8A8_UNORM DXGI surface format).
Anyway, if you feel like this project won't be able to assist with your
project, then that's regrettable and I wish you luck in finding/building a
library that meets your needs.
(P.S.: I will be on vacation for the next two weeks so if you make a reply
after today, I won't be able to respond until after June 17.)
Original comment by utopiafa...@gmail.com
on 31 May 2012 at 6:06
I am more concerned about supporting IMFSample as input (you are now using
IMediaSample which is required to make it a filter).
But that wont work for a custom presenter (allocator) used by EVR.
IMFSample as part of Media Foundation already has a D3D surface inside it
and surely does not need DX10. As it is a D3DSurface9 and it works on a
patched Win XP.
What I though the project will be is that you will be doing this the other
way around:
- creating a D3D surface from the input frame of transform
- use D3D operations to blend subtitles, doing it on the GPU in the
subtitlecore
- send the resulting surface by rendering it as a bitmap on the output of
the transform
You are right, that I'm wrong when saying it is not done in DirectX. It is
not done on the GPU, while it uses the APIs that DirectX provides, just not
the D3D ().
I just had a very quick look at the source on googcode browser but if I
understand correctly the "Transform" method is asking subtitlecore to give
it the subtitle bitmap, then it is blending the input bitmap with the
subtitle bitmap and sending that to the output.
This is basically what any subtitle filter does, but just by using probably
a better API - but still on the GPU.
My idea when I first read the name "dxsubfilter" was something that will do
Video overlay usign the GPU functionalities of DirectX.
This article is nice to understand what I was thinking:
http://www.codeproject.com/Articles/207642/Video-Shadering-with-Direct3D
If this was the case and if the input was first converted to a surface,
then all the subtitling operations and blending, overlay, animations, color
convertion, stretching ... could be done using the gpu, and would still
support most of the output formats required by the filter, by rendering the
surface to a bitmap.
If the input was a surface then my use case would be feasible without much
modifications to subtitle core (one would be reusing the surface Dx device
from IMFSample for operations for example), while all the work would remain
on the GPU, and I would not require rendering the surface to a bitmap like
the Transform would need (but that would probably be faster than the
blending is right now anyway).
This would mean I can reuse the subtitle core INSIDE my EVR custom
presenter, without attaching it to the filter graph, and feed it with
IMFSample-s (which contain the media time etc). This process would be very
fast. However as it is now, I would need first to render the IMFSample to a
bitmap, then use subtitle core to get the bitmap for the subtitles, then
bledn those two, then create a new surface from the resulting bitmap, then
blend the new surface with the input surface.
That is not something you want to do with DXVA video accelerated custom
presenter :D
BTW, Have a nice holiday man.
I like your coding, you comment well and write nicely.
Original comment by marino.simic@gmail.com
on 1 Jun 2012 at 11:12
Sorry it's taken me so long to reply. I'm currently in the midst of moving to a
new place for my new job so I haven't really had a chance to give this much
thought. On that note, my ability to work on this project will be severely
limited once I start work so I hope you're not relying on this project to reach
a usable state anytime soon.
There are several issues at hand that I'll try to touch upon:
1. As it is now, the subtitlecore interface is basically designed for reuse
with DShow Filters or Media Foundation Transforms. As you pointed out, you
could also feasibly use the current interface with D3D surfaces, but it would
be slow and ugly. That is correct because I have not built out the GPU
accelerated interface yet - see next point.
2. Because subtitlecore relies on Direct2D for actual text rasterization,
interop with D3D must follow the requirements Microsoft set out, namely D3D10.1
API usage with D3D10 GPU hardware. More specifically, you must supply Direct2D
with a DXGI surface in the B8G8R8A8_UNORM format. This is something that is out
of my hands. See this link for more details:
http://msdn.microsoft.com/en-us/library/windows/desktop/dd370966(v=vs.85).aspx
3. If you are able to extract an IDXGISurface or create one from the IMFSample,
then I can extend subtitlecore to accept the IDXGISurface and draw subtitles
directly to the video frame using the GPU. At this point, the process flow
becomes like this:
a. Your renderer uses subtitlecore internally and obtains the necessary renderers from the factory.
b. Perform all necessary color space conversion and video processing until you have a B8G8R8A8_UNORM DXGI surface containing the video data.
c. Pass this surface into the renderer and it will draw all the subtitles onto the video frame.
d. Present the video frame.
4. However, this means you have to use the D3D10.1 API to process and present
the video frames, which in turn means that you can't support Windows XP at all.
In fact, just wanting to use subtitlecore means you need to be willing to not
support Windows XP as DirectWrite/Direct2D requires Vista SP2 with Platform
Update at a minimum.
Hopefully, this illuminates my reasoning and motivation behind not targeting
GPU accelerated subtitle rendering right from the start (plus that wouldn't
play nice with a DShow filter since the D3D surface map/unmap would decimate
performance). It's not entirely infeasible to do, but it's very restrictive and
probably not what you were hoping for. Do let me know your thoughts though.
Original comment by utopiafa...@gmail.com
on 2 Jul 2012 at 2:53
I was not aware of the DirectWrite requirements and DXGI.
That puts everything in another perspective.
But then again who cares about WinXP, it is almost defunct.
The problem with the approach would still be present however because drawing is
done on CPU side. And Map/Unmap is actually copying the buffer from the GPU to
the CPU and viceversa, right ?
I was searching for a way to draw subtitles purely by using the data buffer on
the GPU and GPU methods where the buffer is never swapped between CPU and GPU.
This seems not possible as of now :(
But it does not matter, we must use what we have been given and make the best
of it :)
Original comment by marino.simic@gmail.com
on 3 Jul 2012 at 10:14
"The problem with the approach would still be present however because drawing
is done on CPU side. And Map/Unmap is actually copying the buffer from the GPU
to the CPU and viceversa, right ?"
No, Direct2D will use the GPU to accelerate glyph rasterization if I can use a
DXGI render target. The only time it falls back to its software rasterizer is
if I'm using the WICBitmapRenderTarget like I am now, or if it was unable to
create the render target on the hardware device (e.g. if you were using the
D3D10.1 API on a D3D9 GPU). Otherwise, it works exactly as you were hoping:
font setup and text layout is still handled CPU side, but rasterization of
glyph shapes, filling, and rotation is all done on the GPU.
My comment about needing to map/unmap only applies if you were trying to do GPU
accelerated text rendering in the context of a filter, which is basically
infeasible because of the copy latency. However, if you do subtitle rendering
as the last step in the video renderer before frame presentation like I
outlined above, you can do everything on the GPU without needing to copy
anything back CPU side.
All that being said, ASS is rather complex and Direct2D might not provide the
necessary flexibility to support all its features. We'll see as I'm able to
complete more of the SRT renderer.
Original comment by utopiafa...@gmail.com
on 3 Jul 2012 at 4:00
Original issue reported on code.google.com by
marino.simic@gmail.com
on 8 May 2012 at 1:08Attachments: