Open GoogleCodeExporter opened 9 years ago
Hmmm... Buffering is already possible right now with the interface as it is.
You can simply request several subtitles ahead for future video frames and then
store them in a queue. At least that's what madVR is doing.
Maybe you can take the current interface header, add-in your needed new methods
and upload the enhanced header somewhere for us to look at? I think that would
make it easier for us to understand what exactly you need. I think I would
rather put all that stuff into a compatability interface with a new name (e.g.
IMpcIsrCompatabilityLayer or something like that), though, but that's fine
tuning.
Also I'm not sure if our XySubFilter dev is willing to add support for what you
need, we'll have to ask him. But first of all we need exact information about
what you would need exactly.
Original comment by mad...@gmail.com
on 22 Jul 2013 at 10:17
> Hmmm... Buffering is already possible right now with the interface as it is.
You can simply request several subtitles ahead for future video frames and then
store them in a queue. At least that's what madVR is doing.
True, but how do you do invalidation then?
Original comment by Jules.B...@gmail.com
on 22 Jul 2013 at 10:19
Invalidation? You mean if the user selects a different subtitle track? To be
honest, I'm not sure how that works. I suppose XySubFilter disconnects and
reconnects in that situation to clear madVR's subtitle queue? At least in the
moment when XySubFilter disconnects, madVR clears the subtitle queue, but I'm
not sure if XySubFilter does disconnect/reconnect in that situation...
Original comment by mad...@gmail.com
on 22 Jul 2013 at 10:27
> Hmmm... Buffering is already possible right now with the interface as it is.
You can simply request several subtitles ahead for future video frames and then
store them in a queue. At least that's what madVR is doing.
Wouldn't it be more convenient then if you knew for how long those subtitles
are displayed? Then you don't need to keep calling RequestFrame for the same
subtitle.
Original comment by Jules.B...@gmail.com
on 22 Jul 2013 at 10:31
[deleted comment]
Just checked: When changing subtitle tracks, playback is stopped and restarted.
The stopping clears madVR's subtitle queue.
> Wouldn't it be more convenient then if you knew
> for how long those subtitles are displayed?
> Then you don't need to keep calling RequestFrame
> for the same subtitle.
The madVR frame queue has an information record for each video frame. One field
of that record is a pointer to an ISubRenderFrame. So calling RequestFrame()
once for each video frame and getting an ISubRenderFrame instance for each
video frame is actually very intuitive/convenient for madVR's queues.
Of course it would also be possible for the consumer to check how long that
subtitle frame is valid and not request a new subtitle until that time is over.
That would be an alternative approach, not necessarily worse or better than the
current approach, just different. For me, personally, the current interface
approach works better. But that's simply due to how the madVR queues are
working internally...
Original comment by mad...@gmail.com
on 22 Jul 2013 at 10:38
And our queue works differently, so allowing a slightly different approach
would make it more convenient for us (and possibly others who would rather
implement a queue based on timestamps).
Ofcourse I can just implement a new queue that behaves the same way as your
queue including buffering ahead, but it would cost me more time to do so. Also
I think the approach would make less sense since subtitles are separately
buffered from the frames.
I'll add the relevant functions to a suggested interface, but before I make a
suggestion I would like to know whether the subtitle id is known ahead of time,
before it is renderer/requested.
Original comment by Jules.B...@gmail.com
on 22 Jul 2013 at 10:51
> Wouldn't it be more convenient then if you knew for how long those subtitles
are
> displayed? Then you don't need to keep calling RequestFrame for the same
subtitle.
You need to do that anyway, a second subtitle rect may be available for another
frame, for example, while the first still remains valid.
Original comment by h.lepp...@gmail.com
on 22 Jul 2013 at 10:59
> You need to do that anyway, a second subtitle rect may be available for
another frame, for example, while the first still remains valid.
Only animated subtitles support that, they are requested frame-by-frame. If the
devs don't like supporting that animated/static distinction then I have to
implement a new queue for XySubFilter. I don't mind doing that if the devs
think such a distinction should not be part of the interface. I'm just trying
to save some time, so I'm asking whether the xy devs want to provide that
information.
Original comment by Jules.B...@gmail.com
on 22 Jul 2013 at 11:09
Why would static subs not support multiple rects at the same time?
I could show one line of text at the top from frame 1 to 10, and a second line
of text at the bottom from frame 5 to 15, and both are static and valid for a
prolonged time, however if you ask at frame 1, you only get the top line, valid
until frame 10, so you still need to keep asking every frame to find the new
subs at frame 5.
I believe PGS for example can do this, and i see no reason why many
non-animated formats wouldn't be capable of this.
Original comment by h.lepp...@gmail.com
on 22 Jul 2013 at 11:13
Another use case would be a provider that even renders two sub streams at the
same time.
Its a request that comes up again and again, people that want two languages at
the same time for some reason or another, which could show one top and one
bottom, both static text (like SRT), but possibly with different timings due to
differences in the authoring.
Original comment by h.lepp...@gmail.com
on 22 Jul 2013 at 11:17
> I would like to know whether the subtitle id
> is known ahead of time
There isn't really a subtitle id. There only is a bitmap id. Every subtitle can
consist of 0, one or several bitmaps. Every bitmap has an id. E.g. if a static
bitmap is moving over the screen, the bitmap id will stay identical, but the
bitmap position will change in subsequent subtitle frames.
If you need a way to uniquely identify subtitle frames, you can assign a
"context" in your RequestFrame() call. This context will be returned to you in
the DeliverFrame() call. madVR is using this to assign a unique id to each
subtitle. You could do the same.
Original comment by mad...@gmail.com
on 22 Jul 2013 at 11:20
You're right, I misunderstood the current subtitle queue, it won't stop
requesting subtitles for every frame it'll just request what subtitles are
displayed at the timestamp before requesting the actual subtitle frame. Also,
it supports multiple subtitles at the same timestamp so that's not a problem.
Original comment by Jules.B...@gmail.com
on 22 Jul 2013 at 11:30
I have very little time to give it a full review right now, so I am basically
listing what I believe is important to myself and integration with Zoom Player:
1. Easy method of integration
2. No auto-loading, or at least an easy way to disable it. I only want the
filter to load when I detect external sub files or a subtitle output pin on a
splitter filter.
3. Control of subtitles on-screen position.
4. Control of subtitle synchronization.
5. Distinguishing between embedded and external file-based subtitle tracks.
6. A pascal version of the header files.
Other than #1, these are all features that exist in xy-vsfilter, which I
require to maintain functionality.
Original comment by inmat...@gmail.com
on 22 Jul 2013 at 11:32
You will probably need a pascal version of the header files only if you wanted
to create your own Provider or Consumer based on this new subtitle interface.
If you just want to make use of XySubFilter + madVR, you probably don't need
direct access to the new interface at all, I think. Or did you plan to make
your own renderers (VMR9 etc) capable of working with XySubFilter, too? In that
case you definitely do need a pascal version of the header files...
Original comment by mad...@gmail.com
on 22 Jul 2013 at 11:36
> Invalidation? You mean if the user selects a different subtitle track?
> To be honest, I'm not sure how that works. I suppose XySubFilter
> disconnects and reconnects in that situation to clear madVR's subtitle
> queue? At least in the moment when XySubFilter disconnects, madVR clears
> the subtitle queue, but I'm not sure if XySubFilter does disconnect/reconnect
> in that situation...
Nothing has been done to deal with that...I hate madVR's cache queue and hope
users won't set it too big so that they won't notice the delay with their
operation...
Well, a "InvalidateTime" field is necessary.
@Jules.Blok
I'll agree to add a method, returning the current frame's display times, in the
provider for you as long as it can be simple and clear enough. I know with
ISubPicProvider, you can get a POSITION by a call to GetStartPosition, and then
that POSITION can be sent to GetStart/GetStop/IsAnimated. But the POSITION can
be invalidated in between those calls so you need to lock first and unlock when
you finish. Ah that is too much. I'll stay with the current interface if going
that way. The idea in my mind is:
STDMETHOD(GetFrameStartStopAt)(REFERENCE_TIME rt, REFERENCE_TIME &start, REFERENCE_TIME &stop);
I don't like it at all. As the name suggests, it should return subtitle frame
start/stop time AT time rt. It can always return start/stop as rt/rt+1. When
there's no subtitles, the returned start/stop should be equal to rt. But I
guess that is not what you want. It would be better if it returns the
start/stop time of the next subtitle frame when there's no subtitle at time rt.
That is UNCLEAR/UNCLEAN IMO. And another problem is we already have
STDMETHOD(RequestFrame)(REFERENCE_TIME start, REFERENCE_TIME stop, LPVOID context);
which request a subtitle frame with two timestamps: a start AND stop time. Use
a single rt to query frame start/stop time may lead to some mis-understanding
too.
Do you have any better idea?
Original comment by YuZhuoHu...@gmail.com
on 22 Jul 2013 at 1:18
> Nothing has been done to deal with that.
> I hate madVR's cache queue and hope users
> won't set it too big so that they won't
> notice the delay with their operation...
Well, I'm open for suggestions. We could add a "Clear()" command to the
consumer interface which you could call when the user switches subtitles
tracks. Would that solve your dislike of the madVR cache queue?
Original comment by mad...@gmail.com
on 22 Jul 2013 at 4:14
These are the changes I'm proposing:
https://dl.dropboxusercontent.com/u/10853398/SubRenderIntf.h
The GetStart() and GetStop() information I need from the subtitle frame is
already being retrieved from the provider in XySubRenderProviderWrapper.cpp at
lines 87, 132, 318 and 388. The "animated" field should just return
IsAnimated() from the subtitle provider.
With these additions I believe I can more easily provide support for
XySubFilter and they may also be useful for others. These additions will
require no changes in the way madVR handles subtitles.
Another option would be that instead of adjusting SubRenderIntf.h you provide
us access to the ISubPicProvider interface. But in that case you'd end up
maintaining two interfaces instead of one.
Original comment by Jules.B...@gmail.com
on 22 Jul 2013 at 4:24
> Well, I'm open for suggestions. We could add a "Clear()" command to the
consumer interface which you could call when the user switches subtitles
tracks. Would that solve your dislike of the madVR cache queue?
I would also support an invalidation call to the subtitle consumer.
Original comment by Jules.B...@gmail.com
on 22 Jul 2013 at 4:25
Personally i think relying on some flag like "IsAnimated" indicates a design
failure in the consumer.
In any case, if that information is exported, it should simply be a property
through the options interface, no need for a function.
Original comment by h.lepp...@gmail.com
on 22 Jul 2013 at 4:32
I'm not sure I understand the purpose of the "index" parameter in your
"GetStart()" and "GetStop()" functions? Do you want to know when each bitmap of
the current subtitle frame starts/stops displaying? Or do you want to know when
the overall subtitle frame (eventually consisting of several different bitmaps)
starts/stops displaying?
My suggestion would be to join both new methods into one like this:
// Retrieve when this subtitle starts/stops displaying.
STDMETHOD(GetStartStopTimes)(REFERENCE_TIME * start, REFERENCE_TIME * stop) = 0;
Generally, since this is an addition to the provider side of the subtitle
interface, I'd like to know what yuzhuohuang thinks about your proposed
changes. I could certainly live with them.
Original comment by mad...@gmail.com
on 22 Jul 2013 at 4:33
> Personally i think relying on some flag like "IsAnimated" indicates a design
failure in the consumer.
I won't argue against design failures in the SubPicQueue, I just want to try
and quickly provide support for the interface without having to reimplement
SubPicQueue. We may change the way this is done in the future, in which case we
can see if we want to deprecate the additions again.
> In any case, if that information is exported, it should simply be a property
through the options interface, no need for a function.
I've already made it an optional field in the subtitle provider that is assumed
to be true.
Original comment by Jules.B...@gmail.com
on 22 Jul 2013 at 4:37
[deleted comment]
> This part, the "animated" flag indicates whether the bitmaps need to be
refreshed every frame.
I'm not sure i understand this part, really.
You need to ask for the subs for every frame anyway to find new bitmaps, and
it'll give you all the subs again - so you get the id of the bitmaps, which
will tell you if its the same, or a new one.
With the id you can determine if you need to refresh a sub or can just keep the
old one.
Original comment by h.lepp...@gmail.com
on 22 Jul 2013 at 4:44
> I'm not sure I understand the purpose of the "index" parameter in your
"GetStart()" and "GetStop()" functions?
I want to know what the timings are of the subtitle the bitmap belongs to. The
bitmap may change between GetStart() and GetStop(), which is indicated by the
"animated" field.
If yuzhuohuang doesn't agree to the additions then I'm prepared to make a
different SubPicQueue implementation specifically for XySubFilter which does
not rely on these timings.
Original comment by Jules.B...@gmail.com
on 22 Jul 2013 at 4:57
> I want to know what the timings are of the
> subtitle the bitmap belongs to. The bitmap
> may change between GetStart() and GetStop(),
> which is indicated by the "animated" field.
I'm now officially confused. If the bitmap does change, then I would expect
GetStart/GetStop to reflect that. Otherwise GetStart/GetStop would be
misleading! And I'm still not sure if the "index" parameters makes sense or not.
Let's talk about an example. There are 3 separate subtitle bitmaps:
(1) A static subtitle line at the top of the screen, which stays there during
the first minute of runtime.
(2) A small bitmap at the middle of the screen which moves from left to right,
but only the position of the bitmap changes, not the bitmap itself. And the
position changes once every second.
(3) An animated bitmap at the bottom of the screen which totally changes once
every second.
In this situation the start/stop times of bitmap (1) would be 0 seconds and 60
seconds. The start/stop times of bitmap (2) would also be 0 seconds and 60
seconds. Although this is debatable because although the bitmap (2) itself
doesn't change, the position does change. So one could also argue that the
start/stop times should only cover 1 second. The start/stop times of bitmap (3)
would be 0 seconds and 1 seconds. After the first second of runtime, it would
be 1 seconds and 2 seconds.
Now if we're talking about the start/stop times of the whole combined subtitle
(consisting of the 3 mentioned bitmaps), I would say the start/stop times would
be 0 seconds and 1 seconds.
So which of the above mentioned times is it exactly what you want to get from
GetStart/GetStop?
Original comment by mad...@gmail.com
on 22 Jul 2013 at 5:14
It's the start/end time of the subtitle as defined in the subtitle file. Every
bitmap is associated with such a subtitle, right?
Original comment by Jules.B...@gmail.com
on 22 Jul 2013 at 5:30
Is there such a thing as "the subtitle"? Can't ASS files have multiple subtitle
elements which are active at the same time?
I'm still not totally sure if I understand what you exactly want to get from
GetStart/GetStop. I'm starting to wonder if your definition of start/end time
fits at all into the current interface concept. But I'm not really sure I fully
understand it all. It might be my lack of knowledge in how ASS works exactly.
cyberbeing/yuzhuohuang? Can you help out? What do you think?
Original comment by mad...@gmail.com
on 22 Jul 2013 at 5:38
> Can't ASS files have multiple subtitle elements which are active at the same
time?
Yes they can, they each have a bitmap if I understand correctly. So for each
bitmap I want to know what the start/stop times are of the associated subtitle.
I want to make clear that I'm just considering what my options are for
integrating XySubFilter. I want to know if it makes sense for the interface to
expose such information so I can reuse the current subtitle queue.
I'm hearing a lot of arguments against exposing this information, so I guess
for the time being I'll concentrate on implementing a new subtitle queue for
XySubFilter. The discussion has given me more insight in how the subtitle queue
should behave, so it helps me implement one for XySubFilter.
Original comment by Jules.B...@gmail.com
on 22 Jul 2013 at 6:03
@madshi
Let's take this subtitle for example:
https://dl.dropboxusercontent.com/u/10853398/test.ass
It has a subtitle that is displayed from 0 until 2.6 seconds and a background
element displayed for the same amount of time. So I expect, correct me if I'm
wrong, we only get two bitmaps every frame. My interface additions allow you to
retrieve the display time of the subtitles associated with those bitmaps, in
this case for both "start" is 0s and "stop" is 2.6s.
Anyway it looks like some of what it tries to achieve is now handled by
XySubFilter, so it wouldn't make much sense adapting the interface for it. So
put my suggestions on hold for now.
The point about Invalidation is still stands though, if the consumer is going
to buffer subs ahead of time it needs to be informed when they are no longer
valid.
Original comment by Jules.B...@gmail.com
on 22 Jul 2013 at 7:29
Yes, I agree, some sort of "Clear()" or "FlushCache()" or something like that
would probably be a useful addition.
Original comment by mad...@gmail.com
on 22 Jul 2013 at 9:37
> Let's take this subtitle for example
In that script example, XySubFilter decides to combine the overlapping lines
before passing them to the Subtitle Consumer. So we actually have 8 unique
Bitmap IDs in that script, with each unique bitmap being used for multiple
video frames until the next is needed. Internally though, static elements such
as that background vector shape are only rendered once by XySubFilter at the
very beginning, then cached for future lines. Each letter of "T" "E" "S" "T"
"I" "N" "G" will also have multiple elements required for rendering entirely
cached.
> Anyway it looks like some of what it tries to achieve is now handled by
XySubFilter,
> so it wouldn't make much sense adapting the interface for it.
Many of the most fruitful optimizations we've done in xy-VSFilter/XySubFilter,
were based on numerous caches designed to eliminate redundant computation of
identical data. All in all, how XySubFilter decides to optimize internally
doesn't have much to do with the subtitle interface, and it doesn't need to.
Yet even if something isn't particularly useful to XySubFilter, there is still
the question if the subtitle interface should guide future Subtitle Providers
towards smart optimizations, and if Subtitle Consumers should play a part in
such optimizations. I have mixed feelings about this, since keeping the
interface itself as simple as possible has its own benefits.
Original comment by cyber.sp...@gmail.com
on 22 Jul 2013 at 11:52
Thanks for clearing that up, I'll start work on a custom subtitle queue for
XySubFilter tomorrow.
Original comment by Jules.B...@gmail.com
on 23 Jul 2013 at 12:03
> I want to know what the timings are of the
> subtitle the bitmap belongs to. The bitmap
> may change between GetStart() and GetStop(),
> which is indicated by the "animated" field.
That is why I don't like IsAnimated. GetStart/GetStop should always work as
they suggest.
@Jules.Blok
> It's the start/end time of the subtitle as defined in the subtitle file.
> Every bitmap is associated with such a subtitle, right?
No. One subtitle line in the subtitle file can be devided to several bitmaps,
e.g. if it contains several "\N"s. A bitmap may contain several subtitle lines.
So it is not easy to get precise start/stop time of each bitmap.
Anyway, I doubt if that is going to help you at all. If you get several bitmaps
that have different start/stop times and to really make use of them, you'll
have to devide the timespan into several segments using the those timestamps.
Within each segment different group of bitmaps are used. It would NOT simply
your work, is it?
Original comment by YuZhuoHu...@gmail.com
on 23 Jul 2013 at 12:36
I've got it working with EVR-CP and probably the other renderers that support
the ISR also. I just have one small problem: To get it to display the subtitles
correctly I had to invert the alpha values (D3DBLEND_INVSRCALPHA). Is this the
expected output of XySubFilter or did something go wrong in my code?
Original comment by Jules.B...@gmail.com
on 24 Jul 2013 at 8:17
Attachments:
D3DBLEND_ONE for Src and D3DBLEND_INVSRCALPHA for Dst is the correct mode for
blending the RGBA format we defined. Not sure how the ISR rendered the subs to
require something else.
Original comment by h.lepp...@gmail.com
on 24 Jul 2013 at 8:32
> I've got it working with EVR-CP and probably the
> other renderers that support the ISR also.
Great!! Any problems? Or was it mostly smooth sailing? Did you implement some
kind of queue?
Original comment by mad...@gmail.com
on 24 Jul 2013 at 9:41
@Jules, if you want to get the rendering code madVR uses for blending the
subtitles (including clip handling etc) just drop me an email, or send a PM @
doom9. Not sure if it would be helpful for you, but maybe you haven't
implemented clipping yet, so my code might help saving a bit time there...
Original comment by mad...@gmail.com
on 24 Jul 2013 at 10:00
@madshi, no queueing or clipping yet and the current code was mainly aimed at
just getting it to work, so it's also too slow. Any implementation details you
can share would be helpful, you can send me an e-mail at armada@mpc-hc.org.
Original comment by Jules.B...@gmail.com
on 24 Jul 2013 at 10:56
Ah, you're armada, didn't know that! :-) I've already sent you my rendering
code via PM in doom9.
Original comment by mad...@gmail.com
on 25 Jul 2013 at 5:57
>> To get it to display the subtitles correctly I had to invert
>> the alpha values (D3DBLEND_INVSRCALPHA).
The alpha values is inverted in comparison to ISR. XySubFilter has a hidden
bool field "useDestAlpha". If the field is set to true, you should get the same
alpha type as ISR.
Original comment by YuZhuoHu...@gmail.com
on 25 Jul 2013 at 6:15
But don't use that, or you are already violating the spec of this interface. :(
Original comment by h.lepp...@gmail.com
on 25 Jul 2013 at 7:22
Don't worry, D3DBLEND_INVSRCALPHA is the least of the problems I'm facing. I'm
currently concentrating on improving the performance, because currently it's
about as slow as the ISR. xD
Original comment by Jules.B...@gmail.com
on 25 Jul 2013 at 11:12
I've attached an XySubFilter 3.1.0.546 build with verbose logging enabled, just
in case you haven't built one for yourself yet. With the included properties
file, it will write a log to your C:\ directory. If you are unsure what is
causing the performance issue in your Consumer, maybe our logs could help track
it down.
Original comment by cyber.sp...@gmail.com
on 25 Jul 2013 at 12:16
Attachments:
Performance problem solved, I didn't notice that I was still using a version of
XySubFilter that I compiled in the debug configuration.
Original comment by Jules.B...@gmail.com
on 25 Jul 2013 at 12:55
Good to hear.
It's unfortunate, but our Debug builds really are nearly useless for any
testing because they are so slow. Sometimes they are unable to render even
relatively simple scripts in real-time with a high-end cpu. Depending on the
information you need, its often more useful to use a Release log build like
I've attached instead. Performance impact with log builds is usually very
minimal, except on insane scripts where thousands of lines per second are being
rendered.
Original comment by cyber.sp...@gmail.com
on 25 Jul 2013 at 2:33
Why are your debug builds so slow? madVR writes gigabytes of logs in its debug
build without any noticeable drop of performance... ;-)
Original comment by mad...@gmail.com
on 25 Jul 2013 at 2:44
All the crazy inline assembly and intrinsics get butt-slow in debug mode, as it
moves all operands from registers into memory between all operations, so you
can actually debug that.
Original comment by h.lepp...@gmail.com
on 25 Jul 2013 at 2:45
Oh, I see.
Original comment by mad...@gmail.com
on 25 Jul 2013 at 2:47
I'm getting a crash from XySubFilter when my videoOutputRect is empty, which
happens sometimes when restarting playback. Indicating that something is wrong
by returning E_FAIL on GetRect will prevent the crash, but causes XySubFilter
to disconnect itself.
I'll fix this issue by preventing a frame from being requested in case the
videoOutputRect is empty, but in this case XySubFilter should just fail the
frame request instead of disconnecting or crashing.
Original comment by Jules.B...@gmail.com
on 25 Jul 2013 at 3:38
Original issue reported on code.google.com by
cyber.sp...@gmail.com
on 16 Jul 2013 at 6:50