quarnster
commented
10 years ago @KeyserSoze1 wrote:
I've been making changes to some of the audio interface and attempting to add some features that were originally in the boxee code. However I have been finding myself breaking the video in many cases.
The renderer seems to be taking the passed in pts value (which is modified by the audio delay and the display latency?). And then determine how much to add to the pts value. All of this seems to be predicated on the current clock time. I'm trying to understand how all of this is setup and intended to work so I can then understand how I'm affecting it and then make changes if I need to (on the audio or video end).
The original xbmc code was clearly written without hardware acceleration in mind, so I had to hack to counteract these bad expectations made by xbmc's dvdplayer component.
I recommend reading this post and references for some additional info: https://github.com/quarnster/boxeebox-xbmc/issues/5#issuecomment-24811382.
Don't know your background, but hardware does a lot of stuff in parallel; it has a pipeline of some depth where giving it input data it will take a number of cycles until we see the result. This delay is one form of latency.
If we think of this in a software kind of way applied to real life we'd have one guy with a bucket of water running from the lake to the fire and back again. Each step is a clock cycle, the total number of steps from filling the bucket until our guy is back and ready to fill it again is the latency. With software the throughput is the latency, in other words there's no way our guy can do this work any more efficient than he already is.
Hardware in this analogy is to instead have a person and two buckets at each step. Each cycle, each person forwards one bucket full of water to the next person in line, and passes an empty bucket back to the previous person. The latency in this example is still the same; there's no way that the water will get from the lake to the fire any faster, the throughput however has been radically improved; we are able to throw one bucket of water on the fire each cycle.
You quickly realize that what matters here is throughput not latency, and this is typically also the case with hardware where each cycle each "person" doesn't do that much and instead you make up for it by having a lot of people helping out.
Getting back to xbmc, it presumes that the video latency is the throughput and that there is only a single guy doing all the work from reading the data from file to finally displaying the decoded image on screen. In the hardware case this means that while each person is still passing the two buckets to the next and the previous person in line, only a single bucket is actually filled with water at any given time in the system. These wasted "cycles" are called pipeline bubbles as the pipeline is not transporting anything other than air when it should have transported water. Wasteful, but not necessarily a problem as long as the latency is low enough to be able to keep up with this 1/latency inefficency. But name one hardware vendor who report their hardware's capabilities when it's used at 1/latency "efficiency".
The hardware is not only capable of decoding video frames, it can scale, rotate, and display them on the screen at the appropriate time as determined by a timestamp, completely independent of the 3D graphics hardware. In other words, we are in charge of feeding the hardware with data to decode, but for the rest of the pipeline it takes care of itself just fine. It knows when the appropriate time to display a frame is by having the input buffers marked with a timestamp and comparing this timestamp to a hardware clock running at IIRC 90kHz.
The original boxee code ignored xbmc's time sync code and wrote its own code to handle it instead. That introduced a lot of code just dealing with time sync for boxee box specifically and would have to be "re-ported" back to the more recent xbmc code this effort was initially based on. Alternatively it could be thrown out and simplified by making use of XBMC timestamp synchronization code. I chose, perhaps wrong, to do the latter.
Unfortunately again, as xbmc thinks the 3D hardware is in charge of displaying video frames on the screen at the right time and that the guy who's responsible for this task is the same guy who is in charge of bringing the bucket with input video data to the decoder, we then have in psuedo code this situation:
while {
dataToDecode = data.nextPacket();
codecStatus = codec.decode(dataToDecode);
if (codecStatus == WantMoreData) {
continue;
} else if (codecStatus == newOutputAvailable) {
picture = codec.picture();
picture.time = adjustAccordingToUserSettings(picture.time);
waitUntilAppropriate(picture.time);
display(picture);
}
}Remember that we don't see the different stages of the data as it is passed through the hardware, or differently worded; all we do is feed the codec with new input data. But as we want to make use of xbmc's existing code to adjust the timestamp, this means that the input buffers need to have their timestamps modified before they are sent to the video codec. XBMC adjusts the timestamp of "Picture" objects, which it thinks are decoded video frames ready to be forwarded to the "VideoRenderer".
No biggie, hack 1; I introduced a new "SMD" video data format and we can pretend that this still encoded data is just a "SMD" specific rendering format.
But to send this "picture" to the renderer, we need to report that a picture is available, and as xbmc thinks it needs to wait for vsync for each "picture" reported available this will introduce pipeline bubbles and stall the pipeline. To avoid these bubbles, we want to feed the codec data as fast as it can accept, and drivers usually help a little bit by allowing us to give it more data than it can handle so that there's a queue of data waiting to be submitted.
Introduce hack 2; we intentionally report "wantmoredata" from the codec in which case xbmc does not try to display anything, but it goes back into reading the next data packet available to feed the codec as soon as possible.
This gives us pretty good use of the video hardware so that it is able to decode videos of these high resolution and bitrates it is supposed to be able to decode, and we get to make use of xbmc's existing av-sync mechanisms.
Unfortunate side effect: when video is playing, the ui (not the video, but buttons, lists, subtitles, etc) will render slowly as we only report "picture available" for a fraction of the frames so it is rendering the ui elements just a few times per second.
We also need to ensure that the HW clock is in sync with XBMC's clock, which is done in some code referenced in that post I referenced earlier.
So that would be a brain dump of that, hope it made sense :)
KeyserSoze1
cadesalaberry
I prefer to be summoned via @quarnster rather than a PM in the forums. You can ask in this issue or open up a new one if there are followups or any other questions about something else.
@KeyserSoze1 wrote: