tobbee
commented
12 months ago After some iterations, I went the more difficult way and let the audio segment follow the video segments and even let the video segments vary in duration.
At some moment, similar task could be made for subtitles.
An asset where the total audio duration is not equal to video should be playable without drift.
E.g. two segments of "testpic_6s" has a video duration of 12s, but audio duration of 12.010666s since every AAC frame is 21.3333ms.
Just looping works well, but when playing out with dash.js, there is an accumulated drift of audio which lags behind video.
To fix, this livesim2 should consider dropping/repeating a frame at each loop, so that the audio stops at or within one audio frame (21.3333ms) after the video. In the "testpic_6s" case, the video and audio should be fully aligned after 4 segments (24s).
Another, more flexible solution, is to resegment the audio to follow the video. One way would be to calculate the time range that most closely maps to the corresponding video segment, and then extract and combine the corresponding audio samples from one or more segments. Given that we have the information about the time and duration of each VoD audio segment, this should be rather straight forward. With this approach we can also do some tricks like shorten/extend segments to simulate mismatch in the timeline and period boundaries. A caveat, is that we are assuming that we can start an audio segment at any sample/frame. This is true for the most common codecs, AAC, HE-AAC, and AC-3/EC-3, but not for xHE-AAC.
Add new test content and unit tests to make sure that this works. Also make sure that offsets still work.