livepeer / lpms

Livepeer media server
MIT License
282 stars 71 forks source link

Improve VFR support #393

Closed j0sh closed 3 months ago

j0sh commented 4 months ago

Improve VFR support ( 55a5d805d32927d3e6433a64fa1ad343b1370e61 )

Manually calculate the duration of each frame and set the PTS to that before submitting to the filtergraph.

This allows us to better support variable frame rates, and is also better aligned with how ffmpeg does it.

This may change the number of frames output by the FPS filter by +/- 1 frame. These aren't issues in themselves but breaks a lot of test cases which will need to be updated.

Update testcases for VFR ( 1986cd4f47416c8ce2af09f893e902efe3ac8898 )

Notes

Previously we were depending on a calculated frame rate (r_frame_rate) but this isn't set to a sensible value for VFR streams. Packet duration also can't be relied on (see the new VFR test case where each duration is N/A), so we need to do some manual bookkeeping to calculate the timestamp difference between frames ourselves.

I haven't yet fully understood what happens with the FPS filter during the transition between segments but the behavior seems okay and the timestamps don't change too much, aside from the +/- 1 frame difference.

Most unit tests pass on my machine, except I did not test GPU or non-H264 codecs. I don't think things would be much different there, but someone with a GPU should check. The tests in ffmpeg/sign_test.go all timed out for me and there is a segmenter test that seems flaky, but these are probably unrelated.

codecov[bot] commented 4 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 50.20216%. Comparing base (684658a) to head (1986cd4). Report is 4 commits behind head on master.

Additional details and impacted files [![Impacted file tree graph](https://app.codecov.io/gh/livepeer/lpms/pull/393/graphs/tree.svg?width=650&height=150&src=pr&token=sa1XWzLYHR&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=livepeer)](https://app.codecov.io/gh/livepeer/lpms/pull/393?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=livepeer) ```diff @@ Coverage Diff @@ ## master #393 +/- ## =================================================== + Coverage 49.88901% 50.20216% +0.31315% =================================================== Files 12 12 Lines 1802 1484 -318 =================================================== - Hits 899 745 -154 + Misses 852 689 -163 + Partials 51 50 -1 ``` [see 12 files with indirect coverage changes](https://app.codecov.io/gh/livepeer/lpms/pull/393/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=livepeer) ------ [Continue to review full report in Codecov by Sentry](https://app.codecov.io/gh/livepeer/lpms/pull/393?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=livepeer). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=livepeer) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://app.codecov.io/gh/livepeer/lpms/pull/393?dropdown=coverage&src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=livepeer). Last update [ced71c4...1986cd4](https://app.codecov.io/gh/livepeer/lpms/pull/393?dropdown=coverage&src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=livepeer). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=livepeer). [see 12 files with indirect coverage changes](https://app.codecov.io/gh/livepeer/lpms/pull/393/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=livepeer)
j0sh commented 4 months ago

Another quick note that I forgot to mention. I am not sure how this fix would be rolled-out operationally, but because this will produce different frame counts (and thus different timings), it is important that streams be "pinned" to the same version of LPMS if some transcoders are slow to update themselves. Otherwise there might be some breakage in between segment intervals, eg timestamp overlaps or gaps.

leszko commented 4 months ago

Another quick note that I forgot to mention. I am not sure how this fix would be rolled-out operationally, but because this will produce different frame counts (and thus different timings), it is important that streams be "pinned" to the same version of LPMS if some transcoders are slow to update themselves. Otherwise there might be some breakage in between segment intervals, eg timestamp overlaps or gaps.

Good point, thanks for mentioning it!