search

Streampunk / beamcoder

Node.js native bindings to FFmpeg.
GNU General Public License v3.0
397 stars 76 forks source link

PTS Mystery - How to reencode video properly? - With Gist #36

Open josiahbryan opened 4 years ago

josiahbryan commented 4 years ago

Gist

Gist showing the issue: https://gist.github.com/josiahbryan/2292b7b33860367755513a63f8228282

Steps:

  1. Download Gist https://gist.github.com/josiahbryan/2292b7b33860367755513a63f8228282
  2. Download http://dl5.webmfiles.org/big-buck-bunny_trailer.webm
  3. Run Gist
  4. Run ffplay reencode.mp4

What it does:

Reads the trailer .webm and sets up decoder/encoder to decode the file then reencode it as mp4. Runs for 200 frames then writes out the file.

Expected:

Expected result is that ffplay reencode.mp4 plays the file at normal speed (same as you would see if you did ffplay with the original .webm file

What actually happens:

ffplay plays the reencode.mp4 file at what seems like some incredibly small fraction of the original speed. Pressing right arrow to fast forward thru the file shows the frames are there, just not playing at the right speed

Help!

I suspect this has something to do with how the .pts is set on the outgoing packets, but no clue. I don't generate the pts, I just use the .pts on the frames. I based the encoding code on https://github.com/Streampunk/beamcoder/blob/master/examples/make_mp4.js

Any advice? Thanks for your help as always!

josiahbryan commented 4 years ago

Do you need any additional information from me on this? Sadly this is a bit of a show-stopper for me, and I would love any suggestions on fixing this :) Thanks!

scriptorian commented 4 years ago

The example you have started from doesn't have a source file so unfortunately you have missed handling the time_base from the source. Firstly the encoder time_base should be set from the source stream - inVideoStream.time_base rather than [1, 25]. The main change though is that the mux pts/dts adjustment should be: srcTB / dstTB or (srcTB[0] * dstTB[1]) / (srcTB[1] * dstTB[0]). For your example file the source time_base is [1, 1000] giving 90000 / 1000 rather than 90000 / 25. Making this change sorted out the play speed correctly.

josiahbryan commented 4 years ago

Wonderful! Sadly, this was due to a lack of examples covering re-encoding. But thank you so much for this!

My ancillary question would be - how could I do this if I wanted to mix two videos into one? The reason I ask is because your answer places the encoder / muxer params directly dependant on the source video stream. So, how could I handle it if I have more than one source? For example, if I want to splice two videos together which could potentially have different "time_base" values, etc?

I guess is what I'm getting at is this: Is it possible to specify the absolute time base (basically, like the original example in the repo did of [1,25]) and do some sort of "conversion" from whatever the source time base is to the encoder timebase? Much like we do with video frames that could be some other format so we use filters to convert from whatever the format is to whatever the encoder specifies?

On Sun, Feb 23, 2020 at 1:39 PM Simon Rogers notifications@github.com wrote:

The example you have started from doesn't have a source file so unfortunately you have missed handling the time_base from the source. Firstly the encoder time_base should be set from the source stream - inVideoStream.time_base rather than [1, 25]. The main change though is that the mux pts/dts adjustment should be: srcTB / dstTB or (srcTB[0] dstTB[1]) / (srcTB[1] dstTB[0]). For your example file the source time_base is [1, 1000] giving 90000 / 1000 rather than 90000 / 25. Making this change sorted out the play speed correctly.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Streampunk/beamcoder/issues/36?email_source=notifications&email_token=ABEZELBJQSVZLBA34UDWDVLREJN5ZA5CNFSM4KTOVWH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMVZRSI#issuecomment-590059721, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABEZELAV2GMEQUWHVV3EH6TREJN5ZANCNFSM4KTOVWHQ .

-- Josiah Bryan +972-050-794-0566 www.josiahbryan.com https://www.josiahbryan.com/?utm_source=sig josiahbryan@gmail.com

scriptorian commented 4 years ago

Sorry for the shortage of relevant examples. For something developed a bit further you could take a look at scratch/stream_mp4.js that relies on the beamstreams helpers. If you are mixing two videos with different time_base values but the same frame rate you would have to reset the mix output pts/dts values to progress according to your chosen time-base. There are a few different FFmpeg filters that can help with this.

josiahbryan commented 4 years ago

No worries on the examples - I completely understand how the focus is on getting the core right, the examples come later.

However, re progressing the output pts/dts according to the chosen timebase

Advice on either calculating the pts/dts myself or using ffmpeg filters to do that would be incredibly helpful and oh so lovely. Thank you!

On Sun, Feb 23, 2020 at 1:58 PM Simon Rogers notifications@github.com wrote:

Sorry for the shortage of relevant examples. For something developed a bit further you could take a look at scratch/stream_mp4.js that relies on the beamstreams helpers. If you are mixing two videos with different time_base values but the same frame rate you would have to reset the mix output pts/dts values to progress according to your chosen time-base. There are a few different FFmpeg filters that can help with this.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Streampunk/beamcoder/issues/36?email_source=notifications&email_token=ABEZELAGMPHCU7HOAYDD5CDREJQFVA5CNFSM4KTOVWH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMVZ4UI#issuecomment-590061137, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABEZELB4EZGVCHB77UOGJPDREJQFVANCNFSM4KTOVWHQ .

-- Josiah Bryan +972-050-794-0566 www.josiahbryan.com https://www.josiahbryan.com/?utm_source=sig josiahbryan@gmail.com

scriptorian commented 4 years ago

I just had a quick look at the filter documentation and it looks like setpts/asetpts should be able to do what you are looking for. Would need some experimentation! If you want to do it yourself then the maths isn't too bad. For your example file the time_base is [1, 1000] which means that the pts values advance each frame by the number of milliseconds since the last - 40 in this case. If the time_base is [1, 25] then the pts values should advance by 1 each frame. Does that make sense?

josiahbryan commented 4 years ago

Sadly @scriptorian that last comment went over my head lol - I'm really wanting to keep the time_bases agnostic of each other - I want to treat the time_bases (incoming) as a black box and just set one "master" time_base for the output.

Asside: Using one of the examples from the repo, the video encoder has a time_base of [1,25] but the muxer uses [1, 9000] - which is confusing the crud out of me lol! I just copied the example as it was into the gist, hence why it is the way it is, but I don't know which is right and what I should change. Should both time_base values be [1, 90000]?

Filter - setpts

More to the point: I've updated the gist to run all frames thru a beamcoder.filterer which basically executes setpts=PTS-STARTPTS which is the filterspec used in https://ffmpeg.org/ffmpeg-filters.html#setpts_002c-asetpts whenever setpts is used to filter video from diverse sources.

However, the setpts filter did NOTHING to resolve the issue sadly. (See the gist - I tried lots of variations - here is the direct link to that part of the file: https://gist.github.com/josiahbryan/2292b7b33860367755513a63f8228282#file-transcode-test-js-L70)

I was able to hackily fix the timing by forcing the .pts of each frame with a pts ++ - see https://gist.github.com/josiahbryan/2292b7b33860367755513a63f8228282#file-transcode-test-js-L106 - but obviously that is a hack and would probably NOT be good to do in prod, especially if I want to mix multiple sources together.

Any suggestions? I really am out of my depth here - Just trying to find a black box that Just Works(tm) so I can do something like:

Stupid Simple Switching

The desired outcome of course is to have the output "just work" with the video/audio in sync and in proper time when written to the output. Basically trying to make a simple video switcher, but this timing thing is really screwing me up lol.

I checked out the filter streams example you linked earlier as well, and that looks awesome! However, it won't work for this use case since I can't just "fire and forget" - I want to be able to switch between inputs arbitrarily as needed during the encoding process.

The end goal here is to do all this on live video - so live camera captures, but for testing / development purposes, files are obviously easier to use to test and reproduce results.

Thanks for your help so far, let me know what else you need from me here! :)

josiahbryan commented 4 years ago

Just revisiting this issue here, and I've tried a variety of things to get the PTS to be correct, even tried recreating the PTS math from FFMPEG, but to no avail. Have you been able to come up with any other suggestions by chance, based on the last comment?

dthompso99 commented 3 years ago

Fighting pretty close to the same issue... I think whats missing here is the emulation of the "-re" flag ffmpeg would use... https://stackoverflow.com/questions/27942168/set-re-flag-in-libavformat might be some direction

josiahbryan commented 3 years ago

@dthompso99 thanks! @scriptorian any way to emulate that flag '-re' that @dthompso99 mentioned?

Interesting that you should mention this just today because I have a meeting on the use-case for this project tomorrow - good to see people are still live here haha

dthompso99 commented 3 years ago

I'm not sure how critical "Real Time" is for your application, but i think for this to work the workflow needs to be reverse what any typical libav workflow is (I've been trying to accomplish this in straight C as well)

When i say reverse.... your "Output" needs to drive the pts/dts, grabbing the appropriate frame from your "Input"... so far its a PITA, but i'd love some better ideas!

dthompso99 commented 3 years ago

also: didn't work for me, but run across this filter: https://ffmpeg.org/ffmpeg-filters.html#realtime_002c-arealtime