search

mifi / lossless-cut

The swiss army knife of lossless video/audio editing
https://losslesscut.app/
GNU General Public License v2.0
28.27k stars 1.37k forks source link

😍 Implement "smart cut" #126

Open mifi opened 5 years ago

mifi commented 5 years ago

We cannot cut between keyframes, it's a technical limitation of modern video codecs. So ffmpeg will choose the nearest keyframe, which will not be the exact time you requested.

Might be able to use avcut for h264 although it is experimental. See https://github.com/mifi/lossless-cut/issues/372#issuecomment-761844667

See also discussion in #13

Update: Have made an initial implementation of this. This experimental feature will re-encode the part of the video from the cutpoint until the next keyframe in order to attempt to make a 100% accurate cut while losslessly copying the rest of the segment.

VideoRedo

VideoRedo is a now-defunct software that had feature overlap with LosslessCut.

Smart Media Cutter

Recently built in order to replace VideoRedo

They open sourced their smart cutting algorithm, could take inspiration from it: https://github.com/skeskinen/smartcut

Remaining issues

Inaccurate seeking

Seeking is not always accurate, causing glitches and/or jumps in the stitch points. This is often because I haven't found a way to consistently seek to and cut on an exact frame. Sometimes it could be caused by the "audio samples" problem below. If you want to experiment, you can try to remove the audio track and cut only video to see if it makes a difference.

Audio samples problem

Cut time accuracy/glitches when audio samples are large? (try without audio and see if it makes a difference). A workaround seems to be to cut audio/video separately and then merge them, with the tradeoff of potential audio/video de-sync:

subtitles

Other improvements

Done

Limitations

ilaw138 commented 5 years ago

Hi guys,

Interesting that it is a limitation of the codecs. Reason I say this is that I can do a frame specific "lossless" cut using Apple's (very old) Quicktime Player 7 in Pro mode. For the cut to take at the specific frame it has to export to a new file (rather than edit the existing file). But you can do this just passing through the source data with out any form of recoding (both video and audio tracks).

Not sure exactly how it does it, but seems that som,ehting could be possible. Save time isn't too bad either as it just seems to shuttle the data into a new file.

Happy to send you a before and after file of a minute or so duration showing what it can do, if you'd like that.

This is about the only thing that I would need to replace the ancient Quicktime Player 7 with lossless-cut as my mainstay tool. Other enhancemnets wouldbe nice, but this one is pretty essential for frame level edits.

Hope this helps.

Cheers,

Ian

mifi commented 5 years ago

Have you tried quicktime pro with a large file? Like 1gb, does it still take only a couple of seconds? If it's quick then I'm thinking maybe they do a smart cut, re-encoding only the inter-keyframe portion

DGrv commented 5 years ago

This would be awesome really :)

CoeNT commented 5 years ago

I more or less grew up with SvenOver's DVBcut which did implement "keyhole surgery" editing. He never updated it for HD (H.264 and etc) and, in the end, I shifted to VideoReDo which handles top-and-tailing and advert removal very efficiently and reliably using "smart cut" techniques and FFmpeg as the internal engine. Alas, it's Windows-only, not open-source and non-free (but relatively cheap) but it's a good benchmark to follow. So ... I'm absolutely enthralled if you're planning to add the facility to your LosslessCut ... all power to your elbow!

In the meantime though, I wonder if you might consider the following: There has been some discussion here already on "cut at the natural frame-break before or after the desired cut-point " (so apologies if the matter has already been settled). How about a pair of extra buttons adjacent to the < and > buttons that skip to the natural break respectively before or after? That way the user can choose (to taste) whether to exclude the last (intrusive) advert frame or include the first (treasured) movie frame. At the moment, it's not obvious which sacrifice one will make :-).

I'm enjoying getting to know your creation - it's a valuable addition to the linux toolkit. Thank you!

David

ghost commented 5 years ago

I'd like to point out that if you use -ss to cut at a keyframe after the input, the resulting video doesn't seem to play the first segment correctly. Setting it to a tenth of a second before the keyframe does work, but it does go to show that using -ss before the input is superior.

It's a quirk to keep in mind when implementing smart cut.

pkamb commented 5 years ago

So ffmpeg will choose the nearest keyframe

Does this ever lose frames? If there are key frames at frames 10 and 15, and your trim starts at frame 13, does it take the *nearest keyframe?

Do you lose the first 2 frames of video? Or does it look for the first keyframe before/after your in/out points?

danielmee commented 4 years ago

Seeing as @ilaw138 didn't reply, I can. I've been using QTP7 and MPEG Streamclip for the past 10 years to manage the media files of a weekly show at work. It's possible to do a perfect cut on any size file, here are the steps:

mifi commented 4 years ago

Hi. Thanks for the info. I'm not really sure how QuickTime Pro does this, and I don't really understand why you need to use Streamclip in addition to quicktime pro?

ilaw138 commented 4 years ago

Apologies. Missed your e-mail about large files.

In short, no I haven’t tried to edit out segments frame exact on a large file of 1GB.

QTP7 didn’t really like large files in any event - at least in my experience - so never tried anything too challenging for the software. Have found if I save file from QTP7 (as opposed to export) it didn’t cut the frames exactly - so assume using key-frame - but export does.

Sorry I can’t be of more help.

Regards,

Ian

On 14 Feb 2020, at 12:55, Daniel Mee notifications@github.com wrote:

Seeing as @ilaw138 https://github.com/ilaw138 didn't reply, I can. I've been using QTP7 and MPEG Streamclip for the past 10 years to manage the media files of a weekly show at work. It's possible to do a perfect cut on any size file, here are the steps:

Trim to the exact position you want using the in/out markers in QTP7 Save it as a separate file (you can use QTP7's reference file option though so it will save a new file however fast your computer can write to disk and the size of the file) Open the file in MPEG Streamclip and select File > Save As In the save dialogue box change the dropdown to be mp4 and Save. Done, no reencoding, exact to the frame and plays as expected. Feel free to ask me to provide you anything you need that might help you to solve this issue. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mifi/lossless-cut/issues/126?email_source=notifications&email_token=ADIUWJYFHDTB5JX5QPTQ2MDRC2IFLA5CNFSM4GWMKI2KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELY54EY#issuecomment-586276371, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADIUWJ2G4QRDHZRPB6U6CJLRC2IFLANCNFSM4GWMKI2A.

danielmee commented 4 years ago

... and I don't really understand why you need to use Streamclip in addition to quicktime pro?

Quicktime doesn't let you save a clipped file as a .mp4, only a .mov, Streamclip allowed me to get it back to .mp4 without recompressing. The only other way to get it out of QTP7 as a .mp4 was to export it but then it would compress it to those export settings.

danielmee commented 4 years ago

QTP7 didn’t really like large files in any event - at least in my experience Have found if I save file from QTP7 (as opposed to export) it didn’t cut the frames exactly - so assume using key-frame - but export does.

Not meaning to contradict you Ian but my experience working with QTP7 for 10 years pretty much every day was totally different. It regularly handled the 5, 10 or even 50G files I threw at it and its accuracy was spot on to the frame each time. If it wasn't I wouldn't have been able to use it as the transitions between segments in the program had to be edited out to the frame and QTP7 worked flawlessly each time. I also used the info panels to find out the duration to the millisecond (see the screenshot of a video's details, note the Current Time)

Screen Shot 2020-02-15 at 2 24 29 am

If it's helpful I can do a screencast of myself using it and how accurate it was... if it's going to help.

ilaw138 commented 4 years ago

Hi Daniel,

I am sure that we all have different experiences of the same software, I too have been using QTP7 for many years. I use export and set the compress options for audio and video to pass through, so no change to compression, but as I recall (I haven't used them for some time since I started to use Handbrake to compress my file before editing) was that large files and export was a hassle and quite slow to complete.

Yes you can save them as you mentioned, but I found that the saved output was not always "clean" in that there was sometimes a residual frame or two that I had cut out in the output file. From memory reference files did the same thing, but I didn't use StreamClip and may be that "cleans" up the file to the exact cut point.

So certainly for me only using QTP7 I found large files more cumbersome to export than already compressed files and Handbrake handles files of all sizes (again my experience) elegantly.

So Mikael, thinking about things I now presume that QTP7 when exporting did recode the entire file and set new key frames based upon the content of the edited file (sorry not that literate in exactly how things work, so am guessing here). As I used relatively small files - normally < 1GB it didn't take long, even on my MacBook Air (Core 2 Duo!)

Hope this helps the conversation. As I said originally with 32 bit now history from Apples perspective, it would be great if a solution could be found using Lossless Cut. As this is a great Application and handles files very well.

Cheers,

Ian

On 14 February 2020 at 15:56, Daniel Mee notifications@github.com wrote:

QTP7 didn’t really like large files in any event - at least in my experience Have found if I save file from QTP7 (as opposed to export) it didn’t cut the frames exactly - so assume using key-frame - but export does. Not meaning to contradict you Ian but my experience working with QTP7 for 10 years pretty much every day was totally different. It regularly handled the 5, 10 or even 50G files I threw at it and its accuracy was spot on to the frame each time. If it wasn't I wouldn't have been able to use it as the transitions between segments in the program had to be edited out to the frame and QTP7 worked flawlessly each time. I also used the info panels to find out the duration to the millisecond (see the screenshot of a video's details, note the Current Time)

If it's helpful I can do a screencast of myself using it and how accurate it was... if it's going to help. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

mifi commented 4 years ago

By the way I think normal quicktime that comes with mac os (not pro) can aslo trim losslessly. I just tried a relatively big file and it was instant and accurate, so im thinking it must be lossless. However it only works on mov/mp4

geimist commented 4 years ago

In my opinion this feature would be a key feature for lossless-cut. Maybe 'avcut' would be a good solution. I have been using it for some time now for smart rendering in my project.

Here you can find the repository, and here is a blog about it.

If you like, you can test a static build (Linux 64Bit) of mine.

Even if only a few frames (before and after the keyframes) have to be rendered, this editing variant still needs much more CPU resources. If one could build avcut/ffmpeg in combination with hardware acceleration (vaapi / qsv), it would probably almost eliminate this disadvantage.

mifi commented 4 years ago

I tried starting to implement a keyframe accurate cutting with re-rendering the part outside the keyframes, similar to avcut, but I was not able to make it work consistently. When I cut exactly on keyframe timestamp, sometimes the beginning of the output file gets corrupted, sometimes not. depending on the video file I test on.

I managed to figure out how to render key frames effectively on the timeline however, so in the newest version you will have keyframes show up on the timeline, and you can zoom in to seek closely. Also now seeks should align with frames based on frame rate.

Shakil-Shahadat commented 3 years ago

This issue along with #372 should be a pinned issue.

mattack1 commented 3 years ago

Sorry I didn't read the whole thread. But can't the out point cut be precise, as long as it's NOT on an i-frame?

The next frame (beginning of the next segment) would be an I-frame?

FurkanGozukara commented 3 years ago

I just used handbrake it is precisely cutting but does re-encoding as well. Maybe you can implement like them?

mifi commented 3 years ago

@mattack I think for most files, the out cut is indeed precise.

pldavid2 commented 3 years ago

This seems like a very interesting feature and upon checking I found this answer on StackOverflow https://stackoverflow.com/a/63604858

Basically the idea when having a start frame not on a keyframe seems to be -doing a (minimal) reencoding between the exact start frame and the first keyframe -copy without reencoding between the first keyframe and the end frame. -concatenate both segments

I tried myself and seems to be working. Do you see any problem with this procedure, for example with audio?

By the way, your program is awesome @mifi

mifi commented 3 years ago

@pldavid2 That's how I'm thinking it would work also. The biggest problems I think are:

  1. need to find out where to set the start cut point for the lossless cut. Sometimes when cutting at or near a keyframe, ffmpeg will instead cut at the frame before that, depending on the file.
  2. I believe the re-encoded part needs to have the exact same parameters like encoding profile, resolution, color space, FPS, bit depth, bitrate, and possibly more parameters. And this all differs from codec to codec.
  3. Audio, subtitles etc needs to be cut separately and we would probably need to do the same with some audio codecs? (e.g. re-encode some part of it with exact same parameters)
  4. Then need to re-merge the separately cut audio/video/other tracks and hopefully everything will stay in sync.

We could add a smart cut like this as an experimental option for selected codecs like h264, h265, aac, mp3, but there is still a lot of work to implement that unfortunately.

wkunz439 commented 3 years ago

For cutting commercials from a movie, there's still no better tool than QTP7. I'm keeping an old iMac running just for this purpose.

I have found that the only usable option to save the video is Save. The Export option, even with passthrough set for video and audio, removes all extra audio tracks, e.g. additional language tracks.

smurfix commented 3 years ago

You might be able to steal some code to do this from https://github.com/ozmartian/vidcutter.

wkunz439 commented 3 years ago

Hard to tell. My latest Mac is running Big Sur. The older Mac has macOS Sierra; double-clicking the VidCutter on this machine doesn't do anything at all. The version is 6.0.5.1, created June 19, 2021, size 304 MB.

Thanks anyway.

Willy

Am 03.07.2021 um 11:54 schrieb Matthias Urlichs @.***>:

You might be able to steal some code to do this from https://github.com/ozmartian/vidcutter https://github.com/ozmartian/vidcutter.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/mifi/lossless-cut/issues/126#issuecomment-873380199, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUARAA2EJBSPUEBYSATCONLTV3M3ZANCNFSM4GWMKI2A.

NickDoom-IDKFA commented 3 years ago

It's probably the most wanted feature for my current needs. Thanks a lot! Can it also work for merging videos? Sometimes I have one of two videos in a different format — and I want to convert only those videos, leaving the main flow intact (and lossless). It's technically the same as "re-encoding the part between the cutpoint and next keyframe" (as long as those videos have the same resolution and frame ratio and only codec is the difference).

strawhatgami commented 2 years ago

Looslesscut is a very handy tool i like much. However my current need is accurate cuts.

So I make a piece of code that allows me to have smart cuts, but by continuing using Losslesscut: https://github.com/strawhatgami/smart-cut-llc

eleius commented 2 years ago

I wish lossy mode was available in addition to normal & keyframe cut modes. Compared to other tools, losslesscut is lightweight and easy to use, but I often end up reencoding with something like ffmpeg -i input.mp4 -ss 00:00:01.600 -preset slow -b:v 5000k to remove extra (milli)seconds from the resulting file.

mifi commented 2 years ago

@strawhatgami cool that you made a script that solves your problem! If I understand correctly, your script will re-encode the whole video. AFAIK that is actually the same as the lossy cut mode that I plan to implement in the not-too-far future, see #372

Droyk commented 2 years ago

@mifi Why can't the LLC reconstruct the frame where you want to cut from the most recent key frame and all intermediate partial frames? Isn't that exactly what every video player does when you play a video?

mifi commented 2 years ago

@Droyk yes, but losslesscut doesn't operate on frames, it just operates on timestamps, and the player is doing that internally. AFAIK ffmpeg doesn't have any built in way to reconstruct (re-encode) only the part until the key frame, so losslesscut has to implement that, using exact same codec parameters as the codec in question.

mifi commented 2 years ago

Have now made an initial implementation of this.

This experimental feature will re-encode the part of the video from the cutpoint until the next keyframe in order to attempt to make a 100% accurate cut while losslessly copying the rest of the segment.

Note:

Inspired by smart-video-cutter.sh fronm https://github.com/mifi/lossless-cut/issues/330#issuecomment-1052324451

Thanks @fernandoherreradelasheras !

HarryDymond commented 2 years ago

Hi there @mifi , I am very pleased to have found this tool and commend you on your efforts to implement a method of cutting a stream at a precise location. However, I would like to re-iterate earlier comments that QuickTime 7 Pro is able to edit at any frame, and then save without doing any re-encoding (to the best of my knowledge; certainly the save process seems too fast for there to be re-encoding happening).

I understand the very basics of video codecs but don't really know much about the finer points. Let's say there's a keyframe at frames 45, 55 and 65, and the user cuts out frames 50 to 60 inclusive. Is it possible that when QuickTime does the edit, the saved file ditches the intermediate frames from 50 to 54, but retains the keyframe at frame 55, and then puts something in the stream that instructs the player to skip from frame 49 to 61 when playing back?

I know this offer was made earlier, but it wasn't responded to directly, so I would like to ask again: would it be instructive for you to inspect a video file that has been edited using QuickTime 7? If you provide a source file, I would be happy to edit it at specific points as specified by you and then send back for you to inspect. I feel strongly that it would be a real coup to figure out exactly how QuickTime 7 does its trick!

EDIT: seems like it probably is as I described above, using "edit lists/atoms": https://forum.videolan.org/viewtopic.php?t=115190 https://developer.apple.com/library/archive/documentation/QuickTime/QTFF/QTFFChap2/qtff2.html#//apple_ref/doc/uid/TP40000939-CH204-BBCCFBEF

EDIT 2: Scott Lamb, who appears to be active on GitHub, claims to use Edit Lists in a project of his: https://news.ycombinator.com/item?id=16889403 Perhaps you could get in touch?

madwyn commented 2 years ago

Have now made an initial implementation of this.

This experimental feature will re-encode the part of the video from the cutpoint until the next keyframe in order to attempt to make a 100% accurate cut while losslessly copying the rest of the segment.

Note:

  • Only works on some files:

    • I've had success with some h264 files, and only a few h265 files seem to work.
    • Other codecs may or may not work.
  • Currently only works on Linux/Windows. I'm working on Mac support (MacOS H264/H265 hardware encoder (h264_videotoolbox) seems to produce corrupted files when merged with the original, need to use x264, x265)
  • Only supports cutting a single video stream. All other video streams have to be disabled (in the tracks panel). You can still probably do multi-passes and then recreate a file with all the streams.
  • Only scans up to 60 seconds to find a keyframe.
  • It tries to auto-detect bitrate from input stream, however if it fails, then it will calculate bitrate from the whole file's size/duration (may be wrong)
  • Tries to copy timebase from source to destination

Inspired by smart-video-cutter.sh fronm #330 (comment)

Thanks @fernandoherreradelasheras !

I tested it on macOS with a full HD recording from Zoom, it worked flawlessly. The encoding took less than a minutes, since it's a screen sharing recording, there weren't many key frames, using smart cut is a must.

mifi commented 2 years ago

@HarryDymond that's very interesting research. I'm also guessing that's how Apple does it. Although I assume it only works for MOV files, it would indeed be an interesting alternative smart cut implementation. The problem is that I also have the same basic understanding of codecs as you have, so I don't know how to implement it. I'm not going to implement a MP4 writer in C/C++ so we would need a MOV atom editor software library or tool that supports Edit List Atoms, and I don't know of any library that can do that.

I'll try to mention @scottlamb here, and see if he'd be interested in sharing how he implemented writing Edit List Atoms to mov files.

scottlamb commented 2 years ago

I'll try to mention @scottlamb here, and see if he'd be interested in sharing how he implemented writing Edit List Atoms to mov files.

Sure. Yeah, my software uses edit lists for something similar.

Let's say there's a keyframe at frames 45, 55 and 65, and the user cuts out frames 50 to 60 inclusive.

I'll assume simplicity for the moment: key frame = IDR frame, and each non-IDR frame is a P frame using the previous frame as a reference, forming a chain, so reconstructing a frame needs everything up to and including the last key frame. There are other scenarios possible though. B frames (which I think are common in Blurays or the like but never used by my security cameras). Some of my IP security cameras have a "SVC" mode which makes half of the frames non-reference frames. Some support more intricate patterns of reference frame use.

Here's how the .mp4 could be written in this scenario: omit frames 50–54, write frames 55 onward, and have an edit list that makes frames 55–60 not get displayed (by skipping over their time range). No encoding necessary. Frames 55–60 are still in the file and can be used as references by frames 61–64 even though they're not displayed. Now, this means that someone can later remove the edit list and see those frames, so this approach is only valid if that's acceptable privacy-wise.

My software isn't using ffmpeg, so I don't know ffmpeg's options to achieve this.

mifi commented 2 years ago

Thanks for your detailed explanation Scott, that's very helpful. Do you use an open source library or tool to write the edit lists to the .mp4, or some proprietary code or tool?

scottlamb commented 2 years ago

It's my own open source code. It's specialized to what I'm doing (serving byte range requests for my HTTP API from my database schema, trimming away just the start and end of a "segment" as defined in my glossary) rather than a general-purpose library.

https://github.com/scottlamb/moonfire-nvr/blob/841e06e35458db1e40e16c72730cd2959cfdc5e6/server/src/mp4.rs#L1372

JokerQyou commented 2 years ago

I've tried the smart cut mode and there's a question. I saw the commandline llc invoked was like this:

/usr/share/losslesscut/resources/ffmpeg \
    -hide_banner \
    -i /home/xx/Videos/test/input.mp4 \
    -ss 2255.48540 \
    -t 2.01427 \
    -map 0:0 \
    -c:0 h264 \
    -b:0 2627472 \
    -map 0:1 \
    -c:1 copy \
    -ignore_unknown \
    -video_track_timescale 16000 \
    -f mp4 \
    -y \
    /home/xx/Videos/test/input-smartcut-segment-encode-0.mp4

This took more than 3 minutes to finish, despite there's only 2 seconds of content to re-encode. I've previously seen on this thread about the position of -ss and -to/-t argument affecting ffmpeg behavior, so I went ahead and manually ran this:

/usr/share/losslesscut/resources/ffmpeg \
    -hide_banner \
    -ss 2255.48540 \
    -t 2.01427 \
    -i /home/xx/Videos/test/input.mp4 \
    -map 0:0 \
    -c:0 h264 \
    -b:0 2627472 \
    -map 0:1 \
    -c:1 copy \
    -ignore_unknown \
    -video_track_timescale 16000 \
    -f mp4 \
    -y \
    /home/xx/Videos/test/1.mp4

which is exactly the same as the one llc invoked, except I moved the -ss and -t before -i. This command finished in less than 2 seconds. This resulted in a file with different size, but it has the same duration and I think it's literally the same as the one produced by llc's command. I've attached their corresponding mediainfo result.

I wonder if you could adjust the argument position to speed up smart cut?

llc-smartcut-mediainfo.txt custom-encode-mediainfo.txt

mifi commented 2 years ago

Thanks for researching. Can you see if the files are exactly the same? Are the frames the same around the cutpoint in the output?

I did add a comment in the code when I implemented it, that having -ss before -i will lead to issues: https://github.com/mifi/lossless-cut/blob/9ddcd85211dfedcc532ca00c1c7c1db40b665d98/src/ffmpeg.js#L826

...but I don't remember what kind of issues. So it's something to look more into.

JokerQyou commented 2 years ago

It's hard to tell. The file sizes are different, but according to qctools:

It's a livestream recording without much movement, so I can't tell much from the picture movement whatsoever, and the original file is too large to share. I'll try some other samples once I get the time.

mifi commented 2 years ago

I tested again with -ss before -i for the smart cut encoding, and I can see that output files get a black section in the beginning. However what seems to work is -ss -i input.mp4 -ss 0, for some strange reason. No black section at the beginning, and it's fast too. So I will try that for the next version of llc.

JokerQyou commented 2 years ago

I tried the latest version (v3.45.0) and it indeed is fast! Cutting the same amount of video range only took several seconds (instead of several minutes). The result is quite promising too - for h.264 files (I've not got any h.265 samples so can't test that). Thank you very much for the quick update!

Poikilos commented 2 years ago

So ffmpeg will choose the nearest keyframe

Does this ever lose frames? If there are key frames at frames 10 and 15, and your trim starts at frame 13, does it take the *nearest keyframe?

Do you lose the first 2 frames of video? Or does it look for the first keyframe before/after your in/out points?

I seem to only have reliable results in my program if I use a timecode that is in the "middle" of a frame's lifetime. In other words, to keep a frame, use its starting timecode plus 1/2 of the frame delay (1/60 of a second if video is 60fps). Here's some of my C# code: https://github.com/poikilos/RetroEngine-cs/blob/433773855ab8e792f9756ed0b39c80d4976a097a/RConvert.cs#L257 and below I've converted it to Python:

from __future__ import division
# ^ make sure only `//` (not `/`) can force floor division in Python 2
import math

def frame_to_ffmpeg_timecode(frame_number, fps):
    fps = float(fps)
    frame_remainder = float(frame_number)
    second_to_hour = 60.0 * 60.0
    second_to_minute = 60.0
    hour = int(frame_remainder / (fps * second_to_hour))
    frame_remainder -= float(hour) * (fps * second_to_hour)
    minute = int(frame_remainder / (fps * second_to_minute))
    frame_remainder -= float(minute) * (fps * second_to_minute)
    second = int(frame_remainder / fps)
    frame_remainder -= float(second) * fps
    sec_per_frame = 1.0 / fps
    millisecond = int(frame_remainder * (1000.0 / fps) + 1000.0 * sec_per_frame / 2.0)
    # ^ add sec_per_frame / 2.0 to get to the "middle" of the frame--so as not to undershoot!
    return "{}:{}:{}:{:0=3d}".format(hour, minute, second, millisecond)

If you know why my code wasn't frame accurate without this function, feel free to ignore it, but if you're having trouble with frame accuracy in FFMPEG I suggest you use this code :edit: or just millisecond = int(frame_remainder * (1000.0 / fps) + 1000 * sec_per_frame / 2.0) [code fixed June 20, 2022] if you can fit that into your code. I understand this is a keyframe issue, but using a timecode that is at the exact beginning of a frame (possibly messed up by float storage) seems to require this workaround in ffmpeg.

iopq commented 2 years ago

I have a doubling of the first few frames with smart cut, not with lossless methods. Anyone else have this issue?

I'm cutting a webm file with VP90 codec

mifi commented 2 years ago

@poikilos that is interesting. I think I experimented with something like this before, but I think I couldn't find any value to add that will work consistently for all files I was testing with. (sometimes +1/2 a frame duration, sometimes +3/4 a frame duration was needed). I wonder why counting frames multiplied by fps is beneficial. I would think it would be the same as using the time values given by ffprobe, and maybe even less accurate because then we would have to assume that fps is an accurate number and assume that the file doesn't use variable fps or custom frame PTS.

Poikilos commented 2 years ago

@mifi I had some trouble with other aspects of the coding back then but I've been getting back to making a rotoscoping program, so the point is to only store modified frames and a minimal cache for markers. Adding only a tiny fraction of a frame is what seemed consistent (I was doing SPF/2 instead of 1000*SPF/2), so float accuracy may be all I was overcoming. It seemed good for NTSC dropped frame (29.97 fps). Adding a tiny fraction was a bug in my code I fixed here, so less than half a frame may be better. The one liner could be replaced by adding an offset to a timecode. I'm not sure if the code will behave differently on start times vs end times but that's something to consider as well. Another factor I didnt know back then is that 29.97 and 23.98 and 59.94 aren't the true values. Dropped frame video actually uses a fraction. :edit: The best starting point seems to be to use ffmpeg's own formulas I just found: https://ffmpeg.org/doxygen/2.2/timecode_8c_source.html#l00084 but if using Python cast to int using JavaScript use Math.floor() to match where C stores a result as int.

mifi commented 2 years ago

Ok. I'd really like to get to the bottom of the accurate seeking issue, but I tried many times and I never found something that works consistently. At least I have one more thing to try next time

cjbarth commented 2 years ago

I doubt you'll make it very far. As much as I'd love for there to be a solution, until ffmpeg supports indexing by frame number, this may not be possible. See https://ffmpeg.org/ffmpeg.html#:~:text=-ss%20position%20(input%2Foutput)

By all means, prove me wrong :)

Poikilos commented 2 years ago

With lossless cutting the right timecode may not enough but still seems correct and may avoid issues. Another situation where it isn't enough is if -noaccurate_seek is put in with PR 13 so it seems like that argument should be optional. If all that isn't enough, it will at least be correct, and if there is some third factor (the second being 1/4 or whatever offset if that is still necessary after that, possibly no offset for the end) that could be explored.

mifi commented 2 years ago

@poikilos -noaccurate_seek only exists in that PR (not in losslesscut codebase), and that PR is not merged and probably never will be. It would be nice if someone who knows ffmpeg internals well could explain exactly how seeking works with regards to timestamps, frames and keyframes. But I'm afraid the reason that it's not documented anywhere is because it's not really consistently implemented across formats/codecs/params in ffmpeg so if someone were to document it, it would be wrong.

This stuff is not really related to smart cut though, so I created a new issue where we can discuss this core problem of how to actually seek in ffmpeg.