Open mifi opened 5 years ago
I'm also using smart cut with av1 and it's very slow. It should not take multiple minutes to encode (on a Ryzen 5700X) what adds up to a few seconds at most. Is there a way to manually change the encoder used in smart cut?
This is an issue I have brought up as well, Lossless cut uses whatever the default av1 implementation of ffmpeg is, this happens to be the original
aomav1
encoder and that encoders default settings are molasses slow, I've suggested usinglibsvtav1
in it's place because of the speed increade.See my previous comment here for some numbers.
Hopefully this can be implemented some time soon as it would increase the speed of these encodings by a good amount especially for much slower machines
Yeah, it would be nice if it allowed users to customize parameters. While this could be done via a GUI, that could get quite complicated, but even providing an option to show the command(s) that will be used and allowing the user to modify them would be nice. Granted, this would be a feature only for advanced users, but if you could just manually change part of the ffmpeg command to use whatever codec or settings you prefer, that would allow for full flexibility. And it could allow for automatic replacement of words in the commands, so you could set it to always replace e.g. codec_name_1 with codec_name_2.
Yeah, it would be nice if it allowed users to customize parameters. While this could be done via a GUI, that could get quite complicated, but even providing an option to show the command(s) that will be used and allowing the user to modify them would be nice. Granted, this would be a feature only for advanced users, but if you could just manually change part of the ffmpeg command to use whatever codec or settings you prefer, that would allow for full flexibility. And it could allow for automatic replacement of words in the commands, so you could set it to always replace e.g. codec_name_1 with codec_name_2.
the request to manually modify commands before running them has come up before, but i'm not sure how useful it is, because it involves a lot of manual work, if you have to modify a command line every time a cut is done, and imagine if cutting 100 segments, then you'd have to do it 100 times 😵 - see #750
aomav1
: 19m28s @ 98.4953VMAFaomav1
-usage realtime
: 5s @ 93.8847VMAFaomav1
-cpu-used 8
: 31s @ 98.1873VMAFlibsvtav1
(no custom arguments): 8s @ 96.0291VMAFWhen av1 is detected when using smart cut it should either apply
-cpu-used 8
to increase speed by 37 times or set the encoder to belibsvtav1
for an increase of 146 timesyou could use
-usage realtime
for an even faster encode but the quality is pretty large compared to the other two options
@BuyMyMojo Thanks for researching. The way I understand it from ffmpeg docs, if one wants to set -cpu-used 8
, one also has to set -usage realtime
-usage realtime
activates the realtime mode, ...-cpu-used
values between 7-10 are only available in the realtime mode
it also says:
-cpu-used
sets how efficient the compression will be. Default is 1. Lower values mean slower encoding with better quality, and vice-versa. Valid values are from 0 to 8 inclusive.
So I would think that setting it to 8 would yield the lowest quality? Is that quality even acceptable?
for libstav1 I would have to update the ffmpeg build script, probably with this: https://github.com/markus-perl/ffmpeg-build-script/blob/cef7f703dcccf46e7c061a0462397c31a535717e/build-ffmpeg#L489C11-L489C17
But how does it compare to the other codecs rav1e or aomedia av1 codec?
if one wants to set -cpu-used 8, one also has to set -usage realtime
That is only for libaom which I am still suggesting to move away from when detected as libsvtav1 doesn't require a separate usage flag
Is that quality even acceptable?
more than acceptable, it is still av1 so it still has the many benefits of that codec. I have seen more than decent encoding at usage 10 with svtav1 (which is the default).
in my original post I mention svtav1 getting a VMAF score of 96.0291 which can very basically be seen as a % of quality compared to the original. so svtav1 at it's default preset of 10 can reach a really good quality.
so compared to aom's encoder it is a tiny bit lower quality but only by a percent or two but with a massive speed boost. rav1e is slow by default but can also get a decent speed improvement but from my experience is more annoying to get to encode, needing more flags to encode. overall they are very similar quality but I would probably personally prefer svtav1's speed and quality combo for the smaller smart cut sections
next nightly build will use libstvav1 for encoding as well as dav1d for decoding (ffmpeg built in decoder seems to be very broken).
Also next nightly build will have a new Export Option called "Shift all start times", it can be used to automatically shift all segment start times forward by one or more frames before cutting (up to 10 frames). This can be useful if the output video starts from the wrong (preceding) keyframe. For Smart Cut, cutting from the wrong keyframe would sometimes cause the part around the cutpoint to be repeated (duplicated). When "shift start times" is set to +1 or +2 frames, this seems to resolve the issue for smart cut too for some files.
Hello! Foremost, I want to thank you for this amazing program. It really makes using Ffmpeg so easy. I also need a good smart cut feature, but in lossless cut it usually doesn't produce good results when I use it. In my search for an alternative with a smart cut feature that works, I found this: Using ffmpeg to cut videos with more precision than key frames allow Is this something that might work? I also found two programs that have a working smart cut. They are SolveigMM Video Splitter and TMPGEnc MPEG Smart Renderer Unfortunately, they are closed source, very expensive and have other problems or shortcomings that lossless cut does not have and not as easy to use.
AFAICT that answer is basically saying to do smart encoding, which is what LC attempts to do, but it's a work in progress.
SolveigMM is trash and the developer is useless.
TMPGEnc seemed promising, until I realized it doesn't support all audio codecs, so the video I was testing it with, which is one of many I'd like to edit, couldn't be done without resulting in an output file with no audio. Pretty useless, and ridiculous for such expensive software.
I'm pretty sure I've tried every software available, and few have this feature, and none do it well/properly. Unfortunately, VideoReDo, which was the gold standard and may have worked, is no longer available, so LC seems, to me, to be the most promising option.
I have used this program for a few years to make rough edits of videos, but I wanted to be precise when editing my DVD copies. People in this issue have talked about some of the odd behaviour of Smart Cut, e.g. inaccuracy by a few frames, repeating content or stuttering between connected segments. I have developed the following more involved approach for precisely editing videos with minimal loss in the latest FFmpeg. Perhaps it will help you and anyone else who stumbles upon this post. Hopefully someone can improve upon this.
This only works on MP4 containers, including when they contain multiple audio streams and multiple subtitles. I tested the approach with Matroska, which resulted in freezes between segments no matter what I tried. Perhaps the error is in FFmpeg, since Matroska is usually a robust format.
I have tested this on MP4 files containing H.264, AV1 and VP9 video, AC-3, DTS, AAC and Opus audio and VobSub, SubRip and WebVTT subtitles, the last two encoded using -c:s mov_text
.
Install MPV and build the latest FFmpeg. If you do not want to build FFmpeg: automated builds.
Open the video and note the minutes and seconds of the start and stop times of the content to be cut. If there are multiple cuts, note their timestamp pairs too.
Use FFmpeg's segment option to split the video at the nearest keyframes: Firstly, move each timestamp 10 seconds back if it comes before a cut, or 10 seconds ahead if it comes after. This ensures that you do not select keyframes inside the cut content. 10 seconds are used because keyframes are often up to 10 seconds apart, but you can find a shorter value with a script at the bottom of this post.
As an example: I have noted the timestamps 6:04 and 9:13 in a video. The first timestamp is moved 10 seconds back, the second 10 seconds ahead, giving us 5:54 and 9:23. Then I run this command:
ffmpeg -i INPUT.mp4 -map 0 -map_metadata 0 -movflags use_metadata_tags -c copy -f segment -reset_timestamps 1 -segment_times 5:54,9:23 INPUT_%02d.mp4
-map 0
selects all streams, -map_metadata 0 -movflags use_metadata_tags
preserves all metadata, -c copy
copies the streams instead of encoding them, -f segment
segments the video, -reset_timestamps 1
ensures that each resulting segment starts at 0:00 and -segment_times
indicates the places to split the video. If I had more pairs of timestamps, I would add them with additional commas to the segment_times
argument, e.g. -segment_times 5:54,9:23,11:00,11:40
.
Rename the segments which contain the cut content, e.g. by adding "_cut". In my case, I got three files: INPUT_00, INPUT_01, INPUT_02, where INPUT_01 contains the cut content, so I rename it INPUT_01_cut.
Use FFprobe to find the profile, level and timebase of the original video: ffprobe -show_streams INPUT.mp4
. This gives us the following output, abbreviated:
Stream #0:0: Video: h264 (High), 1k tbn [STREAM] profile=High level=31 (means the same as 3.1) [/STREAM]
The profile and level are in the STREAM section, and "1k tbn" is the 1k timebase.
Open the cut segment in MPV, click the left-hand time label in the player to show milliseconds and find the timestamps of the last frame of the previous uncut content and the first frame of the next uncut content. This can be done by advancing one frame forward or backward using the ,. (comma and dot) shortcuts. In my case, I have opened INPUT_01_cut and noted the timestamps as A=8.32 and B=3:17.08.
Re-encode the segment in parts A and B where the cut content is discarded:
ffmpeg -i INPUT_01_cut.mp4 -map 0 -map_metadata 0 -movflags use_metadata_tags -c copy -c:v libx264 -preset veryslow -crf 18 -tune grain -profile:v high -level:v 3.1 -video_track_timescale 1k -shortest -to 8.32 INPUT_01_cut_a.mp4
ffmpeg -i INPUT_01_cut.mp4 -map 0 -map_metadata 0 -movflags use_metadata_tags -c copy -c:v libx264 -preset veryslow -crf 18 -tune grain -profile:v high -level:v 3.1 -video_track_timescale 1k -avoid_negative_ts make_zero -ss 3:17.08 INPUT_01_cut_b.mp4
-c:v libx264
encodes using x264, -preset veryslow
uses the slowest preset to ensure highest quality, -crf 18
uses CRF=18 which is visually transparent, -tune grain
uses x264's grain tuning which is best for live action film, -profile:v high -level:v 3.1 -video_track_timescale 1k
uses the profile, level and timebase noted earlier. If the original content is not H.264, then you would use a different encoder of course.
On the first line, -to 8.32
encodes up to exactly 8.32 seconds, and -shortest
trims the output to prevent frozen images. On the second line, -ss 3:17.08
starts encoding from exactly 3:17.08 onwards, and -avoid_negative_ts make_zero
ensures that the resulting video starts at 0:00.
If the segment contains not one whole sequence but small pieces of content to be cut, then you will have to encode it into small pieces, e.g. INPUT_01_cut_a/b/c/d. Naturally you will have to use -ss
and -to
at the same time for the middle parts.
If you have multiple cut segments (INPUT_03_cut, INPUT_05_cut etc.), then repeat steps 6 and 7 for those as well. That is, create INPUT_03_cut_a/b, INPUT_05_cut_a/b and so on.
file 'INPUT_00.mp4' file 'INPUT_01_cut_a.mp4' file 'INPUT_01_cut_b.mp4' file 'INPUT_02.mp4'
Note that INPUT_01_cut is not included, only the re-encoded segments INPUT_01cuta and INPUT_01cutb. If you have more segments, e.g. 03 and 04, then of course you should add their parts too, for example:
file 'INPUT_00.mp4' file 'INPUT_01_cut_a.mp4' file 'INPUT_01_cut_b.mp4' file 'INPUT_02.mp4' file 'INPUT_03_cut_a.mp4' file 'INPUT_03_cut_b.mp4' file 'INPUT_04.mp4'
In an alternative example where only the beginning and end of the video are cut, it would look like this:
file 'INPUT_00_cut_b.mp4' file 'INPUT_01.mp4' file 'INPUT_02_cut_a.mp4'
ffmpeg -f concat -safe 0 -i concat.txt -map 0 -map_metadata 0 -movflags use_metadata_tags+faststart -c copy OUTPUT.mp4
-f concat
uses the concat demuxer, -safe 0
disables path safety checks, since it gets in the way for a lot of file names, and -movflags faststart
makes the MP4 file streaming-friendly and faster to play. The resulting file OUTPUT.mp4 contains all audio and subtitle streams with the cut content exactly removed with no visual or audible degradation.
Update: You can use this Linux shell script to find the average GOP (Group-Of-Pictures, i.e. duration between keyframes) of a video so that you only re-encode as much as necessary. For example, if the GOP is ~1 second, then you only need to shift the timestamps by 1 second instead of 10 seconds.
#!/bin/sh
# Sanity checks
if [ $# = 0 ]; then echo "No file provided."; return 1; fi
if [ ! -f "$1" ]; then echo "File does not exist."; return 1; fi
# Extract keyframe entries with FFprobe, then count number of keyframes
printf "Computing number of keyframes... "
ffprobe -v quiet -show_frames -show_entries frame=media_type,key_frame -of csv="p=0" "$1" > .temp.txt
if [ $? != 0 ]; then return 1; fi
num_keyfr=$(grep -o -i 'video,1' .temp.txt | wc -l)
if [ $? != 0 ]; then return 1; fi
echo $num_keyfr
# Extract duration with FFprobe
printf "Extracting duration... "
vid_dur=$(ffprobe -v quiet -show_entries format=duration -of csv="p=0" "$1")
if [ $? != 0 ]; then return 1; fi
echo $vid_dur
# Compute average GOP with awk decimal arithmetic
printf "Computing average GOP... "
awk 'BEGIN {print ('$vid_dur' / '$num_keyfr')}'
if [ $? != 0 ]; then return 1; fi
rm .temp.txt
Very interesting findings. So your research suggests that -f segment
will output perfect segments starting at exact keyframes without any glitches at the cut points (after concatenating), as opposed to using -ss
and -t
which as we know causes glitches? I wonder why that is the case. Maybe there's so many people and companies who rely on segment
working perfectly (for HTTP live streaming and other streaming functionality), that a lot of manpower has been put into implementing and improving that muxer in ffmpeg.
Anyways it's definitely something to try implementing for LosslessCut's smart cut. I wonder how well this segment
method works with other codecs like h265. I know that losslesscut smart cut works well for some h265 files.
You are probably correct that much effort has been put into perfecting the segmenter, at least for MP4 containers, since it is used to create M3U8 playlists for web video players, where uninterrupted playback is critical.
FFmpeg's segment option will pick the nearest keyframe, whether it is ahead or behind the provided timestamps; that is why the timestamps must be shifted by 10 seconds. Note that -ss
and -to
are only used in the re-encoding part. If any glitches occur as a result of re-encoding, then it is due to mismatching profile, level, timebase or some other codec-specific parameter; or as a result of using a container different from MP4; or as a result of including text-based subtitles. As mentioned, several little quirks like that can ruin it, either due to errors in FFmpeg or due to the inherent structure of the containers; I just found something that worked for a narrow MP4-based use case.
Note that MP4 can contain AV1, H.265, VP9 and Opus streams as well, but I have not tested an MP4 file containing such streams yet.
Here is a more advanced example. I apologise for double posting, but I do not have a blog or anything, so I will leave this here.
In this case, I will take #INTRODUCTIONS (2015) by LaBeouf, Rönkkö & Turner and cut it such that only 4 of the 36 acts remain.
00 CUT 5:49 01 Christian Wright: Samurai Sword 6:27 02 CUT 7:31 03 Hanqing Miao: Growing Up In Singapore 8:36 04 CUT 8:58 05 Joshua Parker: Just Do It 10:02 06 CUT 27:29 07 Lewis Tizley: Looking At My Phone & Computer 28:09 08 CUT
$ ./gop.sh INTRODUCTIONS.mp4 Computing number of keyframes... 1876 Extracting duration... 1860.245333 Computing average GOP... 0.991602
Since the GOP is ~1 second, I shift the timestamps -1 before a cut, +1 after.
00 CUT 5:50 (+1 sec) 01 Christian Wright: Samurai Sword 6:26 (-1 sec) 02 CUT 7:32 (+1 sec) 03 Hanqing Miao: Growing Up In Singapore 8:35 (-1 sec) 04 CUT 8:59 (+1 sec) 05 Joshua Parker: Just Do It 10:01 (-1 sec) 06 CUT 27:30 (+1 sec) 07 Lewis Tizley: Looking At My Phone & Computer 28:08 (-1 sec) 08 CUT
ffmpeg -i INTRODUCTIONS.mp4 -map 0 -map_metadata 0 -movflags use_metadata_tags -c copy -f segment -reset_timestamps 1 -segment_times 5:50,6:26,7:32,8:35,8:59,10:01,27:30,28:08 INPUT_%02d.mp4
$ ffprobe -show_streams INTRODUCTIONS.mp4 Stream #0:0(eng): Video: h264 (Main) (avc1 / 0x31637661), 24k tbn [STREAM] profile=Main level=32 [/STREAM]
00 CUT, B=5:49.807 5:50 01 Christian Wright: Samurai Sword 6:26 02 CUT, A=2.044, B=1:04.606 7:32 03 Hanqing Miao: Growing Up In Singapore 8:35 04 CUT, A=1.752, B=22.648 8:59 05 Joshua Parker: Just Do It 10:01 06 CUT, A=2.002, B=17:28.338 27:30 07 Lewis Tizley: Looking At My Phone & Computer 28:08 08 CUT, A=2.044
ffmpeg -i INPUT_00_cut.mp4 -map 0 -map_metadata 0 -movflags use_metadata_tags -c copy -c:v libx264 -preset veryslow -crf 18 -tune grain -profile:v main -level:v 3.2 -video_track_timescale 24k -avoid_negative_ts make_zero -ss 5:49.807 INPUT_00_cut_b.mp4 && \
ffmpeg -i INPUT_02_cut.mp4 -map 0 -map_metadata 0 -movflags use_metadata_tags -c copy -c:v libx264 -preset veryslow -crf 18 -tune grain -profile:v main -level:v 3.2 -video_track_timescale 24k -shortest -to 2.044 INPUT_02_cut_a.mp4 && \
ffmpeg -i INPUT_02_cut.mp4 -map 0 -map_metadata 0 -movflags use_metadata_tags -c copy -c:v libx264 -preset veryslow -crf 18 -tune grain -profile:v main -level:v 3.2 -video_track_timescale 24k -avoid_negative_ts make_zero -ss 1:04.606 INPUT_02_cut_b.mp4 && \
ffmpeg -i INPUT_04_cut.mp4 -map 0 -map_metadata 0 -movflags use_metadata_tags -c copy -c:v libx264 -preset veryslow -crf 18 -tune grain -profile:v main -level:v 3.2 -video_track_timescale 24k -shortest -to 1.752 INPUT_04_cut_a.mp4 && \
ffmpeg -i INPUT_04_cut.mp4 -map 0 -map_metadata 0 -movflags use_metadata_tags -c copy -c:v libx264 -preset veryslow -crf 18 -tune grain -profile:v main -level:v 3.2 -video_track_timescale 24k -avoid_negative_ts make_zero -ss 22.648 INPUT_04_cut_b.mp4 && \
ffmpeg -i INPUT_06_cut.mp4 -map 0 -map_metadata 0 -movflags use_metadata_tags -c copy -c:v libx264 -preset veryslow -crf 18 -tune grain -profile:v main -level:v 3.2 -video_track_timescale 24k -shortest -to 2.002 INPUT_06_cut_a.mp4 && \
ffmpeg -i INPUT_06_cut.mp4 -map 0 -map_metadata 0 -movflags use_metadata_tags -c copy -c:v libx264 -preset veryslow -crf 18 -tune grain -profile:v main -level:v 3.2 -video_track_timescale 24k -avoid_negative_ts make_zero -ss 17:28.338 INPUT_06_cut_b.mp4 && \
ffmpeg -i INPUT_08_cut.mp4 -map 0 -map_metadata 0 -movflags use_metadata_tags -c copy -c:v libx264 -preset veryslow -crf 18 -tune grain -profile:v main -level:v 3.2 -video_track_timescale 24k -shortest -to 2.044 INPUT_08_cut_a.mp4
file 'INPUT_00_cut_b.mp4' file 'INPUT_01.mp4' file 'INPUT_02_cut_a.mp4' file 'INPUT_02_cut_b.mp4' file 'INPUT_03.mp4' file 'INPUT_04_cut_a.mp4' file 'INPUT_04_cut_b.mp4' file 'INPUT_05.mp4' file 'INPUT_06_cut_a.mp4' file 'INPUT_06_cut_b.mp4' file 'INPUT_07.mp4' file 'INPUT_08_cut_a.mp4'
ffmpeg -f concat -safe 0 -i concat.txt -map 0 -map_metadata 0 -movflags use_metadata_tags+faststart -c copy OUTPUT.mp4
There is some choppiness when playing OUTPUT.mp4 with hardware decoding enabled in old versions of VLC. However, the latest version plays it flawlessly, as does other software like MPV or Chrome.
Until LosslessCut can cut exactly, this is my less convenient workflow.
Anyways it's definitely something to try implementing for LosslessCut's smart cut. I wonder how well this
segment
method works with other codecs like h265. I know that losslesscut smart cut works well for some h265 files.
Perhaps relevant: I have tested my approach on an MP4 file which contained AV1, Opus and embedded text subtitles (e.g. SubRip), which worked fine as long as I used the latest FFmpeg version. It also worked fine on an MP4 file containing VP9 and Opus. Some subtitle issues were also fixed by using the latest version. I have edited my original post to reflect this.
Using the latest FFmpeg did not make Matroska freezing issues go away.
I found an issue where ffprobe reports a ridiculously low bitrate for the stream versus the overall bitrate for the file; documented here #1997
This results in an unusable encoded section at the beginning when using SmartCut; 1/40th the expected bitrate in this particular situation. The issue appears due to how ffprobe/ffmpeg calculates the stream bitrate, which appears to be an estimate.
next version will allow setting a custom bitrate
This might not be the proper place to write this up, but couldn't really find an appropriate place. What losslesscut currently tries to set for a file I'm testing:
Looks about 600mbit, which should be plenty, but unfortunately, it gets silently ignored, and we end up with the default x264 options (CRF rate control, with a value of 23), which is going to be a tad low IMO.
Allowing a custom bitrate in the future is nice and well, but setting a bitrate does not yield predictable quality across a range of resolution, framerates, complexity of content etc.
I personally believe that a better approach would be to aim for a quality/accuracy based rate control, with a very high quality default (for x264, say CRF 15-18). Maybe even allow a an adjustable value, or at dropdown of a few presets (0 = lossless'ish, 15 = human visually lossless, 18= very high quality), for example.
Keep up the good work mifi, loving your tool! Heia Norge! ;)
@flaeri If there isn't a proper place for it, you can make a new issue and kindly remove the comment to prevent confusion.
Was the issue of audio stuttering on smart cut ever fixed (h264 format video)? It doesn't happen constantly, but it happens for me consistently 2 seconds in to every video I do, there is a very slight stutter. It works for my purposes for now, but it's the one and only flaw I've seen in LLC. Fresh install of LLC.
Was the issue of audio stuttering on smart cut ever fixed (h264 format video)? It doesn't happen constantly, but it happens for me consistently 2 seconds in to every video I do, there is a very slight stutter. It works for my purposes for now, but it's the one and only flaw I've seen in LLC. Fresh install of LLC.
not solved yet unfortunately. it's most likely a off-by-one-frame issue
We cannot cut between keyframes, it's a technical limitation of modern video codecs. So ffmpeg will choose the nearest keyframe, which will not be the exact time you requested.
Might be able to use avcut for h264 although it is experimental. See https://github.com/mifi/lossless-cut/issues/372#issuecomment-761844667
See also discussion in #13
Update: Have made an initial implementation of this. This experimental feature will re-encode the part of the video from the cutpoint until the next keyframe in order to attempt to make a 100% accurate cut while losslessly copying the rest of the segment.
VideoRedo
VideoRedo is a now-defunct software that had feature overlap with LosslessCut.
Smart Media Cutter
Recently built in order to replace VideoRedo
They open sourced their smart cutting algorithm, could take inspiration from it: https://github.com/skeskinen/smartcut
Remaining issues
Inaccurate seeking
Seeking is not always accurate, causing glitches and/or jumps in the stitch points. This is often because I haven't found a way to consistently seek to and cut on an exact frame. Sometimes it could be caused by the "audio samples" problem below. If you want to experiment, you can try to remove the audio track and cut only video to see if it makes a difference.
Audio samples problem
Cut time accuracy/glitches when audio samples are large? (try without audio and see if it makes a difference). A workaround seems to be to cut audio/video separately and then merge them, with the tradeoff of potential audio/video de-sync:
subtitles
Other improvements
Cannot find any keyframe after the desired start cut point
, see #https://github.com/mifi/lossless-cut/issues/1190#issuecomment-2027227599Done
aomav1
insteadlibsvtav1
instead ofaomav1
(ffmpeg default) (codec needs to compiled for mac in ffmpeg-build-script) #1825Limitations
size/duration
(may be wrong)