Implement lossy operations

mifi commented 4 years ago

Since lossless cutting by nature is inaccurate, and a lot of people request a more accurate cutting, I will consider adding an accurate and lossy cutting option or make a separate app that has some of the following features:

Main feature

Add an additional operating mode in LosslessCut where all operations become lossy. This will effectively make LosslessCut into an accurate video/audio editor, but exports will be much slower.

Replace all -c copy with lossy encoding like libx264 (and possibly per-platform counterparts like videotoolbox on MacOS)

Other suggested features

Video features

[ ] Crop, mirror, rotate, resize, stretch
[ ] Integration with editly or reactive-video?
[ ] Speed up / slow down video/segments #586
[ ] Export segments as GIFs #662
[ ] Create video from images
[ ] color grading #807
[ ] Create a video from a slideshow of 1 or more photos with an audio track added to it
- [ ] Turn the cover art image into a Poster Frame and add a duration setting. I.E. It would display for x seconds at the beginning of the video (need to match encoding of the other video, like #126)
[ ] Add pause/black/title between segments #255

Overlays

[ ] Text watermark: adjust font, font style, size, color, gradient, transparency, text box(shape) & box color, border line(color & thickness) etc.
[ ] Image watermark: cut the imported image, resize it with the mouse, adjust the transparency, Add border line(color & thickness), etc. (Positioning: Click and drag text/image on the screen to move it to the desired position)
[ ] Burning subtitles into video #250
[ ] blur/redact certain area https://github.com/mifi/lossless-cut/issues/691#issuecomment-1053935905

Audio features

[ ] change audio volume / normalize audio
[ ] Mix audio tracks together with amix #607
[ ] #1733
[ ] #495

Audio & video features

[ ] Fade audio and video in/out to/from black #1044 #696 #1610 #1944
[ ] Cross-fade audio/video #1983 #2114
[ ] Export to a different codec

Frequently requested things that are not necessarily lossy but still need a codec conversion:

[x] Convert subtitle format (codec) #418
[ ] Encode audio track (e.g to PCM or FLAC). See #476

Inspiration

https://amiaopensource.github.io/ffmprovisr/

capekasa commented 3 years ago

iskysoft iMedia Converter Deluxe can do lossless edits if source and destination compression stays the same type. Cropping is not pixel exact as it will be restricted to the nearest JPEG blocks as I understand it. so like nearest 8 or 16 pixels. I use cropping a lot and YES, often enough NOT keeping same aspect ratio

mifi commented 3 years ago

Oh, do you mean metadata cropping like this?

crop_left
crop_right
crop_top
crop_bottom

If this is not what iMedia converter is doing, then I don't know how it does it. I have found no information online about how to do lossless cropping.

capekasa commented 3 years ago

I have no idea how they do it technically. as MPG is based on JPG which reduces a picture into blocks I can imagine it should be possible to crop around those blocks. You are the specialist. Maybe add some monetisation options for donations or patreon or so. Popular. I use your app a lot now preparing videos for DeepFace Lab. Awesome fast navigation in video and in/out marking to select specific scenes only.

mifi commented 3 years ago

Yea, I read that for the old MPEG2 someone did something like that, but modern codecs like h264/h265 are really complex stuff and I'm not really a specialist on such low level mathematical codec logic, so I'm not gonna be able to do that. but i will be on the lookout for any such code out there, because I agree it would be awesome to do. I believe the right search word is h264 DCT reencode, however I can only find scientific papers.

orangelynx commented 3 years ago

A first step in that direction might be to allow for custom additional ffmpeg options / flags in expert mode. This way, one could just add eg -c:v libx264 -crf 28 to get some compression.

BotoX commented 3 years ago

https://github.com/anyc/avcut Check out this project for smart perfect cutting "SmartCut"

I've successfully cut H264 with lossless-cut as a frontend/GUI only and transposing the project .csv to command line arguments for avcut. It cuts exact, mostly lossless and with no noticeable quality loss for the non-lossless parts. Integrating that into lossless-cut as an option might already be enough for the start.

mifi commented 3 years ago

Out of curiosity: Why do people want lossy, slow accurate cutting in losslesscut? What's the benefit of using losslesscut instead of something like iMovie, shotcut, openshot etc?

orangelynx commented 3 years ago

For me it's either that the alternatives didn't work as smoothly or were too bloated. My main usecase is trimming game-content captured from nvidia shadowplay and precompressing it as to avoid the overly heavy and speed optimized compression of youtube and streamable.

mattack1 commented 3 years ago

I didn't know about the others. I actually haven't used iMovie, I thought it was for much more sophisticated editing, and made 'project files' out of your movie, and not just edit single existing movie files.

I'm just editing recordings downloaded from my Tivo, either to save the musical performances from TV shows or sometimes comedy skits (e.g. from SNL). If Quicktime Player let you save multiple sections, I'd use it. It only let you trim to one saved section.

joef42 commented 3 years ago

Similar for me, I just needed a tool to cut up several hours of old family videos into very small clips. I tried some other tools, but lossless-cut was just the perfect solution to set all the right cutting points for me. But I wanted to cut exactly between two clips, so any inaccurate cutting didn't work for me. I ended up writing my own script that used the csv file produced by lossless-cut to call ffmpeg with my own settings.

I like the idea as suggested before of having an option to set custom ffmpeg parameters. Yes, it does not really fit the name and the original intention anymore, but I think it would be a great extension for lossless-cut.

mifi commented 3 years ago

one issue I thought about is that h264 may require licensing to use, even in open source projects, so may have to use a different codec for lossy operations (eg VP8/VP9 which afaik are free)

orangelynx commented 3 years ago

Hmm not an expert, but IIRC losslesscut already supports non-keyframe based cutting, for which ffmpeg needs to use H264 anyway, right? Perhaps even for keyframe-based cutting, i don't know. Ultimately, if losslesscut just is an interface for ffmpeg and doesn't distribute ffmpeg binaries itself, I think you are good. And given the amount of open source projects I've seen that distribute ffmpeg or h264, I don't think the license is a big issue for open source projects at all. But of course you should stay on the safe side, legally.

mifi commented 3 years ago

llc doesn't support non-keyframe based cutting, so it doesn't use the h264 codec. it only manipulates the stream without decoding/encoding anything. I know that many open source projects have ffmpeg in them, but I believe that they either:

do not include h264 in their download
or do not ship a compiled version of the app (source code only)
or they are based in a country that doesn't recognise US patent laws (VLC is in france and so is handbrake). OBS studio I'm not sure how they do it, as it seems they include the x264 encoder and they are from California. could it be they have less than 100.000 downloads total? (I doubt it), or they do indeed pay royalties.

I found this article: https://jina-liu.medium.com/settle-your-questions-about-h-264-license-cost-once-and-for-all-hopefully-a058c2149256

As I understand it, on macos, we can use the operating system's built in hardware h264 encoder (-c:v h264_videotoolbox) without paying license (because Apple is already paying license for that when they build their Macs). Windows/linux has something similar but it will depend on GPU and OS support: https://trac.ffmpeg.org/wiki/HWAccelIntro It seems that on windows, there is no encoding library that works with all GPUs intel, amd and nvidia.

Another option is to let the user manually download ffmpeg separately and choose the path to it from llc. #400

I just did some tests and it turns out that x264 is miles ahead of the other free alternatives (vp8,vp9,av1,theora) in terms of encoding speed.

QuantumToasted commented 3 years ago

I just started using this app as an alternative to Win10's own "video editor" for editing game clips. The biggest drawback of it is that it will turn higher-FPS videos into 30 FPS videos with the user having no say or control in the matter from the small amount of research I did.

I found this while looking for recommendations/alternatives and it's great for simply trimming 30 second or 1 minute Shadowplay recordings into the 10 or so second clips I share with friends, etc. without losing the (more) butter smooth 60fps of the source recording.

One thing that I did however enjoy from Win10's offering is that it offered a built-in slow-motion edit tool, to (do its best to) slow down a very small portion of the video for whatever purposes I needed.

I understand that incorporating features like this on something meant to perform lossless operations would kind of defeat the point and the "branding", but at the same time it could certainly be an opportunity to branch out in one way or another to try to expand on its "swiss army knife" video editor status even further.

I'm sure other users of this might agree that even if it might go against the original philosophy of the app, that even including the option (via something like custom FFMPEG parameters like others suggested) for users to perform lossy operations on their video if they choose certainly would only hurt the people who willingly are fine with the fact that it is in fact a lossy operation. If you'd like, you could even make it very clear when a certain operation is in fact lossy, or whatever you desire.

Droyk commented 2 years ago

Hey @mifi a quick question did you manage to find out how Quick Time managed to cut it accurately losslessly? are you going in the same direction? or to get accurate times it will have to be lossy! also, is the Quick time method is completely lossless or there is some quality degradation compared to what lossless cut currently offers?

mifi commented 2 years ago

I think quicktime uses smart cut, which implies only re-encoding the parts around the keyframe, so only those parts will be lossy and the rest will be lossless. I think quicktime only supports a few codecs so it’s easier for them. Losslesscut supports a lot of codecs and they all need a different way of smart cutting. See #126

slrslr commented 8 months ago

Yes, lossy operation (re-encoding) is important for merging multiple files that differ in some parameters and or for the cases when no cutting method (keyframe etc.) works to produce error free joined video out of segments (i have seen several such cases), it would be handy and important if user can turn on re-encoding (lossy operation mentioned in this "issue") which would guarantee that the multiple segments or videos merging will always work.

vertigo220 commented 7 months ago

Not sure if this is better as a comment here or its own feature request, but one option that would be nice to have and would provide one method of performing lossy edits is to use LC to set everything up and then have it send the necessary info to Handbrake to do encodes. That is, it could tell Handbrake what frames to encode. This would, of course, mean the entire video would be encoded, but sometimes that's actually what I want. For example, right now I have some videos I want to both remove parts of and encode, and while a smart encode feature would be extremely useful in certain situations (and I've had the need for it before and will again in the future), in this case it would cause those frames to be encoded by LC only to be encoded yet again when I encode LC's output. So it would be much better to just use LC to select the portions I want to keep/delete and then use Handbrake to encode it all.

The main problem I see with this is that you can only select a single range in Handbrake, and so LC couldn't send multiple ranges for a single encode. It would have to send one range and instruct Handbrake to add that to the queue, then the next, and so on. Assuming this is even possible, the other issue is setting the encoding options. This could be done in LC, but that would require a lot of work, or LC could instruct Handbrake to use a user-defined preset.

Again, I'm not even sure if this all would be possible, or how difficult it would be, and so another, simpler option (though obviously not as streamlined, but still not bad) would be to take the frame numbers from LC once the editing is done and manually enter them into Handbrake. It appears this should already be possible by exporting the frame numbers in CSV, though I haven't tested it yet for accuracy, but I wanted to mention it in case anyone else is looking to do this type of operation and didn't realize that's an option.

Assuming it is possible to automate Handbrake in this way (or another, similar program if not HB), that ability could be manipulated to sort of cheat and implement a smart-cut mode (#126). LC could send just the frames between the keyframes and the cut points to HB for processing and take the output files and merge them onto the segments all in the background, automatically, to achieve this without having to actually do any of the encoding itself.

Droyk commented 7 months ago

until a better robust solution isn't what @vertigo220 suggesting a best overall! @mifi

mifi commented 7 months ago

@vertigo220 I think what you're asking is some custom post-export event that gets triggered when the user clicks export. See issue #980 "afterExport event"

vertigo220 commented 7 months ago

Yeah, looks like it. It could use the various options listed here to feed the info to Handbrake for processing. It's not clear based on that page whether it can add multiple items to the queue and start the encode automatically, so that would be something to test and, if it can't do it, maybe the devs would be willing to add options for that. The idea would be like this:

Configure a preset in Handbrake with all the preferred settings
Have a place in LC's settings to provide necessary info to provide Handbrake via the CLI (Handbrake path, encode destination, and preset name come to mind immediately, though there may be use for others, and there could be an advanced box where users could enter entire command strings which would override anything in the other boxes)
Once done cutting, LC would determine all the frame numbers from each cut point to the nearest keyframe before or after, depending on whether the cut point is at the start or end of a clip, and feed successive commands with the necessary parameters, e.g. something like

Path\To\Handbrake\HandbrakeCLI --preset "Preset Name" -i "Path\To\File\Loaded\In\LC\File Name.mkv" --start-at frames:123 --stop-at frames:321 -o "Configured\Destination\Path\File Name-001.mkv"

LC would then wait until Handbrake is done processing (this part could admittedly get tricky, and maybe it could be left to the user to tell it that Handbrake is done by clicking a button, with confirmation in case of an accidental click) and then automatically take all the encoded files and merge them in order with all their corresponding clips, and merge all the clips together, ideally with chapter marks at each clip. For example, if you end up with Filename-001.mkv, Filename-002.mkv...Filename-006.mkv, then it would create a single file with 001, the clip between 001 and 002, {CHAPTER}, 003, clip between 003 and 004, 004, {CHAPTER}, 005...006. This could be done within LC (I imagine it already has most, if not all, of the capabilities necessary for this) or by feeding yet more commands to MKVToolNix.

A couple potential issues and solutions:

Running out of space while encoding, resulting in Handbrake pausing and holding up and potentially even messing up the process (shouldn't happen if done properly).
- Check available space on the destination drive before beginning and warn the user if too low. Without LC knowing the encoding parameters, it will be impossible for it to know how much space will be needed, but it can look at the total times and estimate, perhaps even by using the same bitrate as the input file, since that would almost certainly be worst-case scenario. Alternately, by providing all the info in LC, it would be able to better estimate this, though that would have its own issues and a rough estimate should be good enough.
- Save the project (all the cut points, etc) before starting to ensure if something goes wrong all is not lost (should be done anyway).
Configuring a preset in Handbrake that isn't compatible with the source video and therefore creates clips that can't be merged, e.g. by setting a different resolution, audio codec, etc.
- Probably the best way to handle this, though also the most involved, is to create an interface within LC identical to Handbrake's, where users can set everything how they want it and then have LC take all those settings and build a command-line output to feed into Handbrake. This way, LC could check the settings and ensure there are no issues. It also may be a bit easier and more streamlined to use. It would also allow for better size estimates as mentioned before. It would just be more work, so not sure if you'd want to do that or not.
- It could alternately encode a single short clip using the preset but at Ultrafast speed (not sure if that can be done, as the speed would be part of the preset, so would have to see if it could override that; if not, it probably wouldn't matter too much since the clip would only be a few seconds) and then perform a test to see if it can merge the clip with a piece of the video. If it can, then it proceeds, and if not, it warns the user (ideally letting them know the issue, which should be doable by looking at the metadata for the input and output files and analyzing it for differences that would cause them to be incompatible). It would then auto-delete the test encode and test merged file. It should probably not auto-delete the encodes even after done merging, just in case there's an issue, though this could be an option.

So essentially, once initially configured, whether in Handbrake with a few options in LC or all in LC, once done setting cut points, it should be as simple as exporting and having LC automatically generate the necessary commands, feed them to Handbrake, and take the output once Handbrake is done and automatically merge them to their corresponding clips and create an output video, making the entire process no different from how it is now except for having to wait and let Handbrake work in the background.

ETA: With this added capability, another benefit would be to allow for the ironic, but very useful, ability to perform cuts in LC then send the clips in their entirety to Handbrake. While I sometimes want to do lossless editing, I sometimes want to edit AND encode, and having this capability with LC would greatly simplify doing that as well.

Freso commented 6 months ago

FWIW, I would very much like for losslesscut to not implement lossy features, exactly because that’s what I’d use a more full‐featured video editor for. losslesscut is fantastic for quick trimming down or splitting up videos prior to being put to storage or imported into some other editor, and I would really dislike seeing it become bogged down with a lot of features of regular AV editors.

I'm sure other users of this might agree […], that even including the option ([…]) for users to perform lossy operations […] certainly would only hurt the people who willingly are fine with the fact that it is in fact a lossy operation.

The code needed would introduce more logic paths into the code making the program bulkier both in disk size and memory size and complexity. It would also be another source for bugs and other issues and otherwise more code maintenance. It also opens up for features like Crop, mirror, rotate, resize, stretch and such that leads down a path where LosslessCut is “just another” general purpose video editor, which won’t work as smoothly or [be] too bloated.

I’m not against LlC introducing “lossless JPEG cropping” style video cropping of video formats where that makes sense (I use cropgui a decent amount for images already) and I can also see the argument for allowing for lossy cutting – and as long as that’s the extent of allowing “lossyness” I probably wouldn’t object. I think it would still be better to do something like what vertigo220 is proposing of simply passing on the parameters from LlC to allow another program to deal with all the lossy editing (could even be passed off to editly!).

Note that some of this post-processing using other tools could also happen outside of LlC, e.g., given a CSV like

483.033497,818.980667,
912.928244,2785.008743,

you could use a script to translate that to 0 483.033497 818.980667 912.928244 2785.008743 - which could then be appended to an avcut command, such as avcut -i input.avi -o output.mkv 0 483.033497 818.980667 912.928244 2785.008743 -.

vertigo220 commented 6 months ago

@Freso Out of curiosity, what other editors have you used that would perform this type of functionality? I came across LC looking for something to do this type of editing and, despite its name, gave it a try because I've not found another good editor. After trying a few others, LC is the one I've found to have the best UI, though of course it's too limited for many of my use cases due to its nature. So unlike you, I'd like to see it expand beyond the limitations of its name, but that's just because I've not been able to find anything else that's halfway decent, and LC seems the most promising for being what I need. So I'm interested in what you've tried and what you use, that perhaps has a similar design but more complete editing capabilities, while not requiring a degree in audio-visual or numerous hours of practice to figure out how to do basic operations. If there's something out there, I'd really love to know.

I paid for one software that turned out to be terrible, and had a bug that prevented it from being useful, which was only fixed (supposedly) in the next major version, meaning I'd have to pay again to actually be able to use the software I'd already bought. VideoReDo seems the most promising from my research, but I'm already hesitant to spend that kind of money on something I only need here and there, especially after the experience with the other paid software (not to mention many other programs I've bought over the years only to be disappointed in or have them completely change or be abandoned).

I like the idea of free software (as in beer and speech), both for the philosophy behind it and because I don't really want to spend $50+ on multiple various programs that I might use a few times a year, and I can't find anything, free or otherwise (aside from possible VRD) with this functionality combined with a good UI. LC is by far the closest I've found, hence my interest in the project. But I'm always open to suggestions, so if you feel LC is best to stay how it is because there are other, perfectly good alternatives in the "full-featured" and "regular AV editor" space, I'm genuinely interested in what you use for those needs, and why you don't just use those editors for the functions LC provides.

Freso commented 6 months ago

This is getting off-topic, but Kdenlive has so far served me well enough when I’ve needed more than what I’ve been able to manage to do using LlC, various CLI utilities, or other more specialised software.

I'm genuinely interested in […] why you don't just use those editors

Because:

losslesscut is fantastic for quick trimming down or splitting up videos prior to being put to storage or imported into some other editor

LlC is far more light‐weight than full‐featured AV editors. It starts up faster, it operates faster, and it has fewer moving parts. My hope is that LlC will stay that way.

vertigo220 commented 6 months ago

@Freso Thanks for the recommendation. Been playing with it a bit, seems like ok software, though not terribly intuitive and took a bit more time to figure out than I'd prefer, but I was able to do what I wanted, which was cutting out segments from a file. Unfortunately, AFAICT it does not have a smart encode feature, so I have to encode the entire video, which not only takes a lot longer, but results in quality loss. Not only that, it doesn't even appear to allow much customization in the encoding settings. So unfortunately, it wasn't much help and just ended up being a waste of time, and I'm back to square one. And so, since it, and all other programs I've tried, can't do that while being relatively easy to use, I'd argue adding more functionality to LC would be beneficial. Considering that KDEnlive, despite its increased complexity, starts up in 5 seconds on my old computer and 2 seconds on my new one, I'm not sure adding some more features into LC should be a concern in that regard.

Alexios13 commented 3 months ago

Out of curiosity: Why do people want lossy, slow accurate cutting in losslesscut? What's the benefit of using losslesscut instead of something like iMovie, shotcut, openshot etc?

Hi! Losslesscut is simple and intuitive. I asked my mum to use it and immediately she was working with no problems. This is the reason I want an option for lossy operations. In her workflow, she needs to cut parts of video (usually just a few seconds) and export it again without that parts (in 264, 265, av1 codecs). The videos she is editing contains multiple subtitles and audio tracks and of course she needs frame accurate cutting. So, until smart cut is working, a way to have the beautiful and intuitive interface of losslesscut with all of its current futures but with the ability to do frame accurate cutting is lossy operations.

iMovie, shotcut, openshot are either hard to use, or have other issues, or are missing a feature. (for example, with losslesscut we can cut a part of a video with subtitles and the subtitles will not lose their sync from the cut and afterward)

Please give us an update if lossy operations would be possible to be implemented in LC

Droyk commented 3 months ago

FWIW, I would very much like for losslesscut to not implement lossy features, exactly because that’s what I’d use a more full‐featured video editor for. losslesscut is fantastic for quick trimming down or splitting up videos prior to being put to storage or imported into some other editor, and I would really dislike seeing it become bogged down with a lot of features of regular AV editors.

I'm sure other users of this might agree […], that even including the option ([…]) for users to perform lossy operations […] certainly would only hurt the people who willingly are fine with the fact that it is in fact a lossy operation.

The code needed would introduce more logic paths into the code making the program bulkier both in disk size and memory size and complexity. It would also be another source for bugs and other issues and otherwise more code maintenance. It also opens up for features like Crop, mirror, rotate, resize, stretch and such that leads down a path where LosslessCut is “just another” general purpose video editor, which won’t work as smoothly or [be] too bloated.

I’m not against LlC introducing “lossless JPEG cropping” style video cropping of video formats where that makes sense (I use cropgui a decent amount for images already) and I can also see the argument for allowing for lossy cutting – and as long as that’s the extent of allowing “lossyness” I probably wouldn’t object. I think it would still be better to do something like what vertigo220 is proposing of simply passing on the parameters from LlC to allow another program to deal with all the lossy editing (could even be passed off to editly!).

Note that some of this post-processing using other tools could also happen outside of LlC, e.g., given a CSV like
483.033497,818.980667,
912.928244,2785.008743,
you could use a script to translate that to 0 483.033497 818.980667 912.928244 2785.008743 - which could then be appended to an avcut command, such as avcut -i input.avi -o output.mkv 0 483.033497 818.980667 912.928244 2785.008743 -.

I think simple solution for a problem like this is to provide a choice. Provide d/f versions in the download/release section that is lossless, lossy, or combined. or add all three into the one & add a launcher to launch whichever version one wants to open for whatever reason. So everyone can remain happy. obviously it means more work.

mifi / lossless-cut