Improve waveform

[mifi]

Waveform can currently only be rendered for a short section (now set to 30 sec), because rendering longer will take a lot of time. Need to find a way to quickly render a waveform of a long track. (hours long)

• [x] Rendering 30 seconds at a time shouldn't prevent you from still displaying the already rendered waveform. It's fine if a 2 minute song renders in 4 sections
• [x] Make waveform bigger (With audio files, the entire space kept for video is wasted real estate. The audio waveform could be much bigger and easier to see) #6
• [x] a bit-perfect waveform editor in the split-and-join scheme for those who just want to edit the audio #736
• [x] allow showing waveform at the same time as thumbnails #843 / allow scan/rendering the entire waveform when I first load the file #1193
• [ ] Find a better way to render an arbitrary portion of a file
• [ ] Rough full-timeline waveform at 1x, finer waveform when zooming in https://github.com/mifi/lossless-cut/issues/260#issuecomment-605603456
• [ ] The full-height waveform view is currently not linked with the bottom editing segment. If they can be linked together, it will greatly facilitate waveform-based editing. For example, when splitting a segment, the waveform graph should also be split into two segments; when deleting a segment, the corresponding section on the waveform graph should also show as deleted (with dark background or diagonal pattern).
• [ ] allow showing channels as separate or combined (currently only separate is supported)
• [ ] allow choosing which track (stream) to render waveform from #1590

Inspiration:

• https://www.audacityteam.org/
• https://rogueamoeba.com/fission/
• http://mpesch3.de/index.html
• https://www.rogueamoeba.com/fission/

[mifi]

The main issue for rendering the whole waveform for the whole file is that I'm not sure how to effectively run through a file of many GB in a few seconds to extract a rough "zoomed out" average of the audio waveform. Even extracting audio as raw frames takes similar time.

ffmpeg -i file.mp4 -c copy -vn -map 'a:0' -acodec pcm_s32le -ar 100 -f s32le - |
ffmpeg -f s32le -acodec pcm_s32le -i - -filter_complex 'aformat=channel_layouts=mono,showwavespic=s=10000x120:scale=sqrt:colors=#ff0000' -frames:v 1 -vcodec png -f image2 -y out.png

Basically here I'm resampling audio to 100Hz which somewhat speeds up but gives less detail when zoomed in. With a 400MB video file that is 1hr long, this operation takes about 10 seconds. vorbis (mp4a / 0x6134706D), 44100 Hz, stereo, fltp (default)

• https://github.com/katspaugh/wavesurfer.js/issues/1171
• https://github.com/bbc/audiowaveform

[misiektw]

Any reason why not to make lookup table/list of 30s fragments for whole video and use that as cache? I don't mind waiting even few minutes upon loading of video, when later I could zoom out with waveform during editing.

[mifi]

No, just that it needs more logic implemented to handle properly the overlap etc

[markusd1984]

I was wondering why I only saw parts of the waveform, that explains :D but definitely would be good to be able to get a full render for up to 1-2hrs max, if somebody wants a waveform I don't see why one wouldn't want to wait 10-20secs. :)

Certainly for smaller videos would be good, mine was less then 2mins and still only showed parts. When clicking into different timeline areas other parts are loaded instead of the waveform being combined/kept, which would be nice to maintain (e.g. I click the beginning, middle and end and would have the full waveform timeline visible).

[mifi]

Yea definitely could implement some longer waveform rendering some day

[markusd1984]

👍 if loading the entire waveform is more difficult perhaps until then a stich together waveforms while clicking into different areas of the timeline (if that's easier to implement).

[opusforlife2]

Rendering 30 seconds at a time shouldn't prevent you from still displaying the already rendered waveform. It's fine if a 2 minute song renders in 4 sections

@mifi Is it possible to tackle just this? I think this is a much lower hanging fruit than actively parsing the entire file.

[mifi]

just keeping the already rendered segments and rendering them all shouldn't be too hard, but I'm afraid that it will cause losslesscut to slow down more and more, as it will have more and more waveforms to render

[opusforlife2]

@mifi How about a sane upper time limit for now? Something like 10 minutes. It covers most songs.

[libor-m]

I was using lossless-cut to clean up some of my lecture videos (3+ hours) - it's an awesome tool, I really did not want to re-encode 10+ hours of video. But I was really missing the waveform, which would help me find and cut out the breaks easily.

I did some research, here's what I found (timing on 1 Gig mp4 files with aac, 6 years old laptop with i5-6300u):

• it's possible to extract RMS levels via ffprobe's astats filter directly, ~50 seconds
• BBC's audiowaveform is much faster than astats (~20 seconds), when the wav is piped to it from ffmpeg

I set the commands to produce 10 numbers per second of audio. The resulting files are around 1 MB (for 3+ hours of video). I think it's more than reasonable to wait 20 seconds and spend 1 MB of drive space to get a nice zoomable waveform for the rest of your workflow (rendered via eg peaks.js or wavesurfer.js).

Regrettably I don't have enough time to familiarize myself with the codebase of this project to create a meaningful PR.

test commands

The input audio rate is 48k.

ffprobe alone, number of frames on the output is given by audio rate / asetnsamples, here set to 10 per second.

INPUT=video.mp4

time ffprobe -v error \
  -f lavfi -i "amovie=${INPUT},asetnsamples=4800,astats=metadata=1:reset=1:measure_perchannel=none:measure_overall=RMS_level" \
  -show_entries frame_tags=lavfi.astats.Overall.RMS_level \
  -of csv=p=0 \
> output-level-48k.csv

ffmpeg and audiowaveform, using --pixels-per-second 10 to get the same number of samples

time ffmpeg -i $INPUT -f wav - | 
  audiowaveform --input-format wav --pixels-per-second 10 -o bbc-waveform.json -b 8

[mifi]

Have now done a minor improvement:

• buffer the last 100 rendered waveform segments

[oliko-tk]

Seria bueno poder tener una opcion para poder exportar todos los fotrogramas de uno o varios segmentos de un rango seleccionado del video.

[gkostov]

Do you think it is reasonable to adjust the ffmpegExtractWindow variable (https://github.com/mifi/lossless-cut/blob/96ca1598e053c1de1b552d20529d431add126481/src/App.jsx#L90) according to the current zoom level? That would mean getting a finer waveform when zooming in. (it will of course require a multi-level cache but that's just a technical point) Also, at many places I've encountered the waveform to be specified at 30 seconds while ffmpegExtractWindow says it is actually 60 seconds (up to 30 seconds before and up to 30 after the current time). Is this correct?

Thanks

[mifi]

i'll add it to the list above

[mifi]

Some new waveform improvements to be excited for coming soon!

https://user-images.githubusercontent.com/402547/224534033-7f13b28b-2d65-412b-ae4b-ba6b02627231.mp4

[Dean-Corso]

@mifi,

sounds good. Can you also zoom into the waveform? It still looks pretty small and not detailed enough to make exact cuts. A zoom in so that you can really see the silences would be good. Just to prevent setting cut points inside an audio crackle location of wave.

[mifi]

yes it can be zoomed in, although resolution is limited

[mandrael]

Thanks for the update! I'm looking forward to a solution for using the waveform for the whole timeline. Currently I write down the cut times in Potplayer (mouse-over video-preview) or davinci resolve (full waveform) and cut afterwards in losslesscut. If I could see the silences of the long zoom conferences/courses, I could do everthing in losslesscut.

[mifi]

• [ ] rough full-timeline waveform at 1x, finer waveform when zooming in Improve waveform #260 (comment)

@mandrael you mean this?

[Dean-Corso]

Hi @mifi,

"implement full screen audio waveform" you said. How? How to make the view larger? Also have a problem when trying to export segments in smartcut... ...whats that? What to do with that tiny wave view? With this you can't do any exact cuts. Found another problem, on the image you can see 2 segments I wanted to cut/merge but I get this error you can see. When I disable the second segment then it does output the first one but in this case it's a buggy cut and the video/audio stream so there is some kind of cut inside the segment and you can hear a clear crack what you not can hear in the preview itself what means the output segment is NOT same as the preview segment. Also the output segment has more frames as the preview segment = no clean cut possible with smartcut. So you have to decide what kind of app you wanna make. Just a solala (round about working) app or a good app? Just don't understand why guys always make everything so complicated. I told you already few times in the past just adding a keyframe cut version and a Re-Encode cut version to make 1:1 cuts like in a normal video editor. I need both versions but then they should also work for 100% and just solala you know. At the moment you app is useless for me which I can not trust to get that results in my output file I did set in the timeline. Also I don't wanna verify each time whether your made output file is same as my cuts points I did set etc you know. Oh boy. You should know that functionality & exactness is A & O so you should better care about that first. Also see you still not managed the themes as some of use wished (not too dark & bright) and now I see you also changed the title bar and menus background only in dark colors even when changing to the bright theme! Another time, Oh boy! All in all I'm still pretty disappointed and at this point I also have to admit that "Lossless-Cut" is no alternatively video tool for me. Maybe it can be one in the future if you do some changes I told you already before etc but somehow I don't believe it anymore that this will happen. Anyway, I wish you good luck for the future @mifi.

[JokerQyou]

Some new waveform improvements to be excited for coming soon!

full_waveform.mp4

I'm sorry, but I failed to find a way to view waveform in such full height mode, is there anything to update to the tutorial? @mifi I also find that waveform in new version (3.54) renders multiple tracks (L and R channels for stero audio) but the view height is still the same. This makes it much more difficult to cut based on waveform. In the old versions the merged waveform is much cleaer.

This is 3.54:

This is 3.45:

[Dean-Corso]

@mifi,

one correction, in case of the theme which was showing dark title bar and menus...so now its showing correctly bright title bar & menus. No idea why it last time just was showing dark only.

About the wave form. So you did not tell us that the larger and zoom able function only works when loading any audio files only without video. In this case I get a better few (but still pretty blur and not sharp) of wave forum but this we want to have always also when videos have one or more audio streams included you know. Make no sense when we get this view for audio files only. Below a image when loading just a audio file AAC in mp4 container (without video)... ...so you have to make this view (zoom able) like this in the audio wave area (small control under video area window). So the ctrl+mouse wheel works already as zoom. My idea: When user moves the mouse pointer on the small wave area control and ctrl+mouse wheel are used then just show the audio wave form in the video window area so that the user can see it very large like on my image to set the exact wanted cut segment start/end. Otherwise you can also add another button to switch from video window screen to audio wave form screen so both can share the same window control of course with a simple switch. Just my quick idea to solve that problem. Just remember trying to improve the wave form view / more details & sharpness what would be much better of course.

Question: Can you do / improve the reaction time of loaded videos when jumping / seeking in the video itself? For a better and quickly work flow it would be much helpfully to seek for/backward more liquid without to wait so long. Same fast like if you would just load a audio file only. Otherwise it could be maybe possible to load entire video into memory so that we users can edit the video also in real time if possible. Thanks.

PS: I hope you are not mad or sad about my messages last time about your app but sometimes its better to hear some opinions you maybe don't wanna hear you know (the truth or just my). Anyway, keep going and I will check your latest versions and send another feedback if desired.

[mifi]

@JokerQyou you double click on the waveform icon, then it will show full screen view. Or isn't double click working? Maybe it needs to be documented better.

I also find that waveform in new version (3.54) renders multiple tracks (L and R channels for stero audio) but the view height is still the same. This makes it much more difficult to cut based on waveform. In the old versions the merged waveform is much cleaer.

Maybe I can have an option for whether the waveforms should be separated or merged together (it's an option for ffmpeg.)

[mifi]

@Dean-Corso I'm not mad, but tbh when reading your comments it seems like you are furious. I'm a bit sad though that I have to spend so much time and energy reading your comments, because they contain 90% rant/negativity and 10% actual useful information, so when I see a 1000 word comment from you it doesn't make me want to read through all of it, and I will instead prioritise other issues. Also most of it is off-topic so it doesn't belong at all in the issue that you post it in.

Please search for existing issue first before you comment and then comment in the respective issue: https://github.com/mifi/lossless-cut/search?q=smart+cut&type=issues

As stated quite clearly, smart cut it experimental, so you cannot expect it to work perfectly. However if you have an issue, please provide a reproducible case so I can work on fixing the issue. Your ranting is not helpful at all.

as for why the title bar doesn't change color to match the dark/light mode, it's because you have to restart the app for the title bar to update.

Just don't understand why guys always make everything so complicated.

I don't know where to start.

[JokerQyou]

@mifi Indeed it works! Although it's not double-clicking, feels more like clicking to open the waveform view and then clicking again to open the full-height waveform view. Previously because I didn't know how to open this view and the dual-track waveform was too difficult to see clearly, I had to downgrade to version 3.45.

Some suggestions:

• The full-height waveform view is currently not linked with the bottom editing segment. If they can be linked together, it will greatly facilitate waveform-based editing. For example, when splitting a segment, the waveform graph should also be split into two segments; when deleting a segment, the corresponding section on the waveform graph should also show as deleted (with dark background or diagonal pattern).
• There should be a prompt somewhere for opening the full-height waveform view. Maybe after the user clicked to open the lower waveform graph, a prompt could pop up?

[mifi]

@mifi Indeed it works! Although it's not double-clicking, feels more like clicking to open the waveform view and then clicking again to open the full-height waveform view.

Not sure I understand, you mean double clicking doesn't work? as in you have to wait a certain time between your clicks, or else it registers as one click only? Which OS are you on?

The full-height waveform view is currently not linked with the bottom editing segment. If they can be linked together, it will greatly facilitate waveform-based editing. For example, when splitting a segment, the waveform graph should also be split into two segments; when deleting a segment, the corresponding section on the waveform graph should also show as deleted (with dark background or diagonal pattern).

Will add a TODO

• There should be a prompt somewhere for opening the full-height waveform view. Maybe after the user clicked to open the lower waveform graph, a prompt could pop up?

I will add that.

[JokerQyou]

Not sure I understand, you mean double clicking doesn't work? as in you have to wait a certain time between your clicks, or else it registers as one click only? Which OS are you on?

It worked but feels strange, and does not seem like a double click. I'm on macOS. Here's the steps:

• By default the lower waveform view is turned off.
• Single click on the "Show waveform" button, now the lower waveform view shows up.
• Single click on the "Show waveform" button again, now the video area turns into full-height waveform view.
• Single click on the "Show waveform" button again, and both waveform views turn off, video area reappears.

Clicking quickly twice would work, of course.

[JokerQyou]

I also find the lower waveform now behaves differently, and feels a lot slower. Previously in 3.45, the lower waveform view would generate new waveform whenever:

• Video is paused, mouse click on an area with no waveform.
• Video is playing, mouse click on an area with no waveform.
• Video is playing, the cursor (the "current time" cursor) moves to an area with no waveform, and I press "space" to pause the video.

But with 3.54, the waveform would only generate when:

• Video is paused, mouse click on an area with no waveform.

This makes editing with waveform much slower, as I have to constantly pause and click to see new waveforms. Is this change intentional?

Update

It seems to only occur for files with very long duration (40 minutes to 1 hour). I'll do more testing.