[EDIT] Automatic tracks alignment

[prokoudine]

One useful feature that would be great to have is aligning several tracks based on similar/matching waveforms. E.g. when you record a video with both a built-in mic and a lavalier mic, you can align two audio tracks, then kill the worse-sounding built-in mic audio track.

Some technical info is available here:

http://web.archive.org/web/20130619023518/http://bemasc.net/wordpress/2011/07/26/an-auto-aligner-for-pitivi/

[prokoudine]

And some more info.

The actual commit to Pitivi:

https://gitlab.gnome.org/GNOME/pitivi/commit/aa684103541d2478d9d31390b181ea461fc5f94f

Another blog post:

https://fortintam.com/blog/2011/07/25/automated-multicamera-clip-syncing/

[itsmattkc]

This is a very interesting concept and I'm sure would be very helpful. I'll try to implement something similar within the next few weeks.

[alcomposer]

A few notes about automatic alignment:

• Audio needs to be able to move sub-video-frame, for second system to sync with video
• One should be able to select 2 clips any time and sync them together, possibly even by adding helper markers
• Syncing is not a final process, it is something that can continue throughout the edit.

A great example of sync done right is Pluraleyes, which takes the approach of allowing the user to feed back into the syncing process, and refine it to their taste.

It would be exciting if there was a way to build this feature into Olive in a way that is not exclusive to one tool or mode. So that at any time a user can go into the sync-sequence (which could be a regular sequence) and change sync. Allowing the user to decide how they build the project, will be very powerful.

[frink]

Note, this really needs to be done at the time of import. Since the audio of each clip can be edited and moved independent once it is on the timeline. That is probably a good thing to have them decoupled on the timeline. But it does mean that things like edit groups probably need to be implemented to do this right.

I really like the way Lightworks deals with these. One of the things that is most incredible about their solution is that it can also do a rough cut of camera switching for you. Their project within project approach has so many uses...

One of the things I'd love to see is a way to record multiple sources where there was live switching at the time of recording (like a muticam setup in OBS) and then see it imported into the NLE for fine tuning.

I'd also love to see more interoperability between audio and video programs. I've probably written 5 times tonight that in the pro-word DAWs and NLEs are both needed as tools of the trade and neither should take the place of the other...

[lutzray]

I've just discovered Olive searching for kdenlive scripting/plugin (non) existent capabilies. I'm cross-posting this on Patreon! Yes, I'm donating $$ without even downloading 0.2 to show you my commitment! I'm designing a cheaper and open source hardware (OSH) version of Tentacle Sync (see https://atomicsynchronator.blogspot.com) and I need to partner with a NLE so the workflow doesn't need batch processing before import. Will your program allow plugins for sound syncing? I don't want to fork Kdenlive just to 'sell' my gizmo. Heck, I can even give a hand with the code! But first, I'll order some cpp books 8-)

[Simran-B]

There will be no extension interface in the foreseeable future and audio is not a priority. Pull requests are welcome however.

[sobotka]

@lutzray Would you be interested in bringing your LTC thoughts to Olive, and integrating audio sync natively? There is already an issue for LTC support. Would be wise to push to that issue if relevant.

[lutzray]

For now it's all vapourware, except for the hardware, all is needed is pictured there (I got them today!) https://mamot.fr/system/media_attachments/files/105/284/117/584/119/349/original/a8a934df0716f0a3.jpeg (the gps module can be bought for much cheaper on aliexpress, 16USD)

in Olive when batch importing do you align the clips on their metadata time stamps, that would do the trick. Because I finally opted for this workflow/combo: hardware dongle during shoot + desktop program for timestamping before importing into NLE (much like Tentacle Sync?) Waveform analysis is too fluky, even Resolve can't do it flawlessly, see https://forum.blackmagicdesign.com/viewtopic.php?f=21&t=95947

on the other hand , I've read good testimonies about https://syncaila.com/

[lutzray]

Well, my Syncing Gizmo and its accompanying audio AUX format are taking shape, and they have a new home: https://hackaday.io/project/176196-atomic-synchronator so it isn't vapourware anymore!

[MarcoRavich]

Sorry for the dup request, anyway check out those resources that may help:

• https://github.com/protyposis/AudioAlign
• https://github.com/alopatindev/sync-audio-tracks
• https://github.com/benfmiller/audalign
• https://github.com/recastrodiaz/align-audio-recordings-julia

Last but not least: since this function it's not a priority for Olive, it would be great to have a standalone app (like PluralEyes or Syncalia, but open source) that exports into a Olive-compatible project format.

Hope that inspires !

[lutzray]

Well, it's not vapourware anymore! See this demo. I'm gonna eventually implement an OpenTimelineIO output so Olive users can get automated audio and multicam syncing into their workflow.

[MarcoRavich]

Well, it's not vapourware anymore! See this demo. I'm gonna eventually implement an OpenTimelineIO output so Olive users can get automated audio and multicam syncing into their workflow.

Very interesting, @lutzray ! It is a part of OTIO or standalone ? How can we add it to our Alignment/synch collection ?

And, of course, it would be great to establish some kind of collaboration with other similar project (such as @benfmiller's Audalign) in order to obtain better results.

For example: have you evaluate to manage the audio phase/polarity correction too ? Check out our related GH issue: https://github.com/benfmiller/audalign/issues/39

Hope that inspires !

[lutzray]

Very interesting, @lutzray !

yes and it's humming along, here's my most recent mastodon thread about it

It is a part of OTIO or standalone ?

it's a standalone CLI python program (isn't OTIO a format standard?)

How can we add it to our Alignment/synch collection ?

I'm a little hesitant to let my tool mentioned in an "official list", it's a pet project, amateurishly coded (test coverage is almost existent and for the last months, broken). Eh! I'm a physics teacher, be warned... Also, it must be stressed that's a hardware solution.

And, of course, it would be great to establish some kind of collaboration with other similar project (such as @ benfmiller's Audalign) in order to obtain better results.

I won't work on video synchronization using camera waveform analysis because it can't overcome the problem of sound delay propagation (1 millisecond per feet!)

have you evaluate to manage the audio phase/polarity correction too ?

What correction is needed? I must confess I'm ignorant of "on the ground" problems, have never done sound capture of any serious event, nor done any real mixing (I'm doing this for my future science teaching video clips). I approach the field with a physicist POV, so for me phase is a radian number between 0 and 2pi :-)

That pun said, I scrubbed through the video linked in [IDEA] Phase/Polarity adjust #39 ... My brain says: there will always be destructive interference occurrences when summing signals from point source recorded by point sensors if room acoustic doesn't do some averaging. This destructive summation can occur post recording (called mixing) or even during the acquisition itself, be it by a mic (sound waves) or an antennae (RF waves), see https://en.wikipedia.org/wiki/Multipath_propagation. Surely I'm missing something about this "phase problem".

[lutzray]

Well, it's not vapourware anymore! See this demo. I'm gonna eventually implement an OpenTimelineIO output so Olive users can get automated audio and multicam syncing into their workflow.

Very interesting, @lutzray ! It is a part of OTIO or standalone ? How can we add it to our Alignment/synch collection ?

Done (generating otio file with synced multicam) : https://youtu.be/pklTSTi7cqs

If at last someone would help me package a one file executable... I'm stuck (tried Nuitka, pyoxidizer, p2exe). I'm a python noob and more of a hardware guy. For now, distribution is with git and pip (+ ffmpeg and sox) . would be great to have a one click installer. I know, I'm off topic, this isn't about Olive, but as said on discord I'll code a OVE exporter soon.

[MarcoRavich]

Bump.

AudioAlign and Aurio finally updated (with prebuilt Windows binaries) after 5 years: check them out.

Hope that helps/inspires !