search

djfun / audio-visualizer-python

a little GUI tool to render visualization videos of audio files
Other
239 stars 57 forks source link

Switch to MLT Framework #27

Open dv-anomaly opened 7 years ago

dv-anomaly commented 7 years ago

We seem to have plateaued as far as performance goes utilizing Pillow-SIMD throughout the stack. After we get the next release out, we should consider switching to a more applicable framework / library for video production. MLT Framework seems to be at the top of that list, with major opensource video production projects using it. It is written in C with a C++ wrapper and has SWIG python bindings.

https://www.mltframework.org

djfun commented 7 years ago

what would be the benefits of using this framework? What parts would you want to change to use MLT?

dv-anomaly commented 7 years ago

Pretty much the entire rendering engine, which is pretty hacked together using PIL and threading. Just to composite a few static components in pillow it is a very expensive task. Switching to a toolkit designed for real-time video playback, layering, and encoding makes sense. It will have a huge performance benefit across the board, including redrawing changes in the preview window. It will also be a lot easier to incorporate real-time preview and scrubbing before export, etc.

It will also take all the extra work out of the blending options / filters we talked about incorporating. Because of the real-time aspect we could also integrate a number of producers and consumers such as SDI, NDI, Pulseaudio, and Jack for integration into live production environments. OBS already supports SDI cards, but NDI is on the roadmap as well. We get all that for free with the toolkit. Of course Tricaster and vMix already support NDI.

There's plenty of reasons why Kdenlive, Flowblade, OpenShot, and many others use it.

The only thing is, the documentation isn't the greatest thing in the world, nor completely up to date. But we can model after Flowblade and OpenShot which are written in Python.

rikai commented 7 years ago

I'm personally not a huge fan of the MLT framework, almost every application i've used (WebVfx, Shotcut, OpenShot, Kdenlive, Flowblade...) that uses the framework has had stability issues and other problems. There may be plenty of reasons that it's used, but there are also plenty of reasons why the applications don't see very wide usage despite being free alternatives to very expensive software.

If stability and minimal bugginess can be guaranteed, i'm all for using MLT, i've just got a bad taste from experiences with other applications that have used it in the past, so i would suggest some rather extensive testing of it before a full-fledged move for that reason alone.

dv-anomaly commented 7 years ago

Not sure that stability of the framework is directly related to the stability of a given application. Such a major change would obviously be tested before being a flat-out replacement. Same as the newgui branch, it should be labeled as a beta or RC on release until we get some feedback.

If you have a better library recommendation I'm all ears. There are also a number of broadcasters that use the library on their own internal solutions which do not get released to the public. I think the primary issue behind the stability is people getting it wrong because the documentation is so bad. It's something to consider, but I haven't found a better option.

rikai commented 7 years ago

Like i said, i'm perfectly happy using it as long as we can guarantee stability. I've just found the MLT framework to be the common thread between the applications and since many of the issues were around things MLT was involved with, it seemed like a logical jump. I'm just suggesting a modicum of caution, is all. 👍