Movie Exporter Rewrite

[J5lx]

This came up in #671 a few weeks ago and I figured it deserved its own issue by now. I’m going to go into details, so this will be long; the tl;dr is that the current movie exporter based on the FFmpeg CLI is inelegant and inefficient and it would be better to use something based on an API such as the FFmpeg API.

Current state of affairs

The current exporter employs a mixed bag of Qt APIs, custom multimedia code and calls to the FFmpeg CLI to export a project in up to 5 (five!) passes, with the exact number depending mainly on the file format:

• if MP4: (re-)encode and save all sound clips as WAV files
• if MP4: mix sound clips and save as WAV file
• encode and save all frames as PNG files
• encode as H.264 and AAC and mux both into an MP4 file
• if MP4: re-encode as H.264 and AAC and mux both into the final MP4 output file
• if GIF: Generate palette
• if GIF: Encode and save as final GIF file

Needless to say, this procedure is far from ideal. It relies on heaps of temporary files (consuming a good amount of memory or disk space, depending on OS and setup), degrades quality due to using lossy codecs for intermediate results (using lossless codecs instead would make resource consumption much worse) and provides more surface area for bugs due to the custom multimedia code that is not even relevant to Pencil’s core functionality. Instead of relying on a CLI which was not conceived for programmatic use, it would be better to use an API. The most obvious choice is the one provided by FFmpeg since it is the de-facto standard for Open Source multimedia handling, it is available on all platforms targeted by Pencil and we are already using it indirectly right now.

New exporter based on FFmpeg API

As mentioned before I have already started experimenting with a new exporter using the FFmpeg API. My current progress can be found on my export-lav branch and nightly builds are available from https://j5lx.de/pencil-builds-j5lx. To enable the exporter, CONFIG+=EXPORT_LAV must be passed to qmake and it must know where to find the libraries.

• On Linux it looks for the pkg-config packages, so those need to be available. On Arch Linux they are in the ffmpeg package, on a modern Ubuntu they are in libavformat-dev, libavcodec-dev, libavutil-dev, libswresample-dev and libswscale-dev.
• On macOS it also looks for the pkg-config package. An easy way to obtain it is via Homebrew (i.e. brew install ffmpeg). When building with the homebrew package, the PKG_CONFIG_PATH environment variable must contain /usr/local/lib/pkgconfig.
• On Windows, download and extract the dev package from https://ffmpeg.zeranoe.com/builds/ and run qmake with LAV_DIR=\path\to\ffmpeg-x.x.x-winxx-dev. At runtime, the dlls from the shared package must be present.

Since the exporter is still very much work in progress, it will certainly contain bugs and there are a number of known issues, so this is where I am right now:

• [x] Muxing
  • [x] MP4
  • [ ] AVI
  • [ ] GIF
• [x] Encoding
  • [x] H.264
  • [x] AAC
  • [ ] MPEG-4 part 2
  • [ ] MP3
  • [ ] GIF
  • [ ] GIF palette generation (?)
• [x] Frame painting
• [ ] Source audio decoding and resampling
• [ ] Audio mixing
• [ ] Make sure nightly builds only contain one copy of the libraries used by both Pencil and the FFmpeg binary
• [x] Perfect progress

While I’m at it, the progress meter is probably the part where the change is most visible to the user since it is already more accurate, applying to the process as a whole rather than just one out of five passes, and also noticeably faster since there are no more unnecessary intermediate encodes and saves.

Concerns

Improvements aside, the main drawback of an API-based exporter is that it introduces an additional build-time dependency. This is practically a non-issue on Linux where the required files such as libraries and headers are usually readily available and can be obtained in one operation, but it complicates the build procedures on macOS and especially on Windows by some degree, as seen in the build instructions above.

@chchwy noted previously that we are basically forced to use pre-built binaries provided by others or undergo the possibly slightly tricky procedure of compiling FFmpeg oneself, but I’d like to point out that the same is true for Qt (is anyone here who has actually compiled Qt by themselves?). The main difference is that Qt has a fancy graphical installer and FFmpeg doesn’t.

@chchwy also suggested an approach with multiple co-existing implementations using the “native” multimedia framework of each platform, for instance FFmpeg on Linux and AVFoundation on macOS. While certainly doable, this approach requires a maintainer for each implementation. Personally I would work mostly on the FFmpeg one; But while I’m also open to look into a Windows implementation if this approach is chosen it would clearly be a second-class citizen to me which is suboptimal given the great amount of Windows users.

At the very least, though, compilation of the API-based exporter can be made optional; in fact it already is: The CONFIG+=EXPORT_LAV option aside, it will be built if the libraries can be found and it will simply be disabled if not, a behaviour also found in many other software packages. Since most potential contributors will not need to work on the exporter anyway, this will allow them to work on their relevant pieces of code just fine.

All in all I believe that the use of an API-based exporter is generally desired. What we actually need to work out in this issue is how to deal with the dependency issue.

[chchwy]

Thanks @J5lx

First, I'm not an opponent of the API-based exporter. Actually, it's great to see Pencil2D has a built-in exporter.

The idea of using an API-based exporter was in my mind but ended up not doing it because of lacking time. Since I have done a couple of API-based exporters in my daytime job, mostly on Windows & iOS. It usually took me 2-3 weeks full-time work hours to make it bug-free and really usable, probably will be shorter in pencil. But comparing to a CLI-based exporter it cost me only 2 days.

So now you guys jump out and say let's make a new exporter, I 100% agree. Actually that's why I extracted all movie exporting code to a single class MovieExporter. Now we can easily extend it by using one abstract interface and having 2 implementations like MovieExporterCLI & MovieExporterFFmpeg. And I can add MovieExporterMediaFoundation for Windows and MovieExporterAVFoundation for Mac in the future. All exporters are easily switchable and leave all the other part of code untouched.

In my opinion, easy to compile is important to an open source project. Qt is precompiled, but it is unavoidable because Pencil2d is built on Qt. As a windows developer, at least Qt provides several choices VS2013, VS2015 and MinGW officially. I compiled Qt before, tried to compiled ffmpeg but failed, and I don't really want to do it again. So personally I wish Pencil2D could be simple on either user side and developer side.

Anyway, great works.

[J5lx]

First of all, I’m sorry if I you feel that I labelled you as an opponent, that was not my intention. Your concerns and suggestions that I mentioned are completely valid, I’m just not 100% happy with the solution since it creates an overhead for each platform.

Regarding the different exporters, it’s good to know that there is someone already familiar with the other relevant APIs. As for choosing an exporter, I’d generally like that choice to be at least not bound to the choice of platform, so that it is possible to use the FFmpeg API exporter on Windows and macOS as well, even if it is not the default. Would that be okay with you?

One more thing that I came to wonder about is format support. While WAV, MP3 and MP4 are probably widely available, I’m less sure about GIF (for encoding at least) and I think that some potential formats for the future (like WebM [#288] or FLAC [which I’d personally like to have for importing lossless audio at some point]) could be outright problematic. Some time ago Microsoft finally added FLAC support to Windows 10, but what about previous versions of Windows? I don’t even have a clue about WebM support. On the other hand, support for many formats and codecs is a strength of FFmpeg and I think it’s one of the reasons why the Libav fork died. If we do end up with such disparities in format support, how would we handle that? Use FFmpeg in official builds to mitigate it? Point users to projects like oggdsf or XiphQT (defunct) that add support for such formats to the OS multimedia layer? Limit ourselves to the lowest common denominator and don’t even consider support for any format that is missing from just one backend? Something else entirely?

Lastly, I’m also interested in reviving the currently defunct movie importer at some point as well as adding a sound exporter. The latter should be trivial since it will share most of its code with the movie exporter but how about the former if we have different multimedia backends for different platforms?

[J5lx]

BTW I haven’t thought about it much but maybe we can also use an approach where we load the FFmpeg libraries dynamically at runtime and bundle necessary headers or something. It would certainly mean some extra effort, but it would allow Pencil to be built with the FFmpeg API exporter even when FFmpeg is not available at build time, and it would simply pick up the libraries when it needs them similar to how current master only needs the FFmpeg binary for exporting video files. Thoughts?

[chchwy]

Yes, surely we can start with an FFmpeg exporter as default on all platforms, and then implement other exporters later.

The multimedia formats issue is a mess, there are 3 points we need to discuss:

1. what formats we want to support on importing?
2. what formats of files we want to keep in a pencil project?
3. what formats could pencil2d export?

A primary concern is what formats QMediaPlayer() can play. It makes no sense to import a sound file which can be heard only after exporting. If you look at the QMediaPlayer's document, the support formats vary between platforms. It uses native OS lib as the backend, for example, DirectShow on Windows, GStreamer on Linux etc. If we want to add an uncommon format to pencil2d's importing list, it will be user's responsibility to install a proper codec.

[J5lx]

Here is my take on formats (for the most part assuming an ideal world where all formats are available):

1. Import:
   • WAV - Lossless audio format, usually uncompressed so it can get huge. De-facto standard in audio production (AFAICT). Currently already offered by Pencil2D.
   • FLAC - Lossless audio format featuring good compression, somewhat slow to encode but fast to decode. De-facto standard for lossless end-user audio.
   • MP3 - De-facto standard for lossy audio, insanely widespread and currently already offered by Pencil2D.
2. Storage
   • Lossless audio: WAV or FLAC. WAV has more widespread support, but FLAC has much smaller filesizes
   • Lossy audio: To avoid further reduction in quality, lossy audio should be stored either in its original codec (i.e. MP3) or in a lossless one
3. Export
   • Possibly MP4 (w/ H.264 + AAC) - currently already offered by Pencil2D
   • Possibly AVI (w/ MPEG-4 part 2 + MP3) - currently already offered by Pencil2D
   • WebM (w/ VPx + Vorbis/Opus) - Open format/codec with good support and good compression, ~already requested in #288~ already offered by Pencil2D with 82a4852e068737bb1a2adff14c291f396b21b8c9
   • GIF - Currently already offered by Pencil2D, also Pencil kind of leans itself toward the kind of web animation that GIF is often used for

Notes:

• Some might argue that since Pencil2D is free software it should also support a free format for lossy audio (e.g. Ogg w/ Vorbis/Opus) Edit: Technically, from April 23, 2017, MP3 is sorta free too.
• I’m not entirely convinced whether we really need both AVI and MP4. One would suffice IMO, preferably MP4.
• ~Regarding WebM codecs, I’m currently not sure about the (dis)advantages of VP8 vs VP9 and Vorbis vs. Opus~ VP9+Opus it is!

As for playback in Qt Multimedia, I believe the safe route would be sticking to WAV and MP3 for storage as we currently do. If we do need to play formats that aren’t supported by the system backend, we could do the decoding ourselves (which we’ll probably need to do anyway in the exporters) and send the raw data to QAudioOutput. Alternatively we might also be able to provide our own multimedia backend, but that might be more work.

[Jose-Moreno]

@J5lx I agree these should serve as a the base file formats, regarding OGG though, I've worked on Game Dev, and everytime the Game Engine guy asked for OGG files as opposed to MP3 or WAV. This is particularly true for Mobile development. So I don't know if OGG should also be included, not only because open source, but also because it's very widespread. On the video side MKV comes to mind (as a pair to OGG Vorbis).

Regarding AVI, it's almost requisite for editorial procedures. Unless you're a freelancer that edits on After Effects where you can use MP4, but Sony Vegas, Premiere and other dedicated video editors have problems importing MP4.

[J5lx]

Ah, now that you mention it I think Ogg is popular for games because it’s lossy (i.e. very small filesizes) and works on all platforms without licensing issues. If we decide to support it we’d still need to determine the codec to use, though (Vorbis or Opus).

As for good old I-can-do-everything Matroska, I’m not sure if it provides that much value over the other formats in our case. The way I see it, its main advantages are:

• It’s free
• It can contain pretty much any codec
• It can contain multiple titles (which makes it attractive for ripping video discs)

However in Pencil2D we don’t need more than one title per file and we don’t need many different codecs (in fact the codecs are “hardcoded” for each format), and that’s why I suggested WebM which is basically a specialised version of Matroska that is also free and very widespread, especially for web videos (go ask YouTube).

As for AVI, I wasn’t aware that it’s such a big deal in video production. I always thought raw image sequences were the most common solution for intermediate results there. But I guess it does make sense to include it after all.

[scribblemaniac]

I think we should look into using gifski with this exporter for GIF exporting. It is Rust-based but has a c api that looks good for our purposes. After some testing I can say that it definitely produces noticeably better results than even the FFmpeg palette generation technique which we are currently using. It's difficult to even tell that this example of their's is a gif!

[MrStevns]

@scribblemaniac Although it's nice, do we need it and will people notice it? high quality gifs are nice in some cases but standard gifs are already hugely inefficient. If it's purely for exporting in higher quality, then i'm not sure about the addition of it. From what I understand the exporting may also become slower because it may have to video->png->gif.

The way I see it: pros:

• better quality

cons:

• even larger gif outcome
• possibly slower export time
• Another 3rd-party dependency that may or may not be kept active.

[J5lx]

I pretty much agree with @CandyFace here. Of course the example you linked looks pretty damn good, but we have to keep in mind that Pencil isn’t typically used to encode live action footage into GIFs, and I wonder if the difference is still as noticeable when it comes to the kind of 2D animation typically made with Pencil. The question is whether the gain in quality for that type of content justifies the extra hassle of including and integrating yet another 3rd-party library (also in regard to difficulty; the FFmpeg exporter turned out to be trickier than I first thought as soon as I started trying to get audio and video into the same file [hence why it’s taking so long]).

[J5lx]

And besides, as it’s Rust-based, it will probably require a Rust compiler at buildtime… which would likely make including it with Pencil even trickier than integrating the FFmpeg sources into our build process…

[scribblemaniac]

Believe me, I'm on the same page as the rest of you in terms of adding dependencies; I was very happy to see quazip and zlib get replaced. However I have come to believe that it could be worth it in this case. My understanding is that one of the reasons we are creating a new exporter is to support multiple backends, so we shouldn't dismiss the idea merely because it requires another dependency.

I rendered a few more gifs with gifski and ffmpeg since initially suggesting this, and I am becoming increasingly convinced that this should be done. First here are some points to clear up based on some comments from a discussion we had on Discord shortly after this was first suggested:

• File size between gifski gifs and ffmpeg gifs is very similar. The improved quality comes from better use of the data, not more data. Gifs have a set limit on palette size, but gifski utilizes that palette better with better quantization (a pngquant algorithm I think) and a "shifting palette"
• GIFs will be inefficient regardless of whether we use FFmpeg, gifski, or anything else. Their quality will also be less than ideal no matter what we do. This change would be purely for higher quality, but should that not be a goal of ours? There are many ways to make export faster at the cost of quality and file size, but we are not pursuing those options. We could for example, ditch the palette generation step, which would simplify and speed up our current gif export considerably, but the output looks terrible compared to what we have now. And I'm telling you, the new output looks terrible next to the gifski output. Yes there's a balance to be had here, but there is not too much of a cost to doing this, at least not for the user (yes integrating this could be a pain, more on that later).
• Gifski is slower, as it does more processing, but it does not require the png intermediate format. The API supports sending rbga arrays, which I think you can get from QImage.bits() faster than even the video->bmp (in-memory)->gif workflow that we will be using for ffmpeg (at least until this issue is closed). This means that it's not horribly slow, and it should have similar memory consumption to ffmpeg.
• Sometimes this isn't useful. In certain situations, particularly for purely greyscale animations, ffmpeg produces output just as good as gifski. This makes sense because it takes 256 colors to accurately represent a 8-bit greyscale image, and the gif palette happens to hold 256 colors!
• But most of the time it is useful. When there are many colors (even flat shading takes up more than you think thanks to anti-aliasing), more frames, live-action elements, photos, 3d renders, or gradients, gifski will really start to shine.

I really hate linking. Most of the time I compile my programs, so I don't have a lot of experience with compiling programs so they can be distributed. Is there perhaps some way where we can use the gifski backend only if it is installed on the user's computer? You can obviously do this by changing the build settings if you're compiling it, but is there some way to do load it at runtime (this perhaps)? I wouldn't be against having "too complicated to include" backends as an option for the user to install themselves if they need it.

Finally, as I said, I've done some examples, with actual animations. They are quite long, and thus are too large to attach here. I will try to find some other way to share them. When you see the ffmpeg and the gifski renders side by side, you'll start to see where I'm coming from here.

[J5lx]

Hmm seems like fair arguments, especially regarding efficiency and such, though I am interested in seeing the stuff you converted. As for dynamic loading, sure, it should be possible, but using a system copy of gifski is only really viable on Linux where it’s normal for libraries to be installed system-wide. On Windows and OSX (AFAIK) third party programs are meant to ship their own libraries, so we’d absolutely have to include our own copy of gifski if we want people on those systems to use it. Some time ago I also considered loading FFmpeg dynamically but since then I’ve come to believe that it creates more trouble than what it’s worth. After all, our deployment mechanisms will be building with gifski available anyway, no matter how much setup is needed, so the only ones that would profit from not depending on those libraries would be developers and users who chose to compile the program themselves. As for the former, they can just disable the exporters altogether (since they are a non-essential part of the program) and then start hacking just fine (unless they want to work on the exporters, in which case they’ll want development files of the libraries anyway). As for the latter, if they are willing to compile a program that they could just download in a ready-to-use package, I think it’s fair to expect them to put with one or two libraries as well, and if they don’t want to go through that procedure, they too can just disable them.

That turned out much longer than I imagined, tl;dr is I think we should go for build-time linking and let those who don‘t want to obtain development files for the libraries disable the respective exporters.

Also, sending RGBA arrays sounds good, but, as a heads-up, make sure to pay attention to endianness. QImage’s internal format depends on endianness while FFmpeg’s does not, and it took me some fiddling to realise that and account for it. Just in case the situation is similar for gifski.

Lastly, maybe we should consider splitting this generic movie exporter rewrite issue into separate issues for FFmpeg and gifski, it’s getting pretty large by now.

[chchwy]

A good way to do this is by making "Plugins". http://doc.qt.io/qt-5/plugins-howto.html#the-low-level-api-extending-qt-applications

1. First, define an ExportVideo interface.
2. The main application communicates with exporter via the interface, it doesn't care what it actually is.
3. Each export plugin follows the ExportVideo interface and has its own backend implementation.
4. The main application will load the plugin at runtime, so it's ok to run the main application without any plugins.
5. The only thing shared between the main application and plugins is one header file which contains the interface definition.
6. The plugins can be compiled separately, no need to be part of the main application. It can even be in another repo. So basically we don't have dependency issues. It's plugin's dependencies.

But I have only done Qt plugins in Windows. Not sure it is doable in Linux.

[J5lx]

That‘s a pretty cool idea! In general plugins shouldn’t be a problem on Linux, Qt itself also has some like the xcb plugin or the gstreamer one.

[MrStevns]

I'm still not convinced. The amount of work which is required to get slightly better and in our case barely noticeable different gif quality doesn't seem worth it to me but... if it's optional via plugins of some sort then I'm fine but what will the average user think of our removal of gif exporting when they've always had it and now requires a plugin?

[J5lx]

@CandyFace The average user will use the binary builds we provide, thanks to Travis and AppVeyor we can include gifski there just fine. The “optional” part comes into play mainly when compiling the program oneself and for development, so it is possible to compile the core program without the exporters if setting up the development environment to provide the required libraries is too much trouble.

[Jose-Moreno]

In that sense I agree with CandyFace. Believe it or not Pencil2D is one of the few animation programs that can export to GIF. Even if everyone use FFMPEG, they simply don't export to GIF, which is silly.

However I do like very much the idea of plugins to let the core program be as simple and concise as needed and get new functionality like that via plugins. This could lead the way for people being allowed to make their own plugins to extend Pencil2D as they see fit which has been discussed in the past.

With that said however I strongly feel we should at least leave the "basic" GIF export, and if people want an enhanced GIF export they can use the plugin if they so feel inclined. Although Jakob brings a fair point, since the average user wouldn't notice the difference if we "ship" the plugins with the release version of P2D, but then that means the nightly builds wouldn't necessarily bring the plugin dependencies, right?

[MrStevns]

Ah I misunderstood something then, I thought the plugins would be for people to get externally but if it's all included in the releases, then I have no problem with this.

[J5lx]

To be extra clear here: No matter how we choose to tackle this, absolutely nothing will change for the end user except that their GIFs will look fancier (as per @scribblemaniac's findings). Every single user that downloads a released version of that hypothetical variant of the program will be able to export GIFs no matter what. Nobody is trying to limit the GIF export functionality to a small circle of users who are willing to go an extra mile, this is about improving the existing functionality without creating an unnecessary burden for casual developers or potential new contributors.

When we're talking about plugins here we're not (yet) talking about creating a general-purpose plugin mechanism that allows to extend the program freely (as mentioned in #540), we’re only talking about using the plugin infrastructure provided by Qt to make our export mechanism more flexible.

Edit: Sorry if this sounded a little rough, but I just wanted to clear this up once and for all.

[Jose-Moreno]

@J5lx Crystal clear explanation. Thanks for taking the time to clarify. :smile:

[Jose-Moreno]

@J5lx @scribblemaniac Sorry to bother, but I have to ask just to get a clear update, what's the actual status on this issue?

p.s. Also no matroshka export (MKV)? jk :stuck_out_tongue_closed_eyes:

[scribblemaniac]

I am done with my improvements to the existing movie exporter. There may be an issue with memory consumption (on windows?) which I may look into further, but otherwise it will probably remain as-is until the rewrite is ready. @J5lx will have to cover the progress of the rewrite because I am not directly involved in that.

I am still in favor of a gifski extension/plugin, but many things will have to happen before that is possible. It does not provide a significant benefit for most of our users over the current approach, so is not a high priority for me anyway.