ffmpeg: Discussion of distribution, performance and patented libraries

[thess]

To anyone interested...

Currently certain package builds produced by the buildbots always include PATENTED functionality. While this is OK for individual non-free libraries, it is not always OK for package distributions like ffmpeg to automatically include these libraries. The following are a couple points I'd like us to collectively consider:

• Default ffmpeg packages produced by the buildbots should not include PATENTED libs.
• Should we have both PATENTED and GPL versions (more buildbot work)
• Should the only configuration to have PATENTED functionality be -custom (which is not built by default)

On the performance side, it is clear that there is opportunity to build configurations with platform specific optimizations (See #3830 discussion). However, I am not in favor of doing this based solely on CPU_TYPE. For example: An X86 system with limited memory (ALIX w/ Geode LX) or no SSE would still be a candidate for a enabling a small configuration option. Any proposal for new build options based on CPU and/or other architecture dependencies will need to be built and run tested thoroughly.

My plan, for now, is to keep the current ffmpeg package options and wait to see what we decide going forwared. lede-17.01 will not be affected by this discussion and remain as is.

[diizzyy]

Is Geode really that space constrained that ~2Mbyte matters? We're sadly never going to be able to the all the things (tm) before committing. That said, I really think that arm, x86 pretty much covers 90+% where performance speedups are wanted/doesn't impose any showstoppers as far as size goes (within reasonable limits). PPC could get some love too in that regard but I don't really think it's worth the work due to very limited amount of devices and users in comparison. I honestly think we should do as much as possible optimization on x86 and arm, also as we're using recent software upstream would most likely be interested in fixing bugs too so I don't see that as an issue.

[antonlacon]

RE: Performance

I would like a CPU_FEATURES that held something like the flags from /proc/cpuinfo to allow enabling simd instructions. The current method of depending on whether the cpu is hard or soft float isn't great, as seen by the powerpc issue with the cpu having hardfloat, but not having altivec. No idea how pressed in stone machine profiles are for this kind of extension.

Alternatively, a 'BUILD_LARGE' flag might be appropriate. It would enable runtime cpu detection and all of the simd instructions. I haven't looked into either of these for implementation.

RE: Patents

I'm in favor of dropping the patent flag for the package. The ffmpeg project specifically makes no comment on what code may be affected by a software patent. Should OpenWrt be making a more specific comment? Something in the description that the code should be reviewed for patent encumbrance prior to distribution? Commercial users should be using their own lawyers to review their needs.

RE: GPL

I don't know what functionality we're using out of the GPL code to comment.

[antonlacon]

WIP code here: https://github.com/antonlacon/packages/commits/ffmpeg-3.2.x

Not yet ready for merge, and it has some unrelated changes to what's discussed here (that may or may not be taken), but this holds a new menu configuration for libffmpeg-full with size / license choices. Some stuff to still wire up for libffmpeg-full, and then to bring the changes over to libffmpeg-custom too.

[diizzyy]

Cool, however I'm not sure we really need a lot versions of ffmpeg libs. As much as I think we should make it as useful as possible I think we really need to consider what's actually used. That said, libshine should only be used on non fpu devices as it has inferior quality compared to LAME. This should be set depending on there's a fpu available or not.

ffmpeg-audio-dec-barebone: mp3, opus, flac, aac, (ALAC, DSD should be considered if they don't add too much bloat), I don't see a need for vorbis to be included

ffmpeg-audio: mp3 lame/libshine (fpu vs non fpu), opus, flac (libflac needed?), alac, aac (fdk), vorbis/tremor (fpu vs non fpu), dsd

ffmpeg-audiovideo-light: (minidlna etc) mp2 (dec only), mp3 lame/libshine, aac (fdk), opus, flac (dec only), ogg (vorbis/tremor), dts (dec only), TrueHD (dec only), AC3 (dec only), DSD, DCA(?), H.264 (dec only), HVEC (dec only), MJPEG, MPEG-2 (video, dec only), MPEG-4 (video, dec only), VC-1 (video, dec only), VP9 (video, dec only)

ffmpeg-full: ext libs from ffmpeg-audiovideo-light and full decoding support enabled.

..and use libswresample in all packages (it's about 70-80kbyte).

I still think we should hardset "large" lib(s) for x86 and arm to create a better user experience. Also, platform optimization should always be enabled unless it increases size a lot.

[antonlacon]

fdk-aac requires nonfree being set and that prohibits redistribution. See ./configure --help or the license line after configure completes. End users must build fdk-aac support themselves.

Runtime cpu detect being enabled across the board will cause breakage. My guess is that ffmpeg doesn't expect the software to be used on cpus with such limited features. An example is the ppc hfp issue regarding altivec usage. All the --disable lines were put in to get back to a non-simd path as that had best results on working on the widest variety of setups. I could see it being defaulted for some architectures, but foresee a need for a blacklist too.

I'm concerned with the proposed libraries. Something like DLNA streaming has fairly specific types of media that it can stream. The extent of its need for libffmpeg is media identification. Encoding, SIMD usage, and swresample would be for nothing. Increasing library sizes pushes out devices, or forces them to use extroot.

The proposed libraries tend to focus on adding encode. If most machines capable of encoding at a reasonable speed have large storage sizes, why wouldn't libffmpeg-full meet their needs?

[diizzyy]

Guess I should've explain myself more thoroughly...

The current optimization is pretty good, it does enable for instance neon and/or vfp on platforms where it's available. I'm not sure we build for anything that uses altivec out of the box? Go by the lowest common denominator in general is the only way to go here. We could be a bit more reasonable when it comes to x86_64 however. I'm not sure but --disable-armv* should probably be removed in platforms were it helps performance.

Regarding the proposed libraries, I actually had that in mind and other purposes (hence light) such as streaming or doing transcoding (youtube-dl), handling webcam footage (mjpeg). I'm only suggesting that mp3, opus and ogg gets encoding support. I do see an advantage about having FDK in but if it's not possible I guess that's how it is... :-/

Also, having a resample lib that isn't horrible is not a bad idea and crucial of you want to lets say stream audio.

http://downloads.lede-project.org/releases/17.01.0-rc1/packages/

As for as library size goes I honestly don't think the smaller libs are going to see much of a difference. I probably need to make some proof of concept. As for the platforms where I suggest that size < performance can you actually point me to where it is not valid. Most ARM devices comes with at least 64Mbyte (usually 128Mbyte or more) storage and I doubt x86/x86-64 is limited more than that.

[antonlacon]

The current mask behind the soft-float likely protects featureless cpu's from tripping up ffmpeg's detection of SIMD abilities. That shouldn't be changed lightly. I don't know why runtime detection is turned off for arm with hard-float to weigh in on it. Issue #3320 is where the ppc + altivec issue came up. I know there was a similar issue with mips at one point.

The disable code for the x86 instructions can probably be dropped from the soft float path, as I believe every supported x86 platform has an fpu.

If you would like to write an additional library variant, I'm open to it. I suggest extending custom first to get an idea of the resulting lib's size.

From what I remember, Xscale is an arm platform where size is important. For x86, the test case will probably be Geode. For x86_64, I don't expect size issues.

[diizzyy]

The only arch I know of that uses Xscale and that's still in use is Kirkwood, are there others? About all those devices have at least 128Mbyte flash or storage on USB.

[thess]

All IXP4xx devices are Xscale. Most notable are the NSLU2, IoMega NAS100 and Actiontec MI424WR among others (8MB flash). These are still very viable platforms and I would hate to obsolete them from support due to neglect.

[thess]

Package configs -- here's some more info. In general I do not think we need to change the current set of packages unless we want to provide pre-compiled (L)GPL vs. non-free versions of ffmpeg-full. Adding fdk-aac to the non-free option would be acceptable since it will be clearly identified as such.

• ffmpeg-audio-dec was constructed primarily for support of MPD and mocp on a limited resource system. Encoding and/or trans-coding was not a requirement. It is also used to provide WMA support to squeezelite. I do not want to change this.

• ffmpeg-mini was constructed primarily to support minidlna. It is significantly smaller than a -full config. I'll let the folks interested in minidlna decide what is the optimal config for this package.

• ffmpeg-full currently contains all the pre-configured GPL as well as non-free codecs as the BUILD_PATENTED option is the default for our builds. A custom build of ffmpeg-full may be different from the default depending on other package dependency selections.

• ffmpeg-custom is not normally built for distribution and its configuration can be as complex as we want it to be.

Proposal: I would like to remove all the non-free codecs from ffmpeg-full and include these codecs (currently lame, x264 and fdk-aac) in a new ffmpeg-nonfree package. I'm going to hold my nose and close my eyes with respect to the AVC/H.264 licensing as it relates to us. (http://www.librevideo.org/blog/2010/06/14/mpeg-la-answers-some-questions-about-avch-264-licensing/)

My next posting will address size & performance issues since those will not impact our packaging.

[thess]

As far as the various size/performance trade-offs go, which may be determined by belief of the usage and/or availability of the myriad of hardware platforms we support, I'm in favor of the IETF maxim (roughly paraphrased)...

"We reject: kings, presidents, and voting. We believe in: rough consensus and running code."

With this in mind, I'd like to see some proposals as tested pull-requests (size data can be empirical). We can then push them through all the 36 targets we build and hopefully some number of them will be tested and provide feedback. Anything else is just speculation and opinion.

[diizzyy]

About package configs... I never suggested that we added encoding to audio-dec, I merely suggested that we should "just" support formats that you'll actually come across in ~99.5% of all times. Keep WMA in if you want however I really think that this can go being 2017.

ffmpeg-audiovideo-light is a suggestion having minidlna in mind and "general" usage like youtube-dl etc. I admit that I missed adding that only mp3, aac, opus should have encoders in this package (streaming etc), it's not like embedded hardware will be able to cope with encoding video etc (ARM, MIPS etc). If there's a need we can have a profile called omitting these but I don't think we're doing anyone a favor by having these excluded by default.

ffmpeg-full: All the things, I don't really think we need to pull in external video encoding libs as even most x86 platforms (APU2 etc) won't do video encoding at any reasonable speed.

ffmpeg-custom should at least be renamed and/or even removed, I don't see what it would cover compared to the packages above.

Regarding Xscale: I doubt we'll see (m)any devices running a recent version of OpenWRT/LEDE and ffmpeg given the rather low performance you get on such hardware with "new" software and ARMv5 is pretty much deprecated in any active development dealing with ARM as far as I can tell. Kirkwood also uses Xscale binaries and I'm going to do a qualified guess that it's a larger userbase than legacy Xscale devices and we should optimize for Kirkwood even if it potentially breaks the ability to place it in flash by default (most have external storage anyway) on legacy Xscale devices. I doubt we'll turn off NSLU2 (it's about ~10y old hardware) etc users by having them installing on extroot.

[antonlacon]

Please elaborate on your complaint with libffmpeg-custom. Why should it be renamed? What does libffmpeg-custom mean to you that makes it ambiguous? It is a custom build of libffmpeg. The intent behind it was to serve as a dev-tool, and for users if a variant didn't already meet their need. I use it somewhat regularly.

[thess]

Updated ffmpeg revision, new configs, etc. See edc982430cc93144b0e6edb9fc49f04275eb69ed