search

media-kit / media-kit

A cross-platform video player & audio player for Flutter & Dart.
https://github.com/media-kit/media-kit
MIT License
1.09k stars 175 forks source link

[Enhancement] custom libmpv shared libraries #20

Closed alexmercerind closed 1 year ago

alexmercerind commented 1 year ago

We need own libmpv builds for:

With the highest priority being first:

Windows [Done]

media_kit_core_audio bundles audio-only libmpv-2.dll on Windows. media_kit_core_video bundles minimal video+audio libmpv-2.dll on Windows.

Both are LGPL compliant & allow commercial usage.

References:

Linux [Done]

package:media_kit doesn't bundle anything related to libmpv / FFmpeg etc. etc. It is not feasible either.

alexmercerind commented 1 year ago

Marking as "help wanted".

Quackdoc commented 1 year ago

I was actually looking into checking out the viability of an alternative to MABS based on pkgbuilds (for msys). awhile back. I can look back into that since github has an MSYS CI. that should make it plenty easy to add and remove various codecs. since MPV relies on FFmpeg for codec support, and is where most of the lgpl vs gpl vs whatever issues will come in.

MABS can also be used once configured to also do this I believe

alexmercerind commented 1 year ago

I did experiment a bit & got some results. Maybe the size can be reduced further:

Artifacts: https://github.com/alexmercerind/mpv-win32-build/releases

alexmercerind commented 1 year ago

I think more inspiration can be taken from this: https://github.com/myfreeer/mpv-build-lite. However, I removed everything that was GPL.

Quackdoc commented 1 year ago

However, I removed everything that was GPL.

however despite whats below, a method for bundling our own mpv on windows would be quite helpful, since it would allow application developers to use their own codecs that wont be in mainline ffmpeg (IE. a fair number of encoding enthusiasts I know are quite into the new VVC codec which is not in mainline ffmpeg)

I'm not sure if it might be worth having GPL + LGPL builds of mpv, there are a few features we loose, notable ones listed below, some of which might be quite nice to support directly. but im not sure they really offer all that much, also I think if possible we would want to at least have a minimal build with only common codecs, and a maximal build, which would enable a lot more codecs,

and just an single build for audio, I do already have ffmpeg build config for making very minimal audio only builds, you can see the config here. not sure what would need tweaked to get this working with mpv, but it might cut the size down considerably.

having separate LGPL and GPL builds for audio is completely unnecessary as IIRC the features that GPL offers are pretty much NIL for audio, just a couple streaming protocols IIRC, in comparison LGPL allows the usage of fdk-aac which is necessary for xHE-AAC audio.

among some of the stuff we loose by going lgpl are

As far as I know, these are the only features that would be missing that most people would actually care about, im not sure how audio is handled so I left oss AO in the list.

we also lose some ffmpeg filters that aren't listed here, unfortunately I cannot remeber the list of them, but I dont think too many of them are all that useful

EDIT: TLDR, GPL has some nice features but IMO doesn't offer much and the ability to provide our own mpv builds, even on windows, could sidestep any potential issues there.

alexmercerind commented 1 year ago

Being able to use package:media_kit in proprietary and/or commercial projects is very important.

Most of the Flutter applications in the market are going to be proprietary and/or commercial. With that being said, I'm not going to include any GPL libraries. If this was an end-user software, GPL licensing could've still been fine, but as a library it's completely not. In-fact, most of the applications, which will utilize the services from project's current sponsor are also going to be proprietary (a generalization; not a must). Thus, bundling LGPL-compliant version of libmpv is necessary.

It's not like I'm against libre software etc. I author/maintain a fair amount of open-source code myself. Rest, dependent projects are free to change libmpv themselves for matching their requirements (enabling/disabling some features). I i.e. package:media_kit is not statically linking anything (as required by LGPL).

DVD/CD support is redundant. I've also disabled Blu-ray etc. I agree that rubberband is quite good thing to have. However, that's certainly not something I cannot have in a library. And commercial licensing is quite expensive.

x11 VO is also not required, we are going to be using GL rendering API (as on Windows). Secondly, package:media_kit does not bundle/distribute any shared libraries on Linux. It's all up-to end-users, what they install. Most likely, they will install from some package manager, which will already have a "maxed out" version of libmpv/mpv.

I will document things, for now you can refer to repository I shared above.

NOTE:

Quackdoc commented 1 year ago

Only lgpl sounds good. next one, in regards to size, there are a lot of features MPV has that will be useful to some and not others, for instance libplacebo, libplacebo (also known as gpu-next) can still be utilized directly in libmpv via vf= filter, this is something that might be critical to a specific subset of apps, but irrelevant to others. (one features is proper dolby video support). and of course there are a large variety of codecs that for most applications are useless. but critical for others, again with a "video player" analogy.

so with this being said, how did you want to go about making minimal builds for bundling? is there a criteria for features that should be included? or would it be preferred to get every viable feature that would actually be useful in a flutter app?

maintaining seperate builds (minimal and maximal) isn't too hard, but im not sure how integration would be, as far as I am aware, flutter doesn't have an "optional features" that could be exposed in pubspec. currently, ill finish tommorow hopefully, I have what I think is a good base to build upon, given the repos you linked, finding what features to include though is a seperate story.

alexmercerind commented 1 year ago

Hmm... Maybe package:media_kit_core_video must be split:

By this, dependents will be able to choose whether to bundle the shared libraries for that platform or not. Because chances are that they might use package:media_kit on some platforms & some other package on other platforms (& then we don't want to bundle libmpv for that platform).

Secondly, they'll get ability to decide what version of shared library to bundle. I can offer multiple packages e.g. package:media_kit_windows_video_libs_extended in addition to package:media_kit_windows_video_libs to offer GPL modules.

How does that sound?

EDIT: I'm open to recommendations for names BTW.
Quackdoc commented 1 year ago

that sounds like a pretty flexible solution, sounds good to me,

alexmercerind commented 1 year ago

So with this being said, how did you want to go about making minimal builds for bundling? Is there a criteria for features that should be included? Or would it be preferred to get every viable feature that would actually be useful in a flutter app?

Informally speaking, let's take Google Chrome as an example. A good web browser, can also play most media content on the internet. Only bundling that much. Another example can be default Windows Media Player. It plays most things unless we go like... really specialized. A Flutter application (generally) is just going to play some assets, few local videos, mostly from the internet etc. etc.

alexmercerind commented 1 year ago

I will proceed on this after end of this month. Let's see if I find some free time before that when I get bored after studying. For now, addressing that high-priority crash issue (#17) was important.

Quackdoc commented 1 year ago

I don't have anything to test it on, but I do have the start of an audio-only optimized ffmpeg config https://github.com/Quackdoc/mpv-win32-cmake/tree/audio-tuning

not sure how bad or good it works, and I won't be able to test it for a while since I don't even have a working app to test it on, but the test build I have cut the size down to 5.23 MB

I did not include fdk-aac since I am still unsure of the varying potential patent licensing issues with it. I may have left some codecs out too, not sure,

alexmercerind commented 1 year ago

I don't have anything to test it on, but I do have the start of an audio-only optimized ffmpeg config https://github.com/Quackdoc/mpv-win32-cmake/tree/audio-tuning

not sure how bad or good it works, and I won't be able to test it for a while since I don't even have a working app to test it on, but the test build I have cut the size down to 5.23 MB

I did not include fdk-aac since I am still unsure of the varying potential patent licensing issues with it. I may have left some codecs out too, not sure,

👏👏👏

That's remarkable!

I replaced the libmpv-2.dll in Harmonoid & everything seems to work correctly. All the metadata & album arts are being correctly read, all the files are also playing fine without any kind of issue.

The size is really small & will reduce the overall size of application quite a lot.

However, there is this one file which someone ripped from YouTube because they wanted me to test out something which isn't playing or being read correctly by Harmonoid (& libmpv-2.dll). It seems to be vorbis or something according to ffprobe.

alexmercerind commented 1 year ago

Can something similar be done for the video playback supporting libmpv shared library? I don't mind stripping things out.

Most things e.g. RTMP, AV1 etc. are not really required or only required in very specialized use case, for which the default build may work out for the dependents. I don't know what things does Chromium use in it's FFmpeg compilation process, but that's a good goal for me.

Quackdoc commented 1 year ago

I am working on a video one, but there are a LOT more options for that so it will take a good amount of time.

also I think I know why it isn't working, can you try passing the parameter vid=no?

by default MPV will still try to decode the video track (in the case of music the video track is just a single picture, (typically png or jpeg) but there are a couple of other formats), so there is a good chance mpv is failing on that.

I'm not sure if you plan on allowing mpv to decode the art or not. if so I can try and add the decoders back in. adding in a couple of select decoders shouldn't increase the size all that much

alexmercerind commented 1 year ago

I am working on a video one, but there are a LOT more options for that so it will take a good amount of time.

I understand.

Also I think I know why it isn't working, can you try passing the parameter vid=no?

by default MPV will still try to decode the video track (in the case of music the video track is just a single picture, (typically .png or .jpeg) but there are a couple of other formats), so there is a good chance mpv is failing on that.

Yeah! You are correct.

I'm not sure if you plan on allowing mpv to decode the art or not. if so I can try and add the decoders back in. adding in a couple of select decoders shouldn't increase the size all that much.

I'm going to keep the image decoders since my personal project uses them to extract the album art (with vo=null in libmpv).

https://github.com/alexmercerind/media_kit/blob/20c46b58bb5c8a9e749b6b9760234281e7bf2e02/media_kit/lib/src/platform_player.dart#L37

I merged your Quackdoc/mpv-win32-cmake@audio-tuning commits into my alexmercerind/mpv-win32-cmake@disable-gl.

I enabled few image decoders (not sure if I missed some) which fixed playback issue with that sample I shared above. It was missing the PNG decoder (like you said). Secondly, I noticed that online URLs e.g. YouTube URLs were not playing with this build of libmpv, so I enabled networking support & few protocols explicitly as well.

This seems to be working perfectly on my side with Harmonoid. I hope there aren't any regressions. I have no worries however, because we haven't released anything publicly from package:media_kit yet. So, we are free to make architectural changes as we want. The resulting build size still seems minimal & close to the initial configuration.

Quackdoc commented 1 year ago

perfect, push comes to shove, the build system is rather flexible in any case it shouldn't be too hard to pop more decoders in. I can't think of any missed of the top of my head except maybe JXL, since I have seen a couple of .mka using it (not sure how they were made though, maybe the patch I put out a while ago).

but considering ffmpeg can't even handle those (those being jxl images inside of an mkv/mka) by default without being patched, I wouldn't consider that an issue for now anyways. Im more concerned about potentially missing some codecs here. but IIRC I got all the somewhat popular ones.

alexmercerind commented 1 year ago

We can heavily minimize Windows video shared library AFAIK. There's a large scope.

alexmercerind commented 1 year ago

This thread is mostly redundant. We were able to resolve licensing issues for the platforms that are supported now.