support for dynamic linking and reuse system CFLAGS

[albertz]

My goal is it to create a redistributable dynamic FFmpeg library. This makes it possible. Also, I need custom CFLAGS to specify that I want to use the MacOSX 10.6 SDK so that it is compatible with >=MacOSX10.6.

[zimbatm]

Ok, it's not really in the scope of the project but doesn't add too much complexity either. Are you willing to maintain the dynamic bits if there is an issue ?

[albertz]

Yes I can maintain it. Right now I'm using it for < https://github.com/albertz/music-player>.

But I guess there always will be issues because once there comes a new MacOSX version out or a new clang/gcc, I feel like things will break. Even the static version still does not work for me (still the FFmpeg build problem, after I have sorted out the other bits).

On Wed, Nov 6, 2013 at 4:14 PM, Jonas Pfenniger notifications@github.comwrote:

Ok, it's not really in the scope of the project but doesn't add too much complexity either. Are you willing to maintain the dynamic bits if there is an issue ?

— Reply to this email directly or view it on GitHubhttps://github.com/stvs/ffmpeg-static/pull/22#issuecomment-27881081 .

[zimbatm]

Nice project.

Some day I want to rewrite ffmpeg-static using some kind of compilation framework with recipes and use flags a bit like Homebrew or Gentoo. If you're interested I would love to pair on that. The basic idea is to have a DSL for project dependencies, a resolver that maps to either recipes or system packages, and another DSL for recipes that's similar to Homebrew with added cross-system bits. The recipes could just be stored in your project repo or cross-referenced somehow. Recipes are compiled in a unique path-prefix /prefix/$shasum_of_all_input_variables-$recipe_name to guarantee their uniqueness and then either symlinked or copied into the final destination.

[albertz]

I actually thought about just reusing the Homebrew formulas. Or even the whole Homebrew system and just patch the ENV.rb and maybe some other places so that it can be used to build the libraries in the way we want. Otherwise, we would have to duplicate most of the work which is already done by Homebrew.

But Homebrew is MacOSX specific which destroys the plan for doing this cross-platform.

Creating such a new framework from scratch is probably quite some work, though. At least overkill for only ffmpeg-static. And the maintain work to keep every package working for every platform/arch is immense.

But yea, I have often thought about something similar because it would have simplified my work a lot in creating redistributable binaries for MacOSX and Windows.

For Linux, actually I don't really see such big point / use case in doing this because you just can use the system package manager, don't you?

On Thu, Nov 7, 2013 at 2:39 PM, Jonas Pfenniger notifications@github.comwrote:

Nice project.

Some day I want to rewrite ffmpeg-static using some kind of compilation framework with recipes and use flags a bit like Homebrew or Gentoo. If you're interested I would love to pair on that. The basic idea is to have a DSL for project dependencies, a resolver that maps to either recipes or system packages, and another DSL for recipes that's similar to Homebrew with added cross-system bits. The recipes could just be stored in your project repo or cross-referenced somehow. Recipes are compiled in a unique path-prefix /prefix/$shasum_of_all_input_variables-$recipe_name to guarantee their uniqueness and then either symlinked or copied into the final destination.

— Reply to this email directly or view it on GitHubhttps://github.com/stvs/ffmpeg-static/pull/22#issuecomment-27963679 .

[zimbatm]

Yeah it's a quite large undertaking. I have thought of re-using other things like Homebrew but never found something that is project centric. Here we want to describe the project's dependencies and pull them all into a final asset. Because you hopefully don't need too many dependencies for your project I think it's fine to copy things from Homebrew on demand like they did with macports.

Even for Linux it can be useful to have that kind of setup. For example if you want to deploy an webapp to a server, you could bundle all the app's dependencies, even the exact ruby or php version that you want. Then deploy is just copying in the asset and run it. You'll still be able to deploy even if rubygems or something else goes down. And because everything is inside the asset you can easily switch ruby versions (or others) between deploys. That would be more the use case that I have these days.

[albertz]

I would be really interested in something like this.

What features should such system support?

• Custom flags for packages, i.e. to enable/disable certain features? Like USE flags in Gentoo? For Homebrew, a package might have custom flags.
• Both static and dynamic compilation? For all my use cases, I'm fine with dynamic, but I guess the whole point behind ffmpeg-static is to build a static binary. Also, where would that be specified? Should every lib just compile both versions (.a + .so)? And a final asset (for your project) specifies whether it wants to link them statically or dynamically?
• Cross-compilation support? E.g. be able to build Win32 execs on Linux? I really demanded such a system earlier because it was such a pain to build Win32 execs which depend on a few Open Source libs.
• Mac: Specify the MacOSX SDK. This is important for me if I want to build libraries which are redistributable, i.e. which work on earlier MacOSX versions.
• Mac: Be able to build universal binaries. Some libs support that by themselves. But actually it's easy to do it automatically: Just build all versions for all different archs (32bit, 64bit, etc.) and then there is the lipo tool which can just merge them together into a so-called fat binary/library.

Also, not all combinations are possible. For example, sometimes, 64bit support only works >=MacOSX 10.6 or so. Or certain features are only available if you build for >=MacOSX 10.7. Or they are possible in certain versions of the lib and later versions dropped the support.

On Thu, Nov 7, 2013 at 4:23 PM, Jonas Pfenniger notifications@github.comwrote:

Yeah it's a quite large undertaking. I have thought of re-using other things like Homebrew but never found something that is project centric. Here we want to describe the project's dependencies and pull them all into a final asset. Because you hopefully don't need too many dependencies for your project I think it's fine to copy things from Homebrew on demand like they did with macports.

Even for Linux it can be useful to have that kind of setup. For example if you want to deploy an webapp to a server, you could bundle all the app's dependencies, even the exact ruby or php version that you want. Then deploy is just copying in the asset and run it. You'll still be able to deploy even if rubygems or something else goes down. And because everything is inside the asset you can easily switch ruby versions (or others) between deploys. That would be more the use case that I have these days.

— Reply to this email directly or view it on GitHubhttps://github.com/stvs/ffmpeg-static/pull/22#issuecomment-27974434 .

[zimbatm]

Actually the nix package manager does a lot of things right. There is still a long way to go for "The perfect project compiler"™ but it could be a good start. You can describe all your project's dependencies using nix and compile in OSX and Linux. Windows support, cross-compilation and multi-arch are missing from your wish list. And the biggest (but probably not the hardest): a way to merge all the dependencies into a single archive. Have a look and let me know what you think.

[albertz]

Looks really cool! Unfortunately, I'm quite busy atm with other things but when I get back to this task, I will definitely try it out.

Is FFmpeg already in Nixpkgs?

On Mon, Nov 25, 2013 at 2:01 PM, zimbatm notifications@github.com wrote:

Actually the nix package manager http://nixos.org/nix/ does a lot of things right. There is still a long way to go for "The perfect project compiler"™ but it could be a good start. You can describe all your project's dependencies using nix and compile in OSX and Linux. Windows support, cross-compilation and multi-arch are missing from your wish list. And the biggest (but probably not the hardest): a way to merge all the dependencies into a single archive. Have a look and let me know what you think.

— Reply to this email directly or view it on GitHubhttps://github.com/stvs/ffmpeg-static/pull/22#issuecomment-29199606 .

[zimbatm]

No worries. Just wanted to let you know that I'm probably going to head that way.

For now my feedback is that the nix language is quite foreign. I'm half trough the learning curve but I feel that once I'm comfortable it's going to be great. The whole system makes sense in my point of view. Each package is installed in a unique prefix so you can even have multiple versions of the same software installed. It does make the packaging a bit harder if upstream doesn't support prefix-based installation.

FFmpeg is already packaged. If you're using the default feature flags nix automatically downloads a pre-built version from their site, otherwise it acts like gentoo and builds the software.

[zimbatm]

Closing as probably abandoned. Let me know if you still want to work on this.