Better handling of 3rd party dependencies

lukas-w commented 10 years ago

Taken and adapted from my original comment in #287 See another discussion of this subject in #96

Allow me to say, that the way external dependencies are handled and embedded in LMMS's source is just not a practicable way, and all in all a mess.

In my opinion, one should avoid including source from other projects as far as it's possible. It adds unnecessary compile time for libraries that may be shipped by the linux distro anyway, makes it very difficult to maintain the source, to update the library from upstream, and to push own commits to upstream. Especially the last three points can cause a lot of work.

Basically, it's a matter of principle: Imagine every project out there shipping it's dependencies in the source tree, every project using and installing a different version of the needed libraries. This would lead to a lot of versions of the same library being installed next to each other at the same time which is a huge waste of disk space. It also contradicts the principle and intentions of shared libraries and package management.

Starting my argument at the other end: We can't include every single library we need, for instance we're never going to ship Wine or Qt or Jack or whatever because it's too huge and just not our job as that's what shared libraries and package managers are for. There are good reasons for the fact that including external sources is no common practice and is something that is generally discouraged to do.

Even though I would head to removing as many third party sources from the tree as possible, I (partly) understand Toby's reasons for including them. So if we're going to continue shipping external sources (the question about whether to do this in the first place shall not be the point of my message now), we should at least find a better way to do this.

I therefore suggest:

Have a special directory for dependencies
Move all 3rd party libraries to one single directory in the source root, let's name it extern. The root directory would then look something like this:

lmms/
├── cmake/
├── data/
├── extern/
|      ├── calf
|      ├── caps
|      ├── …
|      └── ZynAddSubFx/
├── include/
├── plugins/
├── src/
├── CMakeLists.txt
├── …
└── README

Use git subtree

Use git subtree to manage the libraries. Basically it works by still having the source code in the tree, but knowing about where it comes from. This way pulling or pushing updates from or to upstream is made possible. Note that it causes no extra efforts for users who for instance just want to clone the git repository and compile lmms (in contrast to git submodule, which requires users to explicitly run a command for downloading the directores). Subtree has been introduced in git-1.7.11. Update: I just noticed this feature may be not included or not enabled in some distributions by default. It is not needed as long as you don't want to push changes in the subproject though.
Use CMake's ExternalProject module

Make use of the functionality CMake provides for this purpose. I think the best option is to use the ExternalProject module. This will help having cleaner CMake scripts, and, if used correctly, should make it possible to remove external projects later without great trouble. This would raise the minimum CMake version to 2.8. The module is also capable of automatically downloading the source or cloning it from git.

Let me hear your opinions on this.

Sti2nd commented 10 years ago

Sure, as long as it don't affect windows users in a bad way. I couldn't find any sources on that there is more LMMS users on Windows than Linux, but I have this feeling! Because on Linux LMMS is one among many free music programs (qtractor...), on Windows LMMS is THE one.

eagles051387 commented 10 years ago

Just a heads up things for windows builds sometimes have to get worse before they get better. Libre office went through that phase, which eventually was solved and 3rd party stuff is no longer an issue.

On Tue, Feb 11, 2014 at 10:10 PM, Stian Jørgensrud <notifications@github.com

wrote:

Sure, as long as it don't affect windows users in a bad way. I couldn't find any sources on that there is more LMMS users on Windows than Linux, but I have this feeling! Because on Linux LMMS is one among many free music programs (qtractor...), on Windows LMMS is THE one.

Reply to this email directly or view it on GitHubhttps://github.com/LMMS/lmms/issues/296#issuecomment-34807295 .

Jonathan Aquilina

dobey commented 10 years ago

@Sti2nd There are plenty of other options besides lmms on Windows too. And a lot more options for music in general. Don't forget that a lot of MIDI hardware comes with bundled editions of various apps, including Pro Tools and Sonar. There are also many lower priced proprietary options on Windows, that many people opt for. Most people I know who are using Windows, are using FL Studio or just buy Pro Tools or something, because "it's what the pros use" and don't use the libre apps at all. Educational institutions also tend to concentrate on teaching Pro Tools or the like, so many people are using it for that reason.

Making lmms a great option on Windows is certainly a nice goal, but "include everything in the source tree" isn't the best/right way to do that. It's just an "easy" way.

Sti2nd commented 10 years ago

@dobey Yeah, I know that Windows users tend to buy software. I think LMMS can be what GIMP is for Windows. 7 years ago few with Windows knew about GIMP, if it was available for Windows at the time... Now most people use free software on Windows to edit their photos. GIMP is one of the popular programs and I know schools around the world use it (not media dedicated schools). Schools use Audacity too. I have yet to see LMMS in any school around here.

Making LMMS great on Win can't be a goal in itself, because the program are made on Linux so it wouldn't be better on Windows. The only thing I am asking for is that we don't forget to ship a complete LMMS for Windows too. If one removes the libraries I reckon they must be put back in the Windows version? Maybe this could be done automatically by a server somewhere?

eagles051387 commented 10 years ago

builds could be automated but would require proper server hardware to do the builds. with some scripts and cron jobs on linux even mac as well as scheduled tasks on windows

On Wed, Feb 12, 2014 at 9:28 PM, Stian Jørgensrud notifications@github.comwrote:

@dobey https://github.com/dobey Yeah, I know that Windows users tend to buy software. I think LMMS can be what GIMP is for Windows. 7 years ago few with Windows knew about GIMP, if it was available for Windows at the time... Now most people use free software on Windows to edit their photos. GIMP is one of the popular programs and I know schools around the world use it (not media dedicated schools). Schools use Audacity too. I have yet to see LMMS in any school around here.

Making LMMS great on Win can't be a goal in itself, because the program are made on Linux so it wouldn't be better on Windows. The only thing I am asking for is that we don't forget to ship a complete LMMS for Windows too. If one removes the libraries I reckon they must be put back in the Windows version? Maybe this could be done automatically by a server somewhere?

Reply to this email directly or view it on GitHubhttps://github.com/LMMS/lmms/issues/296#issuecomment-34913606 .

Jonathan Aquilina

dobey commented 10 years ago

@Sti2nd I don't think most people use free software on Windows. Most people use "brand name" software on Windows. I will certainly agree that free software usage on Windows has definitely increased over the years and there are many more people using it today.

As for automating building of the plug-ins it can be done when building on Windows to pull remote sources to build as well as lmms itself. A server isn't really necessary. It's also pretty easy to do if building an installer with Bitrock for example.

eagles051387 commented 10 years ago

I actually beg to differ in regards to those on windows not using free software. I recently read last week that for example the UK government is researching open source alternatives to get themselves away from microsoft office and save money.

LMMS can and will provide that to the music world. Dj's and producers will understand that hey why should we spend money on a proprietary program, where we can get this program for free and help improve it by using it suggesting features and donating money to the project

On Wed, Feb 12, 2014 at 10:53 PM, dobey notifications@github.com wrote:

@Sti2nd https://github.com/Sti2nd I don't think most people use free software on Windows. Most people use "brand name" software on Windows. I will certainly agree that free software usage on Windows has definitely increased over the years and there are many more people using it today.

As for automating building of the plug-ins it can be done when building on Windows to pull remote sources to build as well as lmms itself. A server isn't really necessary. It's also pretty easy to do if building an installer with Bitrock for example.

Reply to this email directly or view it on GitHubhttps://github.com/LMMS/lmms/issues/296#issuecomment-34922567 .

Jonathan Aquilina

lukas-w commented 10 years ago

Err… Please try to stay on topic, guys ;)

tresf commented 9 years ago

@Lukas-W, shall this go on the roadmap?

-Tres

lukas-w commented 9 years ago

I don't think so, the proposed changes are only relevant for developers and I'd say it's a quite low priority issue.

tresf commented 9 years ago

@Lukas-W shall this one stay on the docket? We're making some slow and steady progress in terms of organizing our code as well as cleaning up some important CMake items, but the concept of extern plugins seems like it will be quite a ways out for a few reasons...

Only OSs which offer the specific external plugins would be able to benefit from linking against them, all others would still have to compile from source.
Versioning becomes an issue as the codebase would have to be tied to a specific version (or stable API) from an external plugin.
For the OSs which would have to compile from source, obtaining the compatible version of source code could prove to be difficult as our integration code slowly drifts from theirs.

I do like the idea of treating 3rd party libraries as 3rd party (if anything, from an organization perspective) we sort-of do this for ZynAddSubFX with our separate Zyn repo but it makes it quite confusing where to submit bugs (lmms, zyn or upstream zyn)

Previously I had mentioned if this should be on the FurtureMap (yeah, the one we aren't really maintaining) but perhaps it should just be a goal we keep in mind when integrating new external plugins.

Since this bug was created, @falkTX helped get Carla plugin support, which uses the native plugin available to the OS... Perhaps his opinion on the topic would help?

tresf commented 5 years ago

@lukas-w do you still want this? We're pretty close on master except external plugins still live in plugins/<PluginName>/<submodule>. I don't hate this design as many of them require custom code side-by-side the submodules anyway.

lukas-w commented 5 years ago

Let's close it, most of what was written here is outdated. I think we're doing OK with the current submodule solution.

LMMS / lmms

Better handling of 3rd party dependencies #296