search

mesonbuild / meson

The Meson Build System
http://mesonbuild.com
Apache License 2.0
5.62k stars 1.63k forks source link

Subproject limitations #422

Closed nioncode closed 5 years ago

nioncode commented 8 years ago

Currently, subprojects are somewhat limited. Example:

B and C are completely unrelated, so I it is not possible to have all projects in a single source tree and just use subdir.

Also, nesting subprojects is impossible, so I can't do:

If I don't have control over X and Y, I can't create a simple Z project that uses Y. As the creator of Z, I should not have to worry if any of my subprojects use other subprojects, which would require me to list all needed subprojects in my own subproject folder.

Did I miss something? Note: I did not test these, I just want to clarify if this is how it is supposed to be and discuss possible future improvements.

jpakkane commented 8 years ago

Using subdir for subprojects is always wrong. Do not ever do it! Use subproject.

If you use subproject foo that uses other subprojects, the master project must add those to its own toplevel subprojects directory. See test case 79 to see an example.

The reason Meson forces you to do this is disambiguation. Suppose you have top project A and subprojects B and C, both of which use subproject D. It is mandatory that there is only one instance of D and both subprojects use the same instance. Otherwise they would each embed a copy, their versions would be different, symbols would clash and things would crash and burn in a way that is very hard to debug if you don't know beforehand that this can be an issue.

nioncode commented 8 years ago

Just to clarify if anyone else is wondering: If you want to include a library lib as subproject which has a subprojects folder of its own, it is NOT necessary to patch the source code of lib to remove the subprojects folder and adjust its meson.build file. The only thing to do is to add all subprojects of lib to the subprojects folder of the main project (with the same name as in lib/subprojects), because meson in fact ignores all subprojects folders except the one of the main project. So far, so good.

However, you will still run into problems, if you have 2 (or more) subprojects that both use the same library, but in a differently named folder inside their subprojectsdirectory (or they use the same name, but the source code is from two different versions). You will then face the exact same issue that one library hides the other (without any diagnostic warning or error) and you might experience runtime errors.

The issue here has nothing to do with subprojects in general, because this is an issue of the C language itself. I don't see why meson forbids recursively defined subprojects then, though. The only case where this protects you from anything is if all subprojects use the same library in the same version with the same name, which makes it trivial to include as a toplevel subproject. But in this case, it would not even matter if the main project links to multiple instances of the library, because the linker will resolve the symbols from the first library that declares them (in its search order), so all references will point to the same symbol.

I am in no way an expert on the C linkage process, so it would be great to get some more details why this is really necessary or does this make it just simpler for the main project maintainer, because he must decide upfront (by setting up the top level subprojects) how the compatibilty between subprojects and their dependencies is fulfilled?

biocomp commented 8 years ago

I randomly opened this discussion, but already have an opinion :)

I think we can agree that there are cases when different versions of same library in one project are useful (when done cautiously, as it is indeed quite fragile).

I also think that this is probably rare case, but is there a reason to explicitly forbid it?

Then, even if forbidding it, could meson allow for recursive detection of subprojects, and then issuing an error, if different versions of same sub project are used? Or even when just same sub project is used twice. And only then user would be forced to do something for disambiguation. This will avoid breaking builds just because some sub-sub-project decided to use some other subproject.

Thank you for your time!

nioncode commented 8 years ago

I think the case where two subprojects use the same library (possibly with different versions) is kinda rare compared to the more common case where a subproject just uses some other subproject that has nothing to do with any other subproject in the build. This should definitely be supported. Otherwise, as the maintainer of the main project, whenever I update any of my subprojects, I'd have to recursively check every subproject and if it was updated then copy those changes to my own subprojects folder. This quickly gets out of hand e.g. when using git submodules for subprojects.

Since it is not possible (at least not feasable) for meson to detect the same-library-used-twice situation reliable (at the moment it just relies on the same subproject name), it's a bid odd to forbid the common case of nested subprojects because of a potential issue that is not really solved with a single subprojects folder.

jpakkane commented 8 years ago

Having two different versions of the same library in the same project is very, very bad. Suppose you have lib1 and lib2 and functions func1(foo *obj) and func2(foo *obj). Func1 is in both libraries (and does the same thing) but func2 is only in lib2. If you generate an app that links both of these in the same executable (whether shared or static) then the symbol resolution gets massively mixed up.

When you call into func2 anywhere in your program you call the same one function. However when you call into func1 you might call the version from lib1 or lib2. Even weirder you might call into the version from lib1 even if the function call happens in lib2. If the internal format of foo has changed between these two versions (which you can do without breaking ABI), anything at all can happen. More annoyingly these sorts of bugs are massively difficult to debug as they usually manifest as random memory corruptions.

Having multiple different versions of any library used in the same executable will not work and, more importantly, can not be made to work. Any system that permits this kind of intermixing is inherently broken.

Otherwise, as the maintainer of the main project, whenever I update any of my subprojects, I'd have to recursively check every subproject and if it was updated then copy those changes to my own subprojects folder.

That is true. Such is the responsibility of being a maintainer. The unavoidable fact is that someone, somewhere has to linearize the list of dependencies to avoid duplicate deps. There is no way around it. Wrapdb aims to make this easier by providing a single source for most of deps.

Not using subprojects of subprojects automatically has other benefits, too. For example if you have a common dependency (zlib, sqlite, etc) that is used by 10 different subprojects then that would mean compiling the dependency 10 times. This quickly becomes infeasible.

If you use distro packages for your dependencies, all of this is taken care of for you. If you don't have the distro's support behind your back, you have to do it yourself.

nioncode commented 8 years ago

I am onboard with the issues you described when using the same library - or even two libraries that both export the same symbol. However, I don't see how forbidding recursive subprojects helps here. It just forces the main project to add all subprojects to itself, but does not help at all with conflicting symbols if the subprojects use different names.

My situation is:

There are no conflicts with any symbols in the hierarchy, so I would expect meson to just apply the subprojects recursively, because managing the correct version of core in main/subprojects/core is hard (especially when there are more subprojects and deeper hierarchies).

I therefore propose to allow recursive subprojects, since those are not any different from nested subdirs, which is explicitly allowed and leads to the same issues. To make multiple subprojects use one global subproject, I propose something like: subproject('foo', overrides: ['a/foo', 'b/foo_different_name']), which gets applied relative to the current meson.build file, so that it is possible to recursively override subprojects.

This proposal:

  1. makes it easy to include a subproject, which uses subprojects on its own
  2. makes subprojects use the same library
  3. and even makes subprojects that use the same library but under a different name use the same library

I can't think of any downside to this approach.

If meson really wants to provide a way to prevent the issues of same symbols in different libraries (which it currently does NOT), it should not conflict with easy to use subprojects, imho.

jpakkane commented 8 years ago

There are two different failure cases here.

The first on is that two completely different expose the same symbol name. If this happens you are screwed in any case. There's nothing any build or packaging system can do to help you. Fortunately this is rare.

The second case is that you have subprojects that depend on the same subsubproject but they each bundle their own version with a different name. This is the most common failure case. Forbidding nested subprojects makes this case impossible because you need to hoist all deps to the top level and there can only be one subdirectory with a given subproject name. This is the Highlander way of dependency management: there can be only one.

Another advantage of the flat hierarchy is that it makes it very simple to determine what subprojects any given project uses. All you need to do is ls subprojects. If you allow nested subprojects this is no longer possible. The only way to be sure is to go through the entire source tree looking for possibly hidden subprojects.

One of the main design points of subprojects is that any project can be used as a master project or a subproject without any changes to the code. Having an override option breaks this because that alters global state on how subproject resolution happens (and if you have many projects doing that, well, welcome to conflict city). That is not something a single project should be allowed to do. All it cares about is building its own targets and requesting its dependencies from the main system.

That being said we might very well develop some tools that make hoisting dependencies to the top level as easy as possible.

(Further: it might not be obvious but the suggested file layout is one git repo per project. You should not have a repository with stuff in its subprojects dir unless you have a monolith app whose deps never change. Subprojects should be gathered by e.g. using wrap files to do git checkouts.)

nioncode commented 8 years ago

How does a flat subprojects hierarchy prevent any issues in this case:

with libZ-foo and libZ-bar being the same library.

What should I do now? I need a way to decare in my main meson.build that subprojects/libZ should be used for both libZ-foo and libZ-bar nested subprojects.

This would be easily solved with either subproject('libZ', overrides: ['libA/subprojects/libZ-foo', 'libB//subprojects/libZ-bar'] or subproject('libA/subprojects/libZ-foo', overrides: ['libB/subprojects/libZ-bar'], which is even nicer, because we don't have to copy the nested subproject to the main project at all. Since all subprojects are declared in the main meson.build file, all dependencies can be resolved from there. When processing libB/meson.build and coming across subproject('libZ-bar') the interpreter would recognize that this subproject gets overridden by another one, just like it does currently when the subproject name matches.

I don't see why an override option would break any encapsulation or subproject resolution, since all overrides are propagated top-down through the source tree, so all valid subprojects are known when processing any meson.build file.

Example from the beginning, but taken a step further:

meson build files: Either:

or:

I get that its much simpler to flatten the hierarchy (but even then we do need a way to resolve same subprojects with different names), but it is also much harder for a user to work with nested subprojects.

I'll want to clarify why I find the current subproject concept limiting: I have a core library that I use in a few of my projects. There are other libraries that are built on top of core. I want to include those libraries in my source tree (with a recursive dependency to core). Since all projects develop at a different speed, each project uses another version of the library (I keep track of that with git submodules and checking out a specific tag). I can't use wrapdb files, because a) I don't want to download the dependencies at build time, but at code checkout time (so I can work offline without worrying about keeping my build directory around to not delete the subprojects' sources) and, more importantly, b) I am actively developing on core from inside main and push the changes back up if issues are resolved. So I don't have a fixed, stable dependency version that only changes very slowly.

Basically there are 2 possible solutions to my issue:

  1. Allow recursive subprojects
  2. Allow projects to be built as separate projects as well as to be included as subdir (and ignoring the 'project' command then). I don't know what this would break, though. It would be great to get some more insight about the actual differences between subprojects and subdirs from an internal point of view.

I hope I got my point across at least a bit :)

BTW: We could also simplify the whole libX/subprojects/libA/subprojects/libZ-foo to libX/libA/libZ-foo since we do know the name of the subprojects folder and can treat each / as the next level in the subprojects hierarchy.

jpakkane commented 8 years ago

with libZ-foo and libZ-bar being the same library.

There's the problem right there. libZ-foo and libZ-bar are two different subprojects. If you want to use a subproject libZ then the call you make is always subproject('libZ'). If you have anything else then that is a different subproject. To handle versions (which is the most common reason for two different pathnames) you specify the version keyword. Usually you would have version : '>=1.0.0' in one dep and '>=1.1.0` in another. Then you just put a new enough version at the top level and you are good. Again, creating tools to make this easier is something we could do.

I can't use wrapdb files, because

There may be a misunderstanding here. Wrap files can be used just fine without wrapdb. You can even run your own instance of wrapdb if you wish. A wrap file just specifies how to obtain the dependency in question. For your case you would create a wrap file for your project called core.wrap that looks something like this:

[wrap-git]
directory=core
url=https://example.com/core.git
revision=head

This tracks git head. You can specify a hash id instead if you wish to do version tracking manually. Meson will do the checkout and all that good stuff for you. For all the details please read the manual.

Allow projects to be built as separate projects as well as to be included as subdir (and ignoring the 'project' command then). I don't know what this would break, though.

CMake has this approach. The problem with this is that subprojects are not isolated but instead can change their parents' state. I have had the displeasure of debugging projects that do this (usually not by intention but by accident). It is misery on end that I don't wish upon my worst enemy. Running subprojects in isolated sandboxes is the only way to make it work. In fact let me put my project lead hat on and state the following for the record:

Permitting the use of subprojects with subdir will never, ever, ever be allowed. Any further discussion on this issue will go directly to /dev/null.

nioncode commented 8 years ago

libZ-foo and libZ-bar are two different subprojects

This requires all project maintainers to do the right thing without any chance for me to correct issues in those projects. Also, there are many reasons for different names for the same (in the sense of a common base, must not be identical) library, e.g.:

So, something like an override or alias would be quite useful. A simple subproject('libZ-foo', alias: 'libZ') would enable the usage of libZ-foo for both libZ-foo and libZ subprojects. This can be done without enabling recursive subprojects, so it would not complicate things at all.

This tracks git head

This is not helpful during debugging sessions, since it only can checkout code that I already pushed to the repo. When I'm currently trying to fix a bug I don't want to push every change I make, but rather change the code, run / test it, and then do a push when I know that it works.

The problem with this is that subprojects are not isolated but instead can change their parents' state.

You mean subdirs right? Why doesn't meson run subdirs in isolation and require the parent to use

sub1 = subdir('sub1')
var = sub1.get_variable('some_var')

like it does with subprojects? This would make meson.build files easier to read, because a variable in a subdir can only be defined in the current file or in one of the parent's. Currently, the variable can be defined in any subdir that was executed prior to the current one, which makes it kinda hard to hunt down variables.

Overall, subprojects seem to be for a very specific use case (also mentioned in the FAQ): including external dependencies. However, including internal dependencies, i.e. libraries that are created in other projects, can't be done easily if they are also build as projects themselves.

How would you tackle the issue of including a library that is created in an outside project (which is owned by me) in a way that allows me to easily modify the libraries code until I fixed a bug or implemented a new feature? I can think of the following:

project('main', 'c')
# Defines the needed 'core' dependency for the lib project.
subdir('libs/lib/libs/core/src')
# Defines the 'lib' dependency.
subdir('libs/lib/src')

main = executable('main', 'main.c', dependencies: lib)
test('t', main)

The libs folder in this example used to be a subprojects folder, but these can't be applied recursively, so there is currently no other way.

434 would provide a solution, because I could point it to my 'core' development folder and work on the bug there.

jpakkane commented 8 years ago

This is not helpful during debugging sessions, since it only can checkout code that I already pushed to the repo.

If you work on your own machine you can define the git url int the wrap file as file:///path/to/local/checkout. Alternatively you can put a symlink in subprojects/foo that points to your other project.

As far as the package naming go, could you please give an example of multiple providing incompatible versions of a dependency and a project that links to them in the same process via current tools (preferably pkg-config).

nioncode commented 8 years ago

Pointing wrap to my local git repo would still require me to commit the code first. The symlink works and should be fine during debugging sessions.

I am not sure what example you'd like to see. I am not talking about incompatible versions, but about compatible ones with different names (in the subproject folder, they very well might both produce a library called libfoo). If I manually wrote a Makefile, I would just link to libfoo and use the one from the folder with the highest featureset (or any of the provided ones if just the folder name has changed).

I have the feeling that I need to recap what issues I think should be solved with subprojects.

  1. Keep root subprojects in sync with recursively defined subprojects: Since we don't want to support recursively applied subprojects, there should be an easier way for the root project to include the same subproject version that one of its subprojects uses. Currently, we can only use a symlink from the root subprojects folder into one of its children, but this is not a clean approach. It would be nicer to have a file (maybe a wrap.file) or a directive in meson.build to say 'hey, get subproject foo from subprojects/a/subprojects/foo instead of from subprojects/foo'. This still requires the root project maintainer to explicitly define the subprojects that should be used, so we don't run into any issues from automatically including all subprojects recursively, but add the flexibility of not having to copy the subprojects source (or manage some semi-automatic download / sync scripts).
  2. Resolve conflicts of two subprojects that use different names, but should really use the same subproject defined in the root project: Something simple like an alias argument for subprojects would fix this easily.

I would gladly provide pull requests for these features, but won't start implementing these until I get an official 'go' :)

Qix- commented 5 years ago

I'm going to make this a long one because this very issue has driven me away from the likes of Bazel, Buck, etc.

In my opinion, as much as I hate to say this, CMake is the only build-system to get subprojects even remotely correct.

Unfortunately, Meson's subproject design is going to force me to drop it after just a few short hours.

Having two different versions of the same library in the same project is very, very bad.

Please stop with these strange assumptions and absolutions about libraries. You're missing the point about the possibility of two unrelated executables being used in a later build step with the same library.

For example, I have two tools that I need to use in order to convert some file formats. They are called tool-a and tool-b. Both of them use my CLI options parsing library xopt but at different versions, and since they are submodules they thus are pinned to different commits.

They do not link to one another in any way.

By your logic, I have to choose one of the xopts, wrap it, and then define it at the top level. This makes no sense and solves nothing.

Forbidding nested subprojects makes this case impossible because you need to hoist all deps to the top level and there can only be one subdirectory with a given subproject name. This is the Highlander way of dependency management: there can be only one.

This design and philosophy will drive away users from Meson. I'm now regretting the several hours I've spent evaluating Meson for my projects. It's clear nested dependencies are entirely broken.

Here's a simple example:

$ mkdir -p subprojects/foo/subprojects/bar
$ cat > meson.build <<EOF
project('top')
message('hello from top')
foo = subproject('foo')
EOF

$ cat > subprojects/foo/meson.build <<EOF
project('foo')
message('hello from foo')
bar = subproject('bar')
EOF

$ cat > subprojects/foo/subprojects/bar/meson.build <<EOF
project('bar')
message('hello from bar')
EOF

$ meson build
The Meson build system
Version: 0.49.2
Source dir: /private/tmp/test-meson-come-on-bro
Build dir: /private/tmp/test-meson-come-on-bro/build
Build type: native build
Project name: top
Project version: undefined
Build machine cpu family: x86_64
Build machine cpu: x86_64
Message: hello from top

|
|Executing subproject foo
|
|Project name: foo
|Project version: undefined
|Message: hello from foo
|WARNING: Dependency bar not found but it is available in a sub-subproject.
|To use it in the current project, promote it by going in the project source
|root and issuing the following command:
|meson wrap promote subprojects/foo/subprojects/bar

subprojects/foo/meson.build:3:0: ERROR:  Failed to initialize 'subprojects/bar':
Subproject directory not found and bar.wrap file not found

A full log can be found at /private/tmp/test-meson-come-on-bro/build/meson-logs/meson-log.txt

There's absolutely no reason why this hierarchy shouldn't work, and the fact that it doesn't on the latest version means I simply cannot use Meson for my projects. I rely on the nested nature of things to build static executables and do WPO.

Such is the responsibility of being a maintainer.

No. This is not at all the goal here. A build system is supposed to relieve these pain-points, not exasperate them.

Wrapdb aims to make this easier by providing a single source for most of deps.

I'm sorry but this isn't feasible in my opinion. The best way to make this work is to give users the ability to vendor Meson scripts in next to either vendored or submoduled code. This is what repositories are already doing, and it has the added benefit with Github that they can do automatic vulnerability alerts based on the submodule links you have set up in your repo - including across forks.

With wrapfiles and URLs being specified, that's all lost.

Further, wrapdb entries don't have their patches in any sort of source control, but instead zip them up as a release. Aside from the fact that's a non-starter from a security point of view, it's very obtuse from a maintenance standpoint.

Not using subprojects of subprojects automatically has other benefits, too. For example if you have a common dependency (zlib, sqlite, etc) that is used by 10 different subprojects then that would mean compiling the dependency 10 times. This quickly becomes infeasible.

As a project maintainer of many heavily-nested projects with CMake, I simply do not care about this as long as the builds are incremental. This is very close to the bottom of my concerns - initial build times are almost a guarantee that they will be long, and this is something most developers I know have come to terms with. What matters more is that the build system gets out of my way and works intuitively, without needing me to think or reference documentation. This is why CMake is so ~rage-inducing~ unattractive to many because even after years of using it it still requires constant referencing of documentation to do the simplest of things.

The first on is that two completely different expose the same symbol name. If this happens you are screwed in any case. There's nothing any build or packaging system can do to help you. Fortunately this is rare.

So rare, in fact, that anecdotally this has only happened to me maybe once or twice in my entire career as a C/C++ developer. This is not a case I'm worried about at all because it's incredibly easy to debug.

Another advantage of the flat hierarchy is that it makes it very simple to determine what subprojects any given project uses.

I don't see how this is an issue unless you're using URL-based acquisition, which - personally - I see as a huge negative feature that shouldn't even be included in build systems because it is A) not their job (they are not a package manager), and B) causes even more unreliability as your build now becomes dependent upon the availability of the remote server.

This is horrifying from a sys-ops and dev-ops perspective and is one of the biggest failures of Go's dependency management system.

Further, the use of remote HTTP URLs, as you said, reduces your ability for introspection.

You're trying to solve it by forbidding nested submodules in order to protect your HTTP acquisition - however, you should be doing it the other way around.

If you allow nested subprojects this is no longer possible. The only way to be sure is to go through the entire source tree looking for possibly hidden subprojects.

How can they be hidden if they all follow the directory convention of */subprojects/*? How can they be hidden if they're included as subproject(...) invocations?

That being said we might very well develop some tools that make hoisting dependencies to the top level as easy as possible.

Please don't. "Hoisting" dependencies sounds like a maintenance nightmare and even in the simple cases feels wrong, looks wrong, and behaves wrong.

Please fix the real issue here.

There's the problem right there. libZ-foo and libZ-bar are two different subprojects. If you want to use a subproject libZ then the call you make is always subproject('libZ').

Not if they're within two disparate subprojects. You can assert that two libraries are the same, albeit maybe different versions, if their names are the same. That's all fine and well.

However, you can also assert the relationships between these dependencies since we very clearly define them using executable(), library(), static_library(), declare_dependency(), etc. You should be able to figure out quite easily which potentially conflicting libraries will and won't be touching each other via graph edge boundaries.

So what you're describing isn't a problem with the project layout. It's a problem with your narrow array of ways you envision people using Meson. This isn't meant to be a 'dig' at you, I'm simply pointing out the shortcomings of many new-age build systems and I'm sad to see Meson have so much potential but ultimately break down at the same place most other build systems break down.

CMake has this approach. The problem with this is that subprojects are not isolated but instead can change their parents' state.

I 100% agree this is a poor design decision in CMake. However, this doesn't negate the philosophy of nested submodules in any way.

Permitting the use of subprojects with subdir will never, ever, ever be allowed. Any further discussion on this issue will go directly to /dev/null.

Literally nobody on this thread is arguing for this.

@nioncode: This is not helpful during debugging sessions, since it only can checkout code that I already pushed to the repo.

☝️ Seconded.

If you work on your own machine you can define the git url int the wrap file as file:///path/to/local/checkout.

This doesn't scale. This runs the risk of pushing bad configuration to a remote, which in the case of CI/CD will break production systems if they're naïvely configured.

Alternatively you can put a symlink in subprojects/foo that points to your other project.

Please don't make me spell out why this is a terrible idea.

As far as the package naming go, could you please give an example of multiple providing incompatible versions of a dependency and a project that links to them in the same process via current tools

DLL proxying, dependency patching, debugging tools that do function call detouring.

While I agree that the build system shouldn't allow this by default, your assertions that there is never a valid reason for linking two libraries with identical symbols is objectively wrong.

Also, you're still missing the point entirely about how libraries are used in nested subprojects.

At this rate, I don't see how Meson is a viable alternative to any build system. I hope you make the right decision to change your philosophical views on project hierarchies because as of right now they are very, very broken.

jpakkane commented 5 years ago

If you want to have multiple different versions of a library in one project, you can do so fairly easily. Suppose you want to have versions 1 and 2 of project foolib. Then you'd set up your directory tree like this:

subprojects
    foolib
       <code here>
    foolib2
        <code here>

And then you could get them via subproject('foolib') or foolib2 as needed. You can even have tens of different minor versions of any one library if you so desire, they just need to have a unique dirname each and be in the top level subprojects dir.

This has the additional benefit that if you want to know how many different versions of foolib you have in your project conglomeration, you can get the answer with a single ls command.

Qix- commented 5 years ago

@jpakkane you're missing all of the points yet again.

xclaesse commented 5 years ago

@jpakkane you're missing all of the points yet again.

I don't understand why you're saying that. I was going to reply exactly the same as what @jpakkane just said, that answer your issue regarding having different versions of the same library.

One thing I have been working on lately is adding the ability to override dependencies. For example:

dep = declare_dependency(...)
meson.override_dependency('foo', dep)

# dep == dep2 here
dep2 = dependency('foo')

What else do you need regarding that issue?

Your other main concern is about nested subprojects, and I agree with you it's not ideal (however, I don't see how that can be blocker in any way). Copying the .wrap from a subproject to the parent project doesn't feels right, we should have a way to rather "link" to it. I already thought we should have some kind of config file in subprojects/ that has a list of sub-subprojects to consider to be part of the master project. So instead copying subprojects/foo/subprojects/bar.wrap you could rather edit a file and tell to use subprojects/foo/subprojects/bar.wrap without copying it. Would that solve your concern?

TheAifam5 commented 5 years ago

I have the same issue now. Exactly the same problem as @Qix- described above.

  1. We can't have separated projects without packing them to the subprojects.
  2. And if we do that, simply does not work.

Example of my project layout:

+ ROOT
| meson.build
|---+ subprojects
    |---+ core
    |   | meson.build
    |
    |---+ graphics
    |   | meson.build
    |   |---+ subprojects
    |       |---+ ogl
    |           | meson.build
    |
    |---+ scripting
    |   | meson.build
    |   |---+ subprojects
    |       |---+ lua
    |           | meson.build

:( I really love Meson, but for this project I need to find an alternative.

meisenzahl commented 5 years ago

@TheAifam5 I just implemented a feature for meson to support nested subprojects:

https://github.com/semasquare/meson/tree/nested-subprojects

I tested it with nested subprojects that use .wrap files:

https://github.com/meisenzahl/meson-dependency-test

Try it out to see if it solves your issue.

Ericson2314 commented 5 years ago

@jpakkane I was thinking I think the name "subproject" might make people continue to ask for recursive subprojects. If instead we had just projects and one "metaproject", I think people might get the point better.

Qix- commented 5 years ago

@Ericson2314 it's not about terminology, it's about functionality. I've used an army of build systems throughout my career and know exactly what I want. Meson gets in my way with its philosophy and dogma, so therefore I cannot use it.

Ericson2314 commented 5 years ago

@Qix- I have worked on an army of build systems in mine, and I agree with @jpakkane that hierarchical subprojects are indeed wrong. Depedencies form a directed acyclic graph, not a tree. Your example with the two executable is what we call a private dependency. Private dependencies are the exception, not the norm, and trees/hierarchies only work if every dependency is private.

I suggest you read about https://www.well-typed.com/blog/2015/03/qualified-goals/ . Cabal, Cargo, NPM, etc are solving build systems in that you must decoratively state your dependencies so that the dependencies can be simultaneously choosen. [Meson has one imperatively state dependencies, which is arguably a mistake as it makes version solving a lot harder.] The "qualified goals" on executables and is basically equivalent to saying that executables only have private dependencies, since nobody links with them.

Cabal also has one "project" which contains "packages" which must have all their dependencies reconciled. This matches my project vs metaproject proposal.

Qix- commented 5 years ago

@Qix- I have worked on an army of build systems in mine,

And I'm the pope. What's your point? Pissing matches don't belong on Github, take them elsewhere.

Cabal, Cargo and NPM are not build systems, they are package managers. I wouldn't expect a build system to solve versioning constraints. I also don't feel build systems and package managers are one and the same, nor should they be.

Your link describes version conflicts, which are NOT analogous to linkage conflicts. A build system is not solving a system of constraints, but instead solve a system of dependencies. Those are very much different domains and problem spaces.

Let me repeat: your build system should not be solving systems of versions. This is a large reason why I moved away from Meson because it's scope creep. There's a reason why Aptitude, brew, npm, etc. don't dictate underlying build systems (I could use automake, cmake, ant, maven, etc. for any or all of those) - so your point doesn't really make sense to me.

This is also why projects like Glide exist - because Go literally has package management built into the language, which people have had to re-learn is bad, it seems. This is also why people tend not to like Cargo to manage their Rust builds and turn to other build systems entirely.

Instead of telling me what is/is not allowed based on your own philosophies, I just need a tool that lets me do my work and doesn't force me to shoot myself in the foot later on down the road. Meson is not that. I have my own philosophies (clearly) so let me operate within them.

Ericson2314 commented 5 years ago

My comments on working on multiple build systems and on imperative dependency specification perhaps being a bad idea weren't supposed to kick off a pissing contest and change the subject, respectively, but rather try to offer some evidence that I'm not some fanatic with blinders that doesn't know anything but Meson and also thinks that Meson is already perfect.

If you want a strict separation between package managers and build systems as those are currently defined, the meson shouldn't have sub-projects at all.

If you want to keep talking a bit off talking on solving and the division in labor, OK consider me now nerd sniped :).

your build system should not be solving systems of versions.

I actually agree. I like that Ninja does all the building and nothing else. I use NixOS and Nix (http://nixos.org/) and someday hope to write a Nix backend; Nix also does no solving.

The issue is configuration vs solving. I think all build-time inspect-the-environment configuration is a workaround the solver not knowing enough statically. If you top-down decide everything, packages should build directly from those assumptions, and what was configure steps instead become tests ensuring the system as assumed matches what was built.

Sub-projects allow one to short-cut environment detection in that a depency required by one project is required by the other, and the configure step is OK with that even though the dependency isn't yet built and cannot be detected the usual way. They also allow for cross-project incremental rebuilds, which is very useful for interesting cross-project refactors. In short 2 things: incremental builds accross boundaries and transposition all configuration before all building.

Take that to its logic limit in the context of something like NixOS and you are building entire machines images with compiler-invocation granularity. Since there can be no environment detection when everything is built from scratch, everything has to be top-down specified. If you pin versions for everything then like NixOS you don't need to solve. If you don't wish to pin versions (or more generally the declarative requirements don't fully constrain the build plan), then some solving naturally comes into play because there are potentially multiple unique solutions and requirements need to be combined.

If you want a separate solver, "configurer" (Meson/Cmake), and builder (Ninja/CMake/Bazel/..), what is left for the configurer to do in this stage? Just be a pure function from what the solver decides to some Ninja? Part of Meson is environment detection and the other part is domain-specific knowledge about how to invoke compilers. So just the other part is needed.

So that's a vision of a far future. Nowhere between now and then do I see a good use for hierarchical subprojects. Subproject combining is always package management in that you are deciding how to resolve dependencies. And it always precludes that dependency from being used by anything else. How many private dependencies do you have exactly?

Instead of telling me what is/is not allowed based on your own philosophies, I just need a tool that lets me do my work and doesn't force me to shoot myself in the foot later on down the road. Meson is not that. I have my own philosophies (clearly) so let me operate within them.

Perhaps you have your own opinions which is good, but it seems that many people have no regard for their build system and just hack things up until it works. (Perhaps it's not their fault, but something they learned from decades of terrible build systems). That usually leads to hard to understand projects that don't compose very well, so I rather the build tool be more opinionated.

Qix- commented 5 years ago

My comments on working on multiple build systems and on imperative dependency specification perhaps being a bad idea weren't supposed to kick off a pissing contest and change the subject, respectively, but rather try to offer some evidence that I'm not some fanatic with blinders that doesn't know anything but Meson and also thinks that Meson is already perfect.

My mistake, apologies.

If you want a strict separation between package managers and build systems as those are currently defined, the meson shouldn't have sub-projects at all.

I don't see how that follows - using git submodules, for example, leaves the "package management" element to Git and would allow Meson to work seemlessly with recursive submodules.

I think all build-time inspect-the-environment configuration is a workaround the solver not knowing enough statically.

I still don't see how this has to do with file hierarchy, seeing as how those subprojects would be present at the initial configuration stage.

Sub-projects allow one to short-cut environment detection in that a depency required by one project is required by the other, and the configure step is OK with that even though the dependency isn't yet built and cannot be detected the usual way.

I still cannot fathom how this is limited by what you call "recursive submodules", nor how it's fixed by enforcing a single level of subprojects.

They also allow for cross-project incremental rebuilds, which is very useful for interesting cross-project refactors. In short 2 things: incremental builds accross boundaries and transposition all configuration before all building.

Has nothing to do with nested subprojects, I'm not even sure what this is talking about.

Take that to its logic limit in the context of something like NixOS and you are building entire machines images with compiler-invocation granularity. Since there can be no environment detection when everything is built from scratch, everything has to be top-down specified

You're assuming a lot about projects, and making generalizations that simply are not true. When building an OS it is entirely common to use packages that are intended to be used by non-standard platforms and thus provide a number of configuration variables to specify how they should be built. This is no different than how CMake handles it - you can absolutely specify the cross-compilation toolchain, and even override sub-sub-subproject target flags/settings/etc, even at the file level.

What you're describing is already possible with existing build systems that also allow nested subprojects. Meson is creating an artificial limitation here out of stubbornness.

If you pin versions for everything then like NixOS you don't need to solve.

Again, I don't see how "versions" have anything to do with anything. Build systems should look at the filesystem and be configured by the file system and build whatever you tell it to build. I don't see how versions have anything to do with Meson if you don't use Meson for managing your dependencies.

If you don't wish to pin versions (or more generally the declarative requirements don't fully constrain the build plan), then some solving naturally comes into play because there are potentially multiple unique solutions and requirements need to be combined.

If you are entertaining the idea that build systems should care about building multiple versions (in build system terminology they're called "variants") then this is also already done by many build systems just fine, even with nested subprojects. I'm not sure where you get the idea that this is somehow impossible.

If you're referring to dependency management, something your build system shouldn't care about, then it should at the very most warn/error about conflicting errors depending on how the packages are going to be used.

If I have compile-language that uses libcliarguments@2 and then I want to use compile-language as a command line utility to build something.mylang which also links to libcliarguments@3, and I want to make something.mylang depend on compile-language so that the build step can actually use the compile-language executable, then Meson should know that even though libcliarguments has conflicting versions they won't actually conflict because we're not linking them into the same executable and thus there will be no risk of symbol shadowing or conflicts.

This sort of configuration happens all the time and the fact Meson cannot refuses to do it simply baffles me.

Take that to its logic limit in the context of something like NixOS and you are building entire machines images with compiler-invocation granularity.

Tup seems to do this just fine.

If you want a separate solver, "configurer" (Meson/Cmake), and builder (Ninja/CMake/Bazel/..), what is left for the configurer to do in this stage?

Generate the ninja build files. CMake doesn't build anythinng, and I'm not sure where Bazel comes in because it does both (and subsequently sucks at doing both FWIW). Ninja is the only thing creating a pseudoterminal, forking, and execing.

A build system comprises both the configuration and the build, hence why "CMake" is widely considered a build system even though it doesn't actually run the builds.

I don't really understand the question.

Part of Meson is environment detection and the other part is domain-specific knowledge about how to invoke compilers. So just the other part is needed.

That's all fine and well, but I seriously do not see how this is at-all related to the discussion.

So that's a vision of a far future. Nowhere between now and then do I see a good use for hierarchical subprojects.

You haven't even addressed subprojects..... I simply do not understand your point. With all due respect, your response is a word soup of nonsense. Please clarify your points.

Subproject combining is always package management in that you are deciding how to resolve dependencies.

Yes, and as the developer of the project, I'm pretty certain I can understand the dependencies that I am consuming just fine. We've been doing it for decades without issue.

And it always precludes that dependency from being used by anything else.

That's a bizarre conclusion, one that is objectively false...

How many private dependencies do you have exactly?

I do not understand this question, as it makes no sense.

Perhaps you have your own opinions which is good, but it seems that many people have no regard for their build system and just hack things up until it works. (Perhaps it's not their fault, but something they learned from decades of terrible build systems). That usually leads to hard to understand projects that don't compose very well, so I rather the build tool be more opinionated.

It's not that difficult to understand. Meson does a lot of things right but falls short of being a real build system.

Here's a great resource that is packed full of falsehoods about compilation and build systems. I don't need a bible, I need a tool.

https://pozorvlak.livejournal.com/174763.html

Here are a few highlights:

  1. Build graphs are acyclic.

  2. In particular, said language does not need a module system more sophisticated than #include.

  3. System libraries and globally-installed tools never change.

  4. Everything lives in a single repository.

And from the comments, another gem:

  • Users will only will only use one specific compiler, library and flags, so you can hardcode them in your build scripts.

I do not understand why build systems need to force their religion down my throat.

Ericson2314 commented 5 years ago

@Qix- I still don't know what problems nested subproject solve for you. I only see them luring users in general---if not yourself---into doing terrible things. If you really think there's something we're missing that would make me change our minds, go find me on IRC or something.

Qix- commented 5 years ago

Please go read my earlier comments, I explain everything in detail.

Meson's job isn't to tell me how to build my projects. Meson's job is to build my projects.

dcbaker commented 5 years ago

This is going nowhere fast. I think everyone's clearly stated their positions on the matter, and no one seems interested in changing them. I'm going to close and lock this thread.