[Proposal] Mutability of Targets & Dependencies

[marstaik]

The current meson ideology is rather stringent and requires the use of tons of global variables (especially with multiple libraries) to add not only sources files, but compiler flags in sub directories that may need some sort of control over the library being built. Throw in optional build sections, and you end up getting build files that are usually much more complex than they need to be.

Here is an example:

meson.build
foo/
    meson.build
    bar/
        meson.build
 car/
     meson.build

meson.build:

foo_cpp_args = []
foo_srcs = []
foo_libs = []
foo_public_cpp_args = []

subdir(foo)
subdir(car)

lib_foo = library("lib_foo", foo_srcs, link_with: foo_libs, cpp_args: foo_cpp_args)
dep_foo = declare_dependency(link_with: lib_foo, compile_args: foo_public_cpp_args)

foo/meson.build:

foo_srcs += [ 'foo.cpp']
subdir(bar)

foo/bar/meson.build:

foo_srcs += ['bar.cpp']
if get_option('use_lib_z'):
    foo_srcs += ['libz_adapter.cpp']
    foo_libs += find_library('libz')

car/meson.build:

if get_option('car_enabled'):
    car_srcs = ['car.cpp']
    car_cpp_args = []
    .... # whatever else car might need, exposed out as variables for subdirectories
    subdir(...) # whatever subdirectories are needed, if any

    lib_car = library('lib_car', car_srcs, cpp_args: car_cpp_args, ...)
    foo_libs += lib_car
    # Car enabled needed for both building foo, and for any dependencies that are using foo...
    foo_cpp_args += ['CAR_ENABLED']
    foo_public_cpp_args += ['CAR_ENABLED']

Note: I realize you can make dep_car with the public CAR_ENABLED compile argument added, and then have lib_foo depend on dep_car. But let's face it, that doesn't make it better, if at all.

Problems with the above:

• Its difficult to simply search for everything related to lib_foo. You have to find lib foo, then find the variables it is referring to for cpp, src, libs, etc, and backtrack those individually to see what goes into foo. The variables don't add any meaning either, they simply another mental and physical redirection that has to be done to get to what you want.
• The declaration of library foo is in a directory level above where it needs to be, unrelated to where the majority of its source files are.
• If we have another library down the line with a configure time check, we cannot easily add a define to lib_foo, among other things. So every time we have to do this, we have to bring up the library to a higher meson.build level, and expose another global variable for a downstream directory to mutate.
• As the number of library's increase, so do the number of variables polluting the files. Not only that, they get progressively longer and longer, with more mental overhead added.

Lets compare the above to say:

meson.build
foo/
    meson.build
    bar/
        meson.build
 car/
     meson.build

meson.build:

subdir(foo)
subdir(car)

foo/meson.build:

lib_foo = library('lib_foo')
dep_foo = declare_dependency(link_with: lib_foo)

lib_foo.srcs += ['foo.cpp']

subdir(bar)

foo/bar/meson.build:

lib_foo.srcs += ['bar.cpp']
if get_option('use_lib_z'):
    lib_foo.srcs += ['libz_adapter.cpp']
    lib_foo.link_with += find_library('libz')

car/meson.build:

if get_option('car_enabled'):
    lib_car = static_library('lib_car', ['car.cpp'])

    subdir(...) # whatever subdirectories are needed, if any

    # Car enabled needed for both building foo, and for any dependencies that are using foo...
    lib_foo.link_with += lib_car
    lib_foo.cpp_args += ['CAR_ENABLED']
    dep_foo.compile_args += ['CAR_ENABLED']

• I can now simply search for lib_foo and find ALL relevant areas that touch foo, instead of having to find the exposed variables
• lib_foo is declared where it is expected to be, in the foo folder
• Optional build requirements for building lib foo WITH lib car are centralized with lib car.
• The only variables introduced in the entire build are 'lib_foo' and 'dep_foo'

This system would also be backwards compatible as well, as you can still declare srcs, cpp_args, etc at library and dependency declaration.

[jpakkane]

The short answer:

No. This will not happen. Ever. Targets are immutable and there is nothing anyone in the known universe can say that will change it. Full stop.

Any further discussion on this will be ignored and not replied to.

The longer answer:

The fundamental issue you have here is that there seem to be files for the library foo that are stored completely outside the foo lib (in this case the car directory). The sources (but not dependencies) that make up the foo lib must be within the foo subdirectory. The only real way of creating scalable systems is to build isolated components and combine them. This is how Meson subprojects and external work and why it's possible to mix and match between them transparently.

In this particular case what you probably intend to do is to link the result into something else, say exe. In this way the code for lib would stay in its own thing and the extra code in car would either be put in the exe target directly or built into its own static library that would then be linked in the exe target. If the contents of car are integral to the functioning of foo, then they should be put under the foo subdirectory directly.

[marstaik]

I honestly don't even think you went through the post, and are instead making assumptions. car has no source code that gets injected into foo, there is only a define being added to foo telling it that it is linking with car.

In this way the code for lib would stay in its own thing and the extra code in car would either be put in the exe target directly or built into its own static library

Did you even bother to read the example before posting?

You have not discussed any of the points I brought up above, and have made incorrect assumptions about the post and the example given.

• You did not mention/counter issues of project file readability.
• You did not mention/counter issues of hoisting unnecessary global variables.
• You did not mention/counter issues of the above points becoming extremely excessive difficult to maintain in large projects.

In large projects where many lateral components have to be integrated together depending on build options and configurations, there are many configuration steps, you made no address to the issue of having to essentially hoist a bunch of global variables into top level meson build files.

When building large projects that are composed of a large amount of libraries, that can be switched between shared and static libraries, it gets extremely difficult when you need to start breaking up configuration/settings that could be cleanly placed in one spot into several.

Again, no address of that either.

So at this point it seems like you just don't care. There is just more and more "my way or the highyway" mentality that just drives people away. Instead of discussing the merits/demerits of such an approach, you directly place the blame on project structures that don't fulfill your ideal criteria, whether you have merit or not.

It's a proposal, meant to be open to discussion. It seems that that is simply not allowed in this project.

[xclaesse]

@marstaik To be fair, I'm with @jpakkane on this, mutability is not going to happen, that's a key design choice that will not change.

I think the issue you have here is you try to present a solution instead of presenting a problem. You should formulate the issue you have and we can see what are the possible solutions.

That said, I did not read the full description yet.

[xclaesse]

From a quick look, it sounds to me that car and bar should be internal static_library() and foo link to those static lib.

[marstaik]

I have presented a problem, in as condensed a form as I reasonably believe I can. The larger a project gets, the messier and messier the build files become with explicit initialization of libraries needed to link against, and defines for those projects down the line.

From a quick look, it sounds to me that car and bar should be internal static_library() and foo link to those static lib.

Yes, you can invert the flow in this example and it would be alright, but that is not always the case. (edit: If I had added the fact that car needs to link against foo, how would you fix it? Its possible, but it requires the hoisting of more variables outside of their logical "context area")

In many cases I have encountered where you try and break apart a large solution into smaller, coherent pieces, either for code maintainability or so that you can build those pieces as shared libraries to accelerate rapid development, There are cases where lib_b depends on lib_a, but lib_a has to be informed to hit some entry point of lib_b via a define. This is extremely common in cases where you have an extremely large project with smaller sub modules that can be enabled and disabled. In such a case, lib_b depends, and must link against lib_a, so you cannot invert the flow.

Projects don't always build small libraries meant to be neatly installed onto a system. And some projects also want to do both at the same time.

[dcbaker]

Note: I realize you can make dep_car with the public CAR_ENABLED compile argument added, and then have lib_foo depend on dep_car. But let's face it, that doesn't make it better, if at all.

We use this exact approach in mesa, and it's honestly eloquent as heck. We have an N:M system, N language frontends (OpenGL, OpenCL, Glide, DirectX 9, etc) to M backends (2 Intel, 3 AMD, 1 Nvidia, 1 mali, 1 qualcomm, etc). We create an installed library for each frontend, which must be linked with each backend, compiled as a static library. A backend creates a driver_FOO = declare_dependency(link_with : libFOO, c_args : ['-DENABLE_FOO']) which is controlled via:

if with_intel
    subdir('intel')  # creates driver_intel_current
else
    driver_intel_current = declare_dependency()

Then the driver looks like:

driver = shared_library(
    'driver',
    'driver.c',
    dependencies : [driver_intel_old, driver_intel_current, driver_amd_really_old, ...]
)

This makes the build system very easy to figure out, specifically because you know that driver_intel_current came from, It's either in src/drivers or src/drivers/intel_current, and you don't have to go hunting for it.

And we make use of internal deps everywhere because it's actually a really good way to structure complex build systems. Our common midlevel compiler, NIR, needs to be linked with our utils library when used, and libnir, and has a number of generated headers you need to depend on. We used to have a ton of problems with people forgetting to add a nir header dependency to their driver, until we switched to this:

idep_nir = declare_dependency(
    link_with : [libnir, libmesa_utils],
    sources : [nir_opcodes_h, nir_builder_opcodes_h, nir_intriniscs_h]
)

This is super awesome too, because when a new generated header is created it's added in one place, just the idep_nir. Which means that someone working on NIR doesn't need to check every driver to ensure that they update the dependency correctly, the number of stable backports I needed to apply at the release manager dropped significantly.

We also have a high level compiler wiht the same structure (called glsl), Intel has a backend compiler shared between the two intel drivers, there's core abstraction bits used to pipe between the frontends and the backends, and all of those back use of declare_dependency because it's really convenient and robust.

And just for reference, Mesa had (last year) about ~20,000 lines of meson.build files. So not Linux, but not insignificant.

[ilg-ul]

about ~20,000 lines of meson.build files

An impressive number. However, as I provide both CMake and meson configurations for my projects, I can state that, for my projects, meson files are generally larger and require more effort to write (the main reason being the lack of macros/functions, which results in lots of redundancies, and immutability, which results in lots of variables that need to be appended before making the final object creation); thus I'm not sure that bragging on meson line number is necessarily a good thing. :-(