Allow configure_file, custom_target and generator output file to subdirectories under builddir

[jasonszang]

I've encountered two cases that would require configurating / generating file into subdirectories.

1. Project configuration header that would be included from other headers in the same project

Say I have a library project named foo and the project layout is like this:

include/
    foo/
        bar/
            foobar/
                foobar.h # includes config.h
            bar.h        # includes config.h
        foo.h            # includes config.h
        config.h.in # configured by meson
src/
    foo/
        foo.cpp
...

We would want our client code use #include <foo/config.h> to include our config.h since we don't install config.h directly to includedir. But for our own code this would be impossible since generated config.h is directly under <builddir>, not <builddir/foo>.

Of course, for foo.cpp the work-around is easy: just #include <config.h>. But it becomes complicated we need to include config.h from bar.h and foobar.h.

In fact, there is no way to correctly include config.h that both work in project directory and after installation. If we want bar.h and foobar.h to be correct after installation we must #include <foo/config.h> or use relative path in them. Including <foo/config.h> won't compile, and although relative path happens to work for bar.h, it won't for foobar.h. Neither will it work if we reside in a subdir. Using #include <config.h> allows us to compile, but bar.h and foobar.h breaks up after installation.

The correctly way to configure and include config.h should be just generate it under builddir/foo and #include <foo/config.h> everywhere, just as if it were in the position of config.h.in. Currently meson does not allow this, and forces a flat include directory, or hacking into builddir.

2. Protobuf generation

This one is simpler. If we have some protos like

protos/
    foo/
        foo.proto
    bar.proto

in which bar.proto imports foo.proto. it would be nice if we could use one generator for both protos and and get under build directory something like this: For the compiled protos to work the directory structure of generated sources must mirror the protos, because bar.pb.h will #include "foo/foo.pb.h". The generated sources must look like this:

protos/
    foo/
        foo.pb.h
        foo.pb.cc
    bar.pb.h
    bar.pb.cc

Althouth this would also require something like @DIRNAME@ substitution if we are to pass generated sources to build targets properly, but without supporting generating source trees we cannot use protos with subdirectories. This also prevents wrapping of existing projects that use protos this way, which is quite a common practice.

PS: I suggest putting every generated file from configure_file() and custom_target() (including ones in subdirs) into one directory tree named something like <builddir>/sources@gen, add it to include directory, and allow outputing to subdirectories under it. This should avoid potential name clashes and give developers control over generated folder layout.

[jpakkane]

I could refactor out some of the shared Meson code to some higher-level meson.build, but that still leaves me with ~70 lines per test.

This still seems a bit much. Hypothethically you should be able to do something like:

test_dirs = ['foo', 'bar', 'baz']

foreach d in test_dirs
    subdir(d)
    shlib = shared_library(d + 'enclave', common_sources, generated_sources,
                                       include_directories: generated_inc)
    exe = executable(d  + 'test', <similar as above>)
    test(d, exe, args: [shlib])
endif

Where each subdirectory would have a meson.build file like this:

generated_sources = custom_target(...)
generated_inc = include_directories('.')

In this way all your targets are defined in one place and you get at least some degree of protection with the generated headers. If your setup is even more complicated than this, then I'd need some more details to help.

[mkow]

I could move just the custom_target invocation to the subdirectory as you suggest and end up with ~15 lines, but it depends on a bunch of variables from the meson.build which builds the tests, which I'd need to pass implicitly through global scope (including the iterator, to make the name of the custom_target unique). So, I still have 15 lines * tests_number copy paste, plus my build code would be quite unreadable because of that implicit parameter passing. What this approach is doing is actually trying to hack around the output subdir restriction by cutting a bunch of lines from the foreach and calling them inside the subdir.

Also, please take a note that as I said, changing the framework can't really help here. It seems that the only clean way to implement this is to allow tests/meson.build to generate files in subdirectories.

[touilleMan]

Hi there, I also ran into this problem so I'd like to share my usecase ;-)

I'm writting a Python binding for the Godot game engine. Most of the code is written in Cython and there is plenty of code generated from templates.

The final (installed) data are:

libpythonscript.so
_pythonscript.so
godot/_builtins.pxd
godot/_builtins.so

The (simplified) build hierarchy is:

libpythonscript.so
├─ pythonscript.c
└─ _pythonscript_api.h
   └─ _pythonscript.pyx  # Cython compiles x.pyx into x.c and x_api.h
   └─ godot/_builtins.pxd
      └─ generate_builtins.py # Script to generate the .pxd from a template

_pythonscript.so
├─ _pythonscript.c
|  ├─ _pythonscript.pyx
|  └─ godot/_builtins.pxd
|     └─ generate_builtins.py # Script to generate the .pxd from a template
└─ libpythonscript.so  # Needed at link time given it contains exported symbols

godot/_builtins.so
├─ godot/_builtins.pyx
|  ├─ godot/_hazmat/gdnative_interface.pxd
|  └─ generate_builtins.py # Script to generate the .pxd from a template
└─ libpythonscript.so  # Needed at link time given it contains exported symbols

One important note is Cython import mechanism requires the .pxd files to be in a correct path in order to be specified in a .pyx (e.g. cimport foo.bar requires that there a foo folder which contains a __init__.py and a bar.pxd)

So:

• In order to have the generated godot/_builtins.pxd file in the correct location I must put it generation code in godot/meson.build
• In order for godot/_builtins.pyx to find godot/_hazmat/gdnative_interface.pxd, it compile code must also be put in godot/meson.build
• Given godot/_builtins.pxd is needed to build _pythonscript.so, the root meson.build must first include godot/meson.build to define the dependency then have it code compiling _pythonscript.so using the dependency
• However godot/_builtins.so depends on libpythonscript.so which depends on _pythonscript.so, so the root meson.build must first contains the compile _pythonscript.so code then include godot/meson.build :'(

The issue come from the fact a folder can have only one meson.build and this file is evaluated in a single pass, hence it's not possible to simultaneously a target in a child folder using a dependency from a parent folder and vice-versa.

So a very simple solution to this trouble would be to be able to generate the .pxd file first by having it code at the beginning of the root meson.build, with an output target pointing to a subdirectory so that the file is in the correct place to be use for the .pyx compilations.

[tristan957]

@mkow could you do something like this: https://github.com/hse-project/hse/blob/master/tests/unit/meson.build#L1

[tristan957]

@touilleMan your problem would be solved by structured_sources() supporting Cython, I think. It was hard to follow. CC @dcbaker

[touilleMan]

@tristan957 thanks a lot for the answer, I didn't try structured_sources given the documentation says it is only allowed in Rust targets.

Assuming it would support Cython (as a matter of fact I'm a bit puzzled why this function is not language-agnostic), this would still require to build the .pxd in the root folder which may cause name clash :/

[mkow]

@mkow could you do something like this: https://github.com/hse-project/hse/blob/master/tests/unit/meson.build#L1

@tristan957: Not sure how this helps (I assume you mean putting tests arguments in an array and then iterating over it), I still need to somehow generate the code to the subdirectories of each test.

[tristan957]

Might help reduce your copy/paste even further.

[mkow]

Nah, this won't help (and I'm actually already doing something similar). The technical details are that I need to invoke the codegen twice for each test, and pass a bunch of arguments to it.

[rulatir]

which takes a single input file (.edl) and outputs enclave_t.h, enclave_t.c, enclave_u.h and enclave_t.h (these names are hardcoded, but I can pick the output directory)

The sad and unfortunate fact is that this is just plain poor design. If you have a system that forces people to generate multiplie headers with the same name within a single project then it is broken because Murphy's law says that they will get mixed up. It is unavoidable.

Arbitrary, rigid opinions like the above are precisely what "opinionated" means in meson.

[plattfot]

Say I have a library project named foo and the project layout is like this:

include/
    foo/
        bar/
            foobar/
                foobar.h # includes config.h
            bar.h        # includes config.h
        foo.h            # includes config.h
        config.h.in # configured by meson

Given you have a directory include/foo/ which contains foo.h, you could have it contain meson.build which generates the configure_file too, and meson's builddir will generate <build_dir>/include/foo/config.h.

You then add include_directories('..') and, include <foo/config.h> magically works everywhere. You don't even need to change directory structure in your project.

Depending on other project layouts that aren't the badly-chosen example in the original report, you can still do the exact same thing BUT it would require tweaking your project layout to include the same tree layout in your source directory too. On build systems supporting in-tree builds, running make/ninja would have created that anyway...

One could argue that it's not nice to be forced to create a source directory layout for this (I don't personally find this bothersome, but YMMV), but it cannot be argued that "it's impossible".

The other example in the original report, protobuf generated headers, was implemented shortly after this report was submitted, via #2764

Generator output files now support subdirectories under builddir, to match their subdirectories under sourcedir. And the testsuite includes protobuf sources testing this functionality.

I encountered a similar issue today, as others in this thread, while trying to generate a version.hpp file for a C++ project. The project contains a small library (let's call it foo) and an executable (let's call it bar). Where I wanted the version.hpp to be placed under foo/ so files in foo and bar could use #include <foo/version.hpp> to include it. My mistake was that I was trying to generate this from the top meson.build file. After reading through this thread, I found @eli-schwartz's message and I realized my mistake.

To make it work I created a meson.build at the foo level and generated my version.hpp there. After that it worked flawlessly.

Maybe the Configuration docs should be updated to either mention this or have an example? It was not clear to me when reading it or the reference docs for configure_file.

[d3vik]

I also ran into this problem while integrating protobuf - wasted a few hours trying out different solutions and ending up with a not so beautiful workaround.

@jpakkane, I get why you prefer "easier debugging" and not generating structured output, but an inflexible build system like this is have no real use when you have to integrate with all kinds of tools that you cannot control (protobuf being just one example). Things like this just makes people waste time in understanding limitations and figuring out workarounds - In this case I would without doubt have preferred debugging a meaningful error and learning from mistakes.

I really hope that this "feature" will be reconsidered. Some may not understand the implications of the workaround they choose to develop for getting around this issue. This can easily cause problems that are even harder to debug than the ones Meson are trying to avoid. For example in the case where sources are not correct generated/rebuilt after an edit because you go around the build system. In the case of protobuf some may also end up choosing the simplest fix; To place the files generated by protobuf within the sources - I believe that is much worse than having a subdirectory in the build directory.

By the way, I would like the add that I in general find Meson to be a fantastic build system. I would really prefer not being forced to look for something else.

EDIT: Just after posting I found the following (I have no idea why I did not discover that argument a few hours earlier): https://mesonbuild.com/Release-notes-for-0-45-0.html#generator-outputs-can-preserve-directory-structure This solves all my issues while working with protobuf. I now have a much cleaner solution!

I still believe that a build system to some extent should have some flexibility to support different methods and tools.

One may find this example code useful:

protoc = find_program('protoc-c', required : true)

protobuf_generator = generator(
  protoc,
  output : ['@BASENAME@.pb-c.c'],
  arguments : ['--proto_path=@CURRENT_SOURCE_DIR@/',
               '--c_out=@BUILD_DIR@',
               '@INPUT@']
)

protobuf = protobuf_generator.process(
  'example1/example1.proto',
  'example2/example2.proto',
  preserve_path_from : meson.current_source_dir()
)

[tristan957]

@needmoreram, have you been working on a protobuf module?

[needmoreram]

yes I have been, though not as quickly as I would have liked. it supports C++ only at the moment.

I've pushed what I have at the moment to my branch: https://github.com/mesonbuild/meson/compare/master...needmoreram:meson:ram/add-protobuf-module

comments and suggestions welcome :)

[tristan957]

It is looking fine. I have a few comments, but will wait for a PR to be made.

[horenmar]

I was fighting meson to do something that should be conceptually simple, like

configure_file(
  input: 'catch2/catch_user_config.hpp.in',
  output: 'catch2/catch_user_config.hpp',
  format: 'cmake@',
  install_dir: get_option('includedir') / 'catch2',
  configuration: conf_data,
)

where I just want to configure a file to be in the same subpath in the build dir as it is in the source dir. But after reading this issue, the way I am supposed to do is to drop another meson.build file into the subdirectory, just for the configure step?

[tristan957]

@horenmar, unfortunately this is the case. Though, it looks like you should just use the catch2 wrap. I believe it exists.

[touilleMan]

The generator with preserve_path_from proposed by @d3vik is interesting, however if the command doing the generation depends on other files (typically in case of a Python script made of multiple files) you need to use the depfile parameter.

This can be done by having the command has to always spit the same depfile containing the list of it own dependencies, but it is a big hack (the script should have to deal with this kind of dependencies relationship, it's the build system job).

Is there a reason why this depfile parameter cannot take a static list of file dependencies (instead of the path of file that may contain the list of file dependencies) ?

EDIT: generator usage is limited to input of build_target(), cf meson documentation:

If you want to generate files for general purposes such as for generating headers to be used by several sources, or data that will be installed, and so on, use a custom_target() instead.

So the generator with preserve_path_from trick is not one-size-fits all workaround for custom_target missing a way to specify outputs in a different directory :/

[eli-schwartz]

That is specifically the purpose of the depends: kwarg.

Depends are for static lists of file dependencies known when writing the build file.

depfile is for when the list of dependencies isn't known at configure time, for example if they are external to the build system but the program that compiles/generates the output knows the list of files as a side effect of processing them. They are never needed on the first build, but are only needed to ensure that incremental builds correctly mark the output as stale and rebuild them. C/C++ compilers, for example, know how to emit these depfile dependencies for header files including those that come from the system, and a number of other popular tools natively support the same concept.

[touilleMan]

@eli-schwartz

That is specifically the purpose of the depends: kwarg.

Yes, but depends takes a list[build_tgt | custom_tgt], so you cannot specify static files :( For instance, custom_target has both depends and a depend_files, the former taking list[str | file].

So I guess my issue would be solved if generator had a depend_files parameter just like custom_target has...

@jpakkane would you accept a PR for ?

EDIT: adding depend_files to generator would not solve my issue (given generator should not be used to generate file that are meant to be installed) :/

[tristan957]

@touilleMan regardless of whether it solves your problem, there is still value in depend_files for generator.

[mserdarsanli]

I've just decided to try meson on a project of mine for the first time and hit this issue. Seems like this really makes it hard to use meson on projects with lots of codegen, as the official recommendation is cluttering your codebase with many meson.build files.

Admittedly I didn't really know much about the design of meson and it makes a bit more sense once I read this discussion. For example I realized that below snippet on a meson.build file

foo_exe = executable('foo',
    ['src_foo.cpp', 'src/foo.cpp'],
)

will fail on configure step with error:

ERROR: Multiple producers for Ninja target "foo@exe/src_foo.cpp.o". Please rename your targets.

as meson tries not to follow original directory structure, and put every generated thing on the same directory.

This is mostly fine, as noone cares about the location of the object files, they will link fine regardless. But codegen is obviously different, as you will often want generated headers to follow original directory structure to prevent conflicts and to #include them via their anticipated paths.

I think suggestion by @jasonszang in the original report, of using <builddir>/sources@gen and allowing codegen to any path inside that directory could work well, object files could still go to the directory meson deems appropriate.

I'm guessing this is a big hurdle for monorepos to switch to meson, and fixing this issue can remove that.

[fluidparadigms]

I'm surprised that this is still an issue after almost 6 years. After reading through the thread, it seems...

1) that most of the thread consists of the following:

• frustrated complaints and/or suggestions
• examples of situations where the users have little-to-no control over the decision to reorganize the code in question
• partial workarounds to get past the issue
• links to closed duplicates of this issue

2) that the only clear justification for not allowing the requested functionality is this:

The sad and unfortunate fact is that this is just plain poor design. If you have a system that forces people to generate multiplie headers with the same name within a single project then it is broken because Murphy's law says that they will get mixed up. It is unavoidable. This does not really help you, but I'd figured I'd write it out explicitly so that it is absolutely clear for everybody.

@jpakkane I understand your reasoning and I plead with you to reconsider based on this: I've maintained build systems for years, as I'm sure you and many others here have. I assume that poor design is something we've all dealt with. Regardless of it being a poor design, this design pattern seems to be a common one. It is also a pattern that could be impossible to remedy without substantial changes to a codebase or the use of kludgy workarounds. What if a user is attempting to convert an existing third-party product (Apache Xerxes is one example I've run into) to use Meson? or a legacy codebase? or code written by a senior developer? or proprietary code with an auto-generated interface layer? If a developer or DevOps engineer approaches a manager or senior developer and asks, "Do you mind changing your directory structure to comply with the rules of the Meson build system?", I suspect they're going to be told to "buzz off" and/or "No. Find a new build system." (among other similar replies).

I may be wrong, but it seems like most of the frustration in this thread is based on what seems to be the withholding of a simple solution to a simple problem. For my use case, Meson solves a great many problems and I've had a hard to fight to win people over from using CMake or our existing homebrew build system. In my experience no one wants to learn or use anything new unless they absolutely have to. Especially when the new thing is the result of someone else's change. Also in my experience, developers can be quite territorial (which creates all sorts of problems). Sure, there are unlimited ways around the limitation, but making this change would make things so much easier, cleaner, and more elegant for those of us who aren't at liberty to fix others' poor design choices.

Again, this is all based on my experiences and may not apply in the wider world.

I say all this to ask if mirrored subdirectories would be such a terrible thing to allow? Perhaps to add a new directory in the build dir that is solely used for configure_file() targets and mirrors the original directory structure by default? Whatever the solution might be, I am sure that it would be a great help to many people (based on the commenters here and on duplicate issues). I know it'd make my own life less painful.

[eli-schwartz]

Shower thought: perhaps we could allow directories in the output filename, but only as long as that output directory segment

1) is never descended into using subdir() 2) does not contain a ../ component, it's a pure child of the current subdir

This would solve e.g. @jpakkane's concern that:

In Meson all outputs go to the build directory that corresponds to the source directory they are declared in. This simplifies debugging build problems, since you always know that files in build directory X must come from the corresponding source dir X.

... because you know that files in build directory path/to/X/ are always defined in the immediately sourcedir-adjacent meson.build or a straightforward walk up the tree to find the first meson.build.

I suspect that this would also make more or less everyone who currently wants to create outputs in directories, happy. It easily allows conforming to existing limitations of existing codebases without creation of excessively nested meson.build files, permits some clever foreach looping of outputs, unlocks a few use cases such as @keith-packard's current experiment in #11954, etc.

[tristan957]

That would be a great loosening of the restriction. I did like your prior functools.partial idea too.

[rulatir]

Whenever I feel the urge to try meson again, I just sit down in the corner and read this thread on my phone until the urge passes.

[fluidparadigms]

... without creation of excessively nested meson.build files ...

When I saw this workaround, I immediately thought, "But I'm trying to escape 'CMakeLists.txt'..."

[jpakkane]

1. s never descended into using subdir() 2 . does not contain a ../ component, it's a pure child of the current subdir

This is not actually sufficiently strict. Check 2 should actually check if the given subdirectory exists in the source dir at all and error out if it does.

This ties into extending custom_targets that I have been thinking about. Maybe it would be useful to be able to define a custom target that is a an opaque directory rather than a file. Which means you could do something like this:

ct = custom_taregt(..., is_opaque_directory: true)
some_source = ct.unsafe_grab_file('foo/bar/baz.c')
executable(..., some_source)

The user is responsible for ensuring that the file specified inside the output directory exists.

[eli-schwartz]

This is not actually sufficiently strict. Check 2 should actually check if the given subdirectory exists in the source dir at all and error out if it does.

Good point.

This ties into extending custom_targets that I have been thinking about. Maybe it would be useful to be able to define a custom target that is a an opaque directory rather than a file. Which means you could do something like this:

I don't believe this would be of any value for the use case I mentioned above about producing multilib libraries that have the same name, but different install directories. A solution that is specific to custom_target will inherently be a limited one.

[keith-packard]

I don't believe this would be of any value for the use case I mentioned above about producing multilib libraries that have the same name, but different install directories. A solution that is specific to custom_target will inherently be a limited one.

For static libraries, I agree -- the basename isn't relevant while building the library, the only requirement is that we be able to install the library to the correct subdirectory using the desired basename, which need not match the basename as built.

For shared libraries, those often do depend upon the basename used while building the library. I think all shared library builders have a way to specify the intended basename separately from the filename though, so it shouldn't matter in that case either. Someone more familiar with the range of shared library building mechanisms should validate this.

Of course, this technique would also solve the multilib issue, so if it is adopted, my need for pr #11954 would disappear.

[eli-schwartz]

I think all shared library builders have a way to specify the intended basename separately from the filename though, so it shouldn't matter in that case either. Someone more familiar with the range of shared library building mechanisms should validate this.

If meson explicitly handled this rather than relying on hacks like "library name is archdir/libXXX" :D then meson could just explicitly treat this the same way it handles actually defining the target in archdir/meson.build. So that shouldn't be a problem.

[eli-schwartz]

@jpakkane if there's consensus that this is a good idea to explore / implement with the restrictions mentioned, I can take a stab at it. But I need to know that it clears the design hurdle first...

[anarazel]

This ties into extending custom_targets that I have been thinking about. Maybe it would be useful to be able to define a custom target that is a an opaque directory rather than a file. Which means you could do something like this:

I think that'd be useful for custom targets that generate many files, e.g. some types of documentation generators. You can kind of do that today, but it doesn't work quite right everywhere today.

I suspect that this would also make more or less everyone who currently wants to create outputs in directories, happy. It easily allows conforming to existing limitations of existing codebases without creation of excessively nested meson.build files, permits some clever foreach looping of outputs, unlocks a few use cases such as @keith-packard's current experiment in https://github.com/mesonbuild/meson/pull/11954, etc.

It'd not make me entirely happy. If you have lots of small binaries for example, you might want to generate those in the directory corresponding to the source code, but without lots of redundant buildsystem code in each directory. The obvious way to implement that would be to have per-directory meson.build files that populate an array, and then higher up iterate over the array and generate the relevant targets. But the output-directory restriction doesn't allow that. Generating all the outputs in a top-level directory gets pretty messy, so also isn't really an option.

In Meson all outputs go to the build directory that corresponds to the source directory they are declared in. This simplifies debugging build problems, since you always know that files in build directory X must come from the corresponding source dir X.

The thing is, that I agree with that, as a goal. It's just that there's tension between a) the 1:1 source/build mapping b) buildsystem code duplication c) not allowing user defined functions. I think something gotta give.

Perhaps part of this concern could be addressed by allowing output into a subdir with a meson.build by having the subdir/meson.build doing something like output_here = current_directory() and then allowing targets above that directory to use that directory as the output? That'd allow to avoid duplication by creating targets in a higher up dir, while still leaving a trace in the build specification that something additional will define targets in the dir.

[keith-packard]

Perhaps part of this concern could be addressed by allowing output into a subdir with a meson.build by having the subdir/meson.build doing something like output_here = current_directory() and then allowing targets above that directory to use that directory as the output? That'd allow to avoid duplication by creating targets in a higher up dir, while still leaving a trace in the build specification that something additional will define targets in the dir.

That wouldn't help the case where the desired directory doesn't existing in the source tree (like the multilib setup). It would be nice to solve both the multilib issue and the subdir problem with the same mechanism, but if not, we can do them separately.

[eli-schwartz]

I think that'd be useful for custom targets that generate many files, e.g. some types of documentation generators. You can kind of do that today, but it doesn't work quite right everywhere today.

Indeed, you can use output: 'htmldir' and then just have the documentation builder build into that -- the problem is that ninja can recognize a directory as output, but not recognize much about it other than that it was created at least once. I'm not sure how an "opaque directory" approach solves that problem at all though -- ninja still can't tell much of anything about it.

The obvious way to implement that would be to have per-directory meson.build files that populate an array, and then higher up iterate over the array and generate the relevant targets. But the output-directory restriction doesn't allow that. Generating all the outputs in a top-level directory gets pretty messy, so also isn't really an option.

Right, this is currently not possible and while my proposal would allow it as long as the filelist isn't generated by populating the array via per-directory meson.build files (I honestly don't see the problem with hoisting the list up, or just using fs.read('subdir/files.lst')), @jpakkane's would forbid it. I am not sure how important that restriction is, and am not particularly attached to the idea of restricting it, but it was suggested by @jpakkane, so...

by having the subdir/meson.build doing something like output_here = current_directory() and then allowing targets above that directory to use that directory as the output? That'd allow to avoid duplication by creating targets in a higher up dir, while still leaving a trace in the build specification that something additional will define targets in the dir.

I don't think that would really solve the concern, because you would not know what that additional thing would be. Moreover, would this allow you to create the output from anywhere? e.g. given a project with two subdirs "sub1" and "sub2", could I define a current_directory() and then have sub1 produce outputs in sub2 instead even though it isn't a parent but rather a sibling? (Even worse: what if it's a nephew? Or a granddaughter?)

Sure, you could forbid it to be anything other than a direct parent but then you could also use my original suggestion and not use sentinel "I know what I'm doing" types.

[anarazel]

Right, this is currently not possible and while my proposal would allow it as long as the filelist isn't generated by populating the array via per-directory meson.build files (I honestly don't see the problem with hoisting the list up, or just using fs.read('subdir/files.lst'))

Hoisting the list up increases the likelihood of conflicts and also just leads to huge files. Using a different type of file that's then read with fs.read() doesn't really work, as there often are generated sources or such for a specific binary. It also leads to not having useful error messages and other such fun.

I don't think that would really solve the concern, because you would not know what that additional thing would be.

It'd at least be easier to search for...

Moreover, would this allow you to create the output from anywhere?

Sure. It's probably going to rarely be useful, but at some point restricting things makes them more complicated without a gain in clarity.

[eli-schwartz]

Sure. It's probably going to rarely be useful, but at some point restricting things makes them more complicated without a gain in clarity.

Let's assume for a moment here that we're talking about a build system that does restrict things, and we're discussing possibilities for compromising and loosening the restrictions a bit.

If we do make that assumption, it seems plausible to say that 6 years of arguing "restricting things is pointless" and getting nowhere isn't going to change today because someone said "restricting things makes them more complicated without a gain in clarity".

But figuring out a way to restrict things that meets the original design goal of the person who enforced a restriction, while still unlocking at least some use cases, feels useful and is, at a minimum, better than before. That's what I, personally, am interested in thinking about.

[eli-schwartz]

Using a different type of file that's then read with fs.read() doesn't really work, as there often are generated sources or such for a specific binary.

I can't imagine why it "wouldn't work". If there are generated sources for a specific binary, those could be generated alongside the target handling, and offloading the filelist to a dedicated file will still serve the purpose of greatly reducing the complexity, just not totally reducing it. A strict win over the current state of affairs, I would say.

It also leads to not having useful error messages and other such fun.

I'm baffled at why error messages would inherently become not useful as a result of this. In fact, I'm baffled at why error messages would lack usefulness at all. It's the same information either way.

[anarazel]

Using a different type of file that's then read with fs.read() doesn't really work, as there often are generated sources or such for a specific binary.

I can't imagine why it "wouldn't work". If there are generated sources for a specific binary, those could be generated alongside the target handling, and offloading the filelist to a dedicated file will still serve the purpose of greatly reducing the complexity, just not totally reducing it. A strict win over the current state of affairs, I would say.

Sure, that's possible - but you end up with a lot of buildsystem code in a single file...

It also leads to not having useful error messages and other such fun.

I'm baffled at why error messages would inherently become not useful as a result of this. In fact, I'm baffled at why error messages would lack usefulness at all. It's the same information either way.

If I have a syntax error in meson.build, I am helpfully pointed at the specific line. If there's an error in a file read by fs.read(), that's not typically going to be the case.

You're likely going to have to build a bit of parsing logic in meson (e.g. filtering out comment lines). Not impossible, but also not exactly optimal.

[anarazel]

Sure. It's probably going to rarely be useful, but at some point restricting things makes them more complicated without a gain in clarity.

Let's assume for a moment here that we're talking about a build system that does restrict things, and we're discussing possibilities for compromising and loosening the restrictions a bit.

I wasn't advocating dropping all restrictions in my comment - I was purely commenting on the parent vs sibling vs nephew aspect.

But figuring out a way to restrict things that meets the original design goal of the person who enforced a restriction, while still unlocking at least some use cases, feels useful and is, at a minimum, better than before. That's what I, personally, am interested in thinking about.

I agree.

[eli-schwartz]

If I have a syntax error in meson.build, I am helpfully pointed at the specific line. If there's an error in a file read by fs.read(), that's not typically going to be the case.

For lexer errors, yes. Anything else, e.g. invalid kwargs or disallowed values, we just point at the line containing the primary operation (usually the function call itself).

You're likely going to have to build a bit of parsing logic in meson (e.g. filtering out comment lines). Not impossible, but also not exactly optimal.

... and there we have it, I viewed the fs.read() thing as a way to build better lists of strings, not a way to offload a DSL and conditional logic to a new file. :D

[jpakkane]

Indeed, you can use output: 'htmldir' and then just have the documentation builder build into that -- the problem is that ninja can recognize a directory as output, but not recognize much about it other than that it was created at least once. I'm not sure how an "opaque directory" approach solves that problem at all though -- ninja still can't tell much of anything about it.

It can treat it like a file output, because if you write something in the subdir itself, the mtime of the directory changes to that value (I think on all platforms but not really sure). But it requires that whatever command generates the output writes something in the top level dir on incremental builds. Otherwise Ninja will run the target again next time.

[zasdfgbnm]

Upvoting this. Was trying to move away from cmake to meson, but eventually give up due to this.

[jpakkane]

What is the exact thing that you needed to do but couldn't?

[zasdfgbnm]

What is the exact thing that you needed to do but couldn't?

I want a soft switch from cmake to meson. For our cmake build, we are using a Python script to convert things like runtime/a.cu into resources/a.h. I want meson to do the same so that #include <resources/a.h> works. I do want the resources/ prefix, because this is better code organization than #include <a.h>.

[jpakkane]

For this specific case you can create a resources subdirectory and have the custom target to do that in that dir's meson.build file. You can either first recurse to the runtime dir and do a files there or just directly specify the source files as '../runtime/a.cu`.

[zasdfgbnm]

For this specific case you can create a resources subdirectory and have the custom target to do that in that dir's meson.build file. You can either first recurse to the runtime dir and do a files there or just directly specify the source files as '../runtime/a.cu`.

This workaround is not perfect, but sounds acceptable. Thanks for the suggestion!

[gabeblack]

There's been several occasions where I've had to wrap my custom_target with a script that copies the generated output in to the custom_target's working directory so that it meet's the custom_target's requirement of the output field not containing a path or subdirectory.

I've been porting the building of source rpms to building with meson. As you might know, source rpms have a structure of rpmbuild_dir/{BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS}. The final installation of building the source goes in the BUILDROOT folder often with a structure matching what one would see in their filesystem (libs go in BUILDROOT/<pkg>/usr/lib64 and headers in BUILDROOT/<pkg>/usr/include).

However with meson's custom_target, I'm forced to copy/flatten the generated directory structure's include files and libraries in to the directory where my meson.build's custom_target lives because of the 'no subdirectory' requirement.

I guess this just reeks of a code smell -- but I'm getting convinced that it isn't in our meson.build files, but rather meson's design in this area. I like the restrictions that meson does in other areas as I think it helps you re-think your design and structure to make sure you do things correctly and cleanly. However, in respect of having to port third-party code generators and now rpmbuild... I think this restriction is actually a code-smell in meson instead. If we could just simply specify our outputs in the outputs attribute of custom_target, we wouldn't have to add wrapper scripts to code generators (to copy the outputs in the flat directory structure). We wouldn't have to have copies laying of libraries, sources, and header files at all! The code smell is having to do the copies...

I like how meson tries to prevent you from having to 'hack' the build system to get it to work, but I think having to 'hack' rpmbuild and other tools to flatten out all their outputs to the calling custom_target directory is just as bad.

I admit I may not be aware of a meson feature that allows me to do what I'm doing, so I'd love to hear suggestions of meson features that would let me work around having to copy around the output of building source rpms to be able to have those headers and libraries used as part of other build targets.

As you can see from other comments in this thread, the restriction doesn't just apply to rpmbuild, but there are many other code generators and other external build tools that create outputs in a nice, clean subdirectory structure ready for user consumption. Meson should not be requiring that clean structure to be flattened or that directory structure to be pre-existing (so you can put meson.build files in them) in order to be consumed.

[eli-schwartz]

As you can see from other comments in this thread, the restriction doesn't just apply to rpmbuild, but there are many other code generators and other external build tools that create outputs in a nice, clean subdirectory structure ready for user consumption.

My interest here is for code generators and external build tools that require structures as part of their design intrinsics, such as generated headers which must match the #include <...> structure (solvable with a specific variant of generator() that doesn't apply to non-C-headers). And more generally things like rust or python require specific output structures in order to be usable, even at compile time.

Producing a declare_dependency() which is interchangeable with the installed version is also important, mainly for external wraps where you cannot just arbitrarily re-order public headers into the common include/* & src/ pattern because you don't own that structure.

However I'm not certain I understand the case for rpmbuild. For rpmbuild, you should be invoking meson install --destdir=BUILDROOT/<pkg>/ for example using these rpmbuild macros: https://github.com/mesonbuild/meson/blob/5e76e2a7ff9e245502f94f7c426dbee28243c864/data/macros.meson#L1-L45

[gabeblack]

However I'm not certain I understand the case for rpmbuild. For rpmbuild, you should be invoking meson install --destdir=BUILDROOT// for example using these rpmbuild macros:

I was unclear, I apologize. I was giving an example of using rpmbuild to build the source code of certain packages, not to use it to generate an rpm. By defining _topdir you can use it to compile a source rpm and stage and install it in a temporary directory. You can then point other projects to those generated includes/libraries -- all within a folder that is temporary (i.e. without installing the rpm in the host environment (often requiring sudo privileges)).

It was probably a bad example, but I'm finding more and more third-party tools that stage built libraries and headers in a directory structure. AFAIK, meson doesn't allow you to consume it without copying the files to either pre-existing folders, or to copy the files to the custom_target folder.

Am I mistaken? Do you know if there any plans to allow output to allow subdirectories?