Allow configure_file, custom_target and generator output file to subdirectories under builddir

[jasonszang]

I've encountered two cases that would require configurating / generating file into subdirectories.

1. Project configuration header that would be included from other headers in the same project

Say I have a library project named foo and the project layout is like this:

include/
    foo/
        bar/
            foobar/
                foobar.h # includes config.h
            bar.h        # includes config.h
        foo.h            # includes config.h
        config.h.in # configured by meson
src/
    foo/
        foo.cpp
...

We would want our client code use #include <foo/config.h> to include our config.h since we don't install config.h directly to includedir. But for our own code this would be impossible since generated config.h is directly under <builddir>, not <builddir/foo>.

Of course, for foo.cpp the work-around is easy: just #include <config.h>. But it becomes complicated we need to include config.h from bar.h and foobar.h.

In fact, there is no way to correctly include config.h that both work in project directory and after installation. If we want bar.h and foobar.h to be correct after installation we must #include <foo/config.h> or use relative path in them. Including <foo/config.h> won't compile, and although relative path happens to work for bar.h, it won't for foobar.h. Neither will it work if we reside in a subdir. Using #include <config.h> allows us to compile, but bar.h and foobar.h breaks up after installation.

The correctly way to configure and include config.h should be just generate it under builddir/foo and #include <foo/config.h> everywhere, just as if it were in the position of config.h.in. Currently meson does not allow this, and forces a flat include directory, or hacking into builddir.

2. Protobuf generation

This one is simpler. If we have some protos like

protos/
    foo/
        foo.proto
    bar.proto

in which bar.proto imports foo.proto. it would be nice if we could use one generator for both protos and and get under build directory something like this: For the compiled protos to work the directory structure of generated sources must mirror the protos, because bar.pb.h will #include "foo/foo.pb.h". The generated sources must look like this:

protos/
    foo/
        foo.pb.h
        foo.pb.cc
    bar.pb.h
    bar.pb.cc

Althouth this would also require something like @DIRNAME@ substitution if we are to pass generated sources to build targets properly, but without supporting generating source trees we cannot use protos with subdirectories. This also prevents wrapping of existing projects that use protos this way, which is quite a common practice.

PS: I suggest putting every generated file from configure_file() and custom_target() (including ones in subdirs) into one directory tree named something like <builddir>/sources@gen, add it to include directory, and allow outputing to subdirectories under it. This should avoid potential name clashes and give developers control over generated folder layout.

[rom1v]

I think I have a similar case: https://stackoverflow.com/questions/46729488/how-to-compile-aidl-files-using-meson

I need these source files:

• src/com/rom1v/example/Service.aidl
• src/another/package/Other.aidl

to be compiled under:

• builddir/gen/com/rom1v/example/Service.java
• builddir/gen/another/package/Other.java

From the java compiler point of view, this is an error for a java file not be in a directory matching the package name.

[liugang]

2548 is possible way to fix this type of issue.

if one project call configure_file() with install_dir, install_headers(), and so on. copy these header files to a include_intermediates_dir, and all compile target include this intermediate directory.

[jasonszang]

@nirbheek Would you please be kind enough to explain to me the considerations behind the design choice of Output must not contain a path segment. in source generation targets like custom_target and why are generating code into subdirectories of build dir considered harmful? I am trying to figure out a way to properly compile protobuf proto files with custom_target (by properly I mean get meson to figure out the dependency relations correctly), which I would like to use in a small production project which for some reason does not fit in our in-house build system. I acturally found a way that work with proto files that has no subdirectories (all protos must live in a single folder), but proto subtrees like the one I wrote at the top can not be built by meson. Then I commented out these two lines in build.py

    if '/' in i:
        raise InvalidArguments('Output must not contain a path segment.')

And it worked surprisingly well. But I'm afraid by doing this I may be breaking something else inside meson. Could you please give me some explanation about this limitation and let me know your thought about its removal (or replacement with something like must not contain .. component if escaping is the problem)? With this limitation proto trees with subdirectories seems never could be properly built by meson.

[rib]

Just to mention that I recently (https://github.com/mesonbuild/meson/issues/3023) also found it a little awkward that configure_file()'s output filename couldn't include a subdirectory while I wanted to generate a .java file based on some build time config options. The javac compiler and Meson's jar() require source code to be laid out according to the namespace of the Java package.

In my case it's possible to work around the limitation by creating a meson.build under src/java/com/impossible/glimpse/meson.build where I use configure_file() which results in a file like <builddir>/src/java/com/impossible/glimpse/<output file>

I wonder if the subdirectory were limited to matching the subdirectory of the input filename that might appease some of the general concern around having random control over the build directory layout. This would still allow meson control to change the top level structure of the build directory but it would have to preserve the subdirectories requested for an output file.

E.g. if I want to generate a GlimpseConfig.java from GlimpseConfig.java.in in the package com.impossible.glimpse with a file under src/java/com/impossible/glimpse/GlimpseConfig.java.in then ideally I could do:

configure_file(input: 'com/impossible/glimpse/GlimpseConfig.java.in',
        output: 'com/impossible/glimpse/GlimpseConfig.java',
        configuration: conf_data)

And in the build directory meson can create: <build>/any/meson/private/top/level/com/impossible/glimpse/GlimpseConfig.java

The top level layout doesn't really matter (at least in my case) so long as the java compiler + ninja backends knows what it is.

So it might not imply spaghetti access to arbitrary build directories, just some control to build a sub-heirachy within the build directory.

[NickeZ]

Also ran into this problem with custom_target and doxygen. The call to doxygen will produce the html files needed and the latex sources needed. Doxygen also produces a Makefile that helps you run pdflatex the right amount of times and in the right order.

I would like to write something like this:

cdata = configuration_data()
cdata.set('TOP_SRCDIR', meson.source_root())
cdata.set('TOP_BUILDDIR', meson.build_root())
cdata.set('VERSION', meson.project_version())

doxyfile = configure_file(
    input: 'Doxyfile.in',
    output: 'Doxyfile',
    configuration: cdata,
    install: false
)

html_target = custom_target(
    'htmldocs',
    build_by_default: false,
    input: doxyfile,
    output: ['html/index.html', 'latex/Makefile'],
    command: [doxygen, doxyfile],
)

pdf_target = custom_target(
    'pdfdocs',
    build_by_default: false,
    input: 'latex/Makefile',
    output: 'latex/refman.pdf',
    command: [make, '-C', '@OUTDIR@/latex'],
)

docs/meson.build:15:0: ERROR: Output must not contain a path segment.

[nussbrot]

I also ran into this problem and currently don't know how to work around it. I'm evaluating meson and am trying to figure out if i can use it to build a FPGA project. One of the problems is IP-generation.

This usually works like this: You generate something like a IP-variation file, which is some kind of XML format, good for versioning and then you run a IP-generator on this file and get a bunch of output files.

Example output files for generating the IP cms_sys_pll:

cms_sys_pll.cmp
cms_sys_pll.csv
cms_sys_pll.html
cms_sys_pll.spd
cms_sys_pll.xml
cms_sys_pll_generation.rpt
simulation\aldec\rivierapro_setup.tcl
simulation\cadence\cds.lib
simulation\cadence\cds_libs\pll_0.cds.lib
simulation\cadence\hdl.var
simulation\cadence\ncsim_setup.sh
simulation\cms_sys_pll.sip
simulation\cms_sys_pll.v
simulation\mentor\msim_setup.tcl
simulation\submodules\cms_sys_pll_pll_0.vo
simulation\synopsys\vcs\vcs_setup.sh
simulation\synopsys\vcsmx\synopsys_sim.setup
simulation\synopsys\vcsmx\vcsmx_setup.sh
synthesis\cms_sys_pll.debuginfo
synthesis\cms_sys_pll.qip
synthesis\cms_sys_pll.vhd
synthesis\submodules\cms_sys_pll_pll_0.qip
synthesis\submodules\cms_sys_pll_pll_0.v

So i tried to put this into a cutom target like so:

qsys = find_program('qsys-generate')
python = import('python').find_installation('python')

files = run_command(
  python, 'cat.py', files('ip/cms_sys_pll.outputs'),
).stdout().strip().split('\n')

custom_target('cms_sys_pll',
  build_by_default: true,
  output: files,
  input: 'ip/cms_sys_pll/cms_sys_pll.qsys',
  command: [
      qsys,
      '--output-directory=@OUTDIR@/cms_sys_pll',
      '--synthesis=VHDL',
      '--simulation=VERILOG',
      '@INPUT@']
)

And of course i ran into the error:

Output 'simulation\\aldec\\rivierapro_setup.tcl' must not contain a path segment.

Do you have any suggestions to work around this issue?

[jasonszang]

Well personally I decided to give up meson and went back to the good old CMake because of this one. And I find modern style CMake comfortable enough I stopped looking for another build system generator. And unfortunately I am not aware of any universal work-around beside removing this limitation from meson.

[nussbrot]

Hi @jasonszang, thanks for your answer. I see this issue was first posted by you. If you don't mind i would like to ask you if you have experience with CMake and let's say "custom toolchains"? But it is really off-topic and this is probably not the right place to ask you that.

[jpakkane]

The question here is does the file really need to be in a cm_sys_pll subdirectory (some tools are nasty in this way). If no, then write it to @OUTDIR@. If yes, then you need to have the custom target declaration in the cm_sys_pll subdirectory (and also have just --outdir=@OUTDIR@. In Meson all outputs go to the build directory that corresponds to the source directory they are declared in. This simplifies debugging build problems, since you always know that files in build directory X must come from the corresponding source dir X.

[nussbrot]

Yeah, you're right, that is a mistake in my build file. I can give the tool any directory, so --outdir=@OUTDIR@ works fine. But i can not control the subfolders like synthesis\... or simulation\... that are created in @OUTDIR@. So i can not add the resulting files as outputs since stuff like output: 'synthesis\cms_sys_pll.qip' does not work.

[ptsneves]

@jasonszang I have the same frustration. One way to work around it is that you create a sub meson.buil in the subdir and do the remaining thing there. Then subdir the newly created meson.build. Not optimal but i think it works

[valpackett]

gir-to-d also generates subdirectories, like

somelib/c/types.d
somelib/c/functions.d
somelib/Obj1.d
somelib/Thingy.d

Why is this ridiculous limitation still present? >_<

UPD: seems like there are workarounds in gir-to-d for this

[rulatir]

One way to work around it is that you create a sub meson.buil in the subdir and do the remaining thing there. Then subdir the newly created meson.build. Not optimal but i think it works

@ptsneves, how does it work for you? It most certainly doesn't work for me. Assuming my project is in foo, then I create foo/images/openresty/meson.build and attach the directory with subdir('images/openresty') in the main meson.build file. Now when I write the command to build the openresty image, there is not a single @ VARIABLE@ that contains the exact string 'images/openresty' that I could use to construct the output file path that the build tool must receive. The build tool is a wrapper that cds into the source directory and executes the remaining arguments as a command, and that is necessary because of course Meson is so opinionated as to disallow setting the working directory for the command, so I must hack around that. Then again the command that the cd-wrapper receives is supposed to be a pipeline (an alternative to which is writing ANOTHER wrapper script, which I want to avoid at all cost because at this point the build configuration and script wrappers combined would become bigger than the Dockerfile they are supposed to build from). It must be a pipeline because it must do two things: build the image and then pull the image .Id from docker inspect and save it in the surrogate output file (because of course Meson has no idea about outputs that don't materialize as files).

Yes, this is becoming a rant. It would seem like Meson is just limitations, quirks and opinionatedness without offering "strength" as a reward for accommodating those. Doing hashes instead of timestamps and correct graph-theoretic minimal incremental builds are about the only two "cool" things that Meson offers, and both are becoming standard in the build system world.

[ptsneves]

@rulatir What about you maintaining the variable for the subdir yourself. You do _subdir(var_with_path) and then use the var_with_path__ as you see fit. Does this not work for you?

[rulatir]

@rulatir What about you maintaining the variable for the subdir yourself. You do _subdir(var_with_path)__ and then use the var_with_path as you see fit. Does this not work for you?

"Why don't you do it yourself" is the ultimate non-answer to every feature request. I mean why even use software? Just compute in your head.

[ptsneves]

@rulatir I understand your point as a user of the software in general. As a developer and user of open source software I have to say that we are in a "tough luck" situation. The thing is that in open source software there are 2 ways to move forward with a feature: To convince somebody or to submit a patch/pull request yourself. In a closed source software you basically only have to convince, but the convincing can be pretty hard to do if there is no business case.

It seems the convincing another person to fix/do this feature failed, but the pull request approach has not failed from what i see in the thread. Even if it is not accepted you can patch your meson to have this feature if you want. It is not ideal but it is yet another workaround. I have been in this last situation a lot of times, but it is damn better well than just not being able to do anything about it.

[mensinda]

@jpakkane are there any design decisions for this restriction or is this only an implementational detail? If not, I would see no reason to not allow this behavior (with tests of course).

[rulatir]

@jpakkane

This simplifies debugging build problems

When you are forced to refactor your build process inside out to accommodate a build system with artificial restrictions, you will inevitably make mistakes, and thus introduce problems.

[dankegel]

generator() has a preserve_path_from option which may help.

custom_target() doesn't, which is the subject of https://github.com/mesonbuild/meson/issues/6418

[xclaesse]

Note that @nirbheek already made a patch that does this, in an abandoned PR: https://github.com/mesonbuild/meson/pull/2617/commits/7c15837bd25d37672bfddcb6ef9402792b3d3c23.

[stapelberg]

I’m running into the same problem, and the following workaround (creating a stamp file like with autotools) worked for me:

anyevent_i3 = custom_target(
    'anyevent-i3',
    # Should be AnyEvent-I3/blib/lib/AnyEvent/I3.pm,
    # but see https://github.com/mesonbuild/meson/issues/2320
    output: 'AnyEvent-I3.stamp',
    command: [
        'sh',
        '-c',
        'cp -r @0@/AnyEvent-I3 . && cd AnyEvent-I3 && perl Makefile.PL && make && touch ../AnyEvent-I3.stamp'.format(meson.current_source_dir()),
    ],
)

test(
    'complete-run',
    complete_run,
    depends: [
        anyevent_i3,
    ],
)

[gtarsia]

This is the second "our way or the highway" thing I see in meson, really not enjoying it. This limitation is annoying and from reading this thread I don't see a real reason for it.

[IcedQuinn]

Ran in to this again trying to package libinfinity. It uses absolute paths for all the files and meson won't let me put the config.h in the right spot. Joy.

[Jehan]

For the record, we have this issue in GIMP too when trying to generate HTML documentation for GObject Introspected API (with g-ir-doc-tool and yelp-build). I guess I'm going to do a workaround similar to the temporary stamp file above (anyway the input/output was bogus/incomplete already as custom_target() forces us to give full input/output list, which is obviously non-practical with html docs of nearly 2000 pages). 😕

[vinipsmaker]

I'm facing the same issue. It's a different scenario related to plug-ins accessing headers for my configured software, but the solution is the same.

#include <config.h>

It doesn't work and never will. It's not robust and it injects abstractions in shared namespaces that are very prone to collide (how many projects use the same #include <config.h> that refer to different <config.h> files I ask you).

If it works now, it's only in a very limited use case that happens to be the most common one, but that's in no way robust or justifiable.

[eli-schwartz]

Say I have a library project named foo and the project layout is like this:

include/
    foo/
        bar/
            foobar/
                foobar.h # includes config.h
            bar.h        # includes config.h
        foo.h            # includes config.h
        config.h.in # configured by meson

Given you have a directory include/foo/ which contains foo.h, you could have it contain meson.build which generates the configure_file too, and meson's builddir will generate <build_dir>/include/foo/config.h.

You then add include_directories('..') and, include <foo/config.h> magically works everywhere. You don't even need to change directory structure in your project.

Depending on other project layouts that aren't the badly-chosen example in the original report, you can still do the exact same thing BUT it would require tweaking your project layout to include the same tree layout in your source directory too. On build systems supporting in-tree builds, running make/ninja would have created that anyway...

One could argue that it's not nice to be forced to create a source directory layout for this (I don't personally find this bothersome, but YMMV), but it cannot be argued that "it's impossible".

The other example in the original report, protobuf generated headers, was implemented shortly after this report was submitted, via #2764

Generator output files now support subdirectories under builddir, to match their subdirectories under sourcedir. And the testsuite includes protobuf sources testing this functionality.

[tristan957]

I am writing Java bindings for my project. I have a couple of Java dependencies, so I am using Maven to compile/package the Java portion of the bindings. Maven creates a lot of output files in various output directories. Maven has two commands: compile and package. Compile creates .class files out of your .java files, and package is a combination of compile and building the physical jar out of those class files. I want to be a good Meson developer, and specify the outputs accordingly, but because Maven places outputs in directories, I cannot correctly specify them in the output kwarg of custom_target() because I would need to specify path segments.

[tristan957]

It is not feasible to create a module for every single tool in existence that doesn't conform to Meson's world view. There aren't that many people who are going to care enough to do that. It is so much easier for me to have a custom target where I just do what I want. I am still left wondering what would break by enabling this feature.

[anarazel]

It is not feasible to create a module for every single tool in existence that doesn't conform to Meson's world view.

Indeed. It'd be one thing if one could provide modules from within a project, but since that's also frowned upon...

There aren't that many people who are going to care enough to do that. It is so much easier for me to have a custom target where I just do what I want. I am still left wondering what would break by enabling this feature.

It's not just that it's a lot of work to add it to meson, it's also that that stretches timelines into something that often not feasible. Needing to wait for a meson release, being OK bumping the minimum meson version etc.

[Flow-It]

It seems this issue has evolved into a discussion about possible workarounds. I'd love to discuss how we could make it work.

@jpakkane Can we write down a (hopefully complete) list of problems which could occur when allowing this?

Looking through this thread and the duplicates, I only see two arguments against it:

• Having a mirrored location for each configured file in both source and build directory simplifies debuging.

Agreed, but it is a conscious choice of the developer to use another location. We could even add a comment at the top of the output file This file was created by meson configure_file, the input was xy. Then there is still the option that an output file was generated from a dict and there never was a source file in a mirrored location in the first place. I would argue that the benefits of the proposed change outweigh this mentioned risk.

• Allowing the user to "arbitrarily" place files in the build directory might break the backend.

Yes, this is a serious one. But we already do this. a) Not the user, but meson itself (so if a break should ever happen, a fix is only necessary in meson, not in every build definition) places its own files in the build directory (for example the meson-private directory). b) That is exactly what configure_file does. Friction with the backend can also happen in the root build directory. See this, completely valid, example:

project('example', 'c')
configure_file(output: 'build.ninja', configuration_data: {'key':'value'})

Does someone make a guess what happens here? Answer: No warnings at all, but ninja overwrites the generated file and it will never be available to compilation.

• We don't need to do this in meson. It's probably a bad project layout anyway, or the generators need to be patched.

I disagree. There are too many source-generating tools out there to patch them all. They should "just work". As for the bad project layouts: Probably it's true in many cases, but definitly not in all. And there will always be ways for the user to do stupid things. Every feature in meson can be abused in some way. All in all, I'd argue that the benefits of the proposed change outweigh the alternatives.

---

So, how could we implement this?

Let's establish a meson-generated-sources (naming is up for discussion) directory in the build directory. Should any backend ever expect something different in this directory, then we have much different issues because directories like meson-private can't be assumed to not interfere with the backend anymore, too.

All paths under this new meson-generated-sources directory are fair game. It is still very much recommended to mirror the source directory. But not under build/ but under build/meson-generated-sources/.

It would now be allowed to use a path for the output of configure_file. This path must be relative, and will be interpreted relative to meson-generated-sources. For obvious reasons, .. will be rejected.

---

Any opinions on this idea? Is this technically feasible?

[tristan957]

Not a fan of meson-generated-sources, but I have supported this issue since day one so I am not someone who needs convincing :).

[rgommers]

The question here is does the file really need to be in a cm_sys_pll subdirectory (some tools are nasty in this way). If no, then write it to @OUTDIR@. If yes, then you need to have the custom target declaration in the cm_sys_pll subdirectory (and also have just --outdir=@OUTDIR@. In Meson all outputs go to the build directory that corresponds to the source directory they are declared in. This simplifies debugging build problems, since you always know that files in build directory X must come from the corresponding source dir X.

I'm running into this problem as well with Cython. To answer the direct question first: yes, my generated sources must be in the subdirectory, and no it cannot be done from subdir/meson.build.

To explain, here is the structure (only the relevant parts of the project):

scipy/
  __init__.py
  meson.build
  special.pxd
  special/
    __init__.py
    meson.build
    _generate_pyx.py   # generates a bunch of files, most importantly `cython_special.pxd`
  stats/
    __init__.py
    meson.build
    _stats.pyx    # contains `cimport scipy.special.cython_special`

Then:

• the build definition for _stats.pyx must contain a dependency on the generated cython_special.pxd.
• to be able to do that in scip/stats/meson.build, that codegen must happen in scipy/meson.build and not in scipy/special/meson.build
• because Cython is build-tree-layout sensitive, that file must be at builddir/scipy/special/cython_special.pxd.
• the __init__.py files and special.pxd must also be present in the build dir.

I've tried various hacks, but there really is no good way to do this, except by doing all the codegen in-tree before invoking Meson. I'd like to do everything with Meson, so there's a single cache and I don't have to write a wrapper around the meson CLI. But it does not seem possible at the moment.

We don't need to do this in meson. It's probably a bad project layout anyway, or the generators need to be patched.

I disagree. There are too many source-generating tools out there to patch them all. They should "just work". As for the bad project layouts: Probably it's true in many cases, but definitly not in all.

In my case, the project layout is not bad. You can argue the problem is the whole design of Cython, and yes it's annoying that its cimport semantics are so path-dependent - but that's not going to change, and there's a ton of Cython code out there.

Note that @nirbheek already made a patch that does this, in an abandoned PR: 7c15837.

That seems to do about the right thing. And the downsides of allowing to place files in subdirectories while still forbidding .. seem very minor compared to the upsides here.

[QuLogic]

to be able to do that in scip/stats/meson.build, that codegen must happen in scipy/meson.build and not in scipy/special/meson.build

That's not true; variable are global and defined in order, so the special directory 'just' needs to be done first. Whether it's possible to do that in order is a different question. Also, I'm assuming we have a correctly working Cython wrapper that knows about its outputs.

[rgommers]

That's not true; variable are global and defined in order, so the special directory 'just' needs to be done first. Whether it's possible to do that in order is a different question.

Interesting. I seemed to not be able to make that work, but the problem was a different one most likely then.

[pfirsich]

I am currently migrating a rather complex build of a large project (proprietary) to meson and I just stumbled across this problem and I see no viable workaround. I would respect if there were good reasons that corresponded to concrete problems, but the reasoning seems very vague and I feel this issue is simply being ignored. That's a real shame.

[fluidparadigms]

I've also run into this issue while porting Apache Xerces to meson. The issue in this case is the statement #include <xercesc/util/Xerces_autoconf_config.hpp> in the internal Xerces source code. The config file is generated from src/xercesc/util/Xerces_autoconf_config.hpp.in or src/xercesc/util/Xerces_autoconf_config.hpp.cmake.in

configure_file() takes the install_dir: keyword argument already - would it be such a stretch to allow a destination directory that is relative to the build dir?

[LAK132]

Not sure if this is related, but I'm currently running into an issue where a subdir is trying to copy a .dll into the root of the build directory (next to the .exe) using configure_file but it's ending up in a private directory for that subdir. really annoying.

[rulatir]

Not sure if this is related, but I'm currently running into an issue where a subdir is trying to copy a .dll into the root of the build directory (next to the .exe) using configure_file but it's ending up in a private directory for that subdir. really annoying.

Build system sandboxing is a failed concept.

[mscofield0]

What exactly is preventing a fix for this? I don't really understand...

I'd just like to do:

configure_file(
  input: 'include/Version.hxx.in',
  output: 'include/Foo/Version.hxx',
  configuration: cdata,
)

[tristan957]

Ideology

[cdelnooz]

Has anything happened on this issue at all? I thought @Flow-It made a workable suggestion to have a "generated sources" tree that I would have thought resolves the main objection of @jpakkane to users messing with the build-tree. I think it would solve many of the valid use-cases mentioned in this thread, plus a few that I have myself :)

What is needed to get this improvement implemented?

[tristan957]

No one is going to work on this on this until it will be accepted as an idea.

[cdelnooz]

Ok, what needs to happen to get this accepted as an idea then? Does it need a further technical/design proposal? Further discussion somewhere?

[3v1n0]

Not sure why I didn't try earlier, but some of these cases can be handled using install_data together with configure_file.

See https://github.com/mesonbuild/meson/issues/7675#issuecomment-1065160221

[mkow]

@jpakkane: Would it be possible to reconsider your stance on this? As I understand, this intentional limitation is supposed to help developers, but instead it's making life much harder for many (as you can see in this thread).

I just spent a few hours fighting with it when trying to build SGX enclaves (they're basically dynamic libraries) with Intel SGX SDK framework.

What I'm trying to do

I have a project where I need to generate a lot of test enclaves (basically, each test group is a separate enclave). Building with Intel SGX SDK framework requires a lot of boilerplate/configuration, so having separate meson.build in each test would be a huge code duplication - ~200 Meson LoC duplicated per test. So, what I'm trying to do is to iterate over the tests with foreach in test/meson.build and have the enclave building script inside the loop. And here's the problem: one step of building enclaves with Intel SGX SDK is running a codegen tool (sgx_edger8r) which takes a single input file (<some_name>.edl) and outputs enclave_t.h, enclave_t.c, enclave_u.h and enclave_t.h (these names are hardcoded, but I can pick the output directory). I'd like to output these files to the test directory I'm currently building in that loop, but it's impossible due to this very issue. Generating it to the test/ directory causes clashes between the filenames from different tests.

Failed approaches

• Having meson.build in each test dir - results in a huge code duplication
• generator() - seems impossible due to another limitation: ERROR: Every element of "output" must contain @BASENAME@ or @PLAINNAME@.
• configure_file() - even when using command (custom generator) it seem to allow generating only a single file
• Trying to change Intel SGX SDK to allow picking output names in the codegen - this won't really help, because I'll end up with tens of headers from different subprojects mixed in the same directory, making all of them visible in include path to all tests.

It seems that the only not-super-terrible solution to achieve this is to allow output to contain subdirectories. Or am I missing something?

Sorry for the tone, but I'm really frustrated at this point. It's not the first time I'm fighting with Meson for hours and I'm starting to regret that I chose Meson for this project :/

[cdelnooz]

but instead it's making life much harder for many (as you can see in this thread).

@mkow: I feel your pain and this arbitrary limitation is certainly making life harder afaic. I have a very similar experience to yours. In my case, I'm trying to build a tree of pdf-documents out of markdown sources that can have a variety of diagrams/images that are each generated from source as well. As the rest of the code uses meson, I tried to do this with meson as well. I've got the exact same list of failed approaches. I got so tired of fighting meson and seeing that this limitation is there on purpose, that I've gone back to good old make to build my documentation. So much easier.

@jpakkane: whilst I see your point about ease of debugging, this thread clearly shows many use cases that would be trivially solved by making it possible for developers to disable this limitation. Maybe a compromise could be a global setting that allows developers to choose the behaviour they need? As a developer, I think we are quite capable of making the trade-off between functionality and ease of debugging for ourselves.

[fluidparadigms]

Could we at least have the option to have the output directory tree mirror the input directory tree?

For example, if you were building 2 executables: "thing_v1" and "thing_v2" with these input files: src/thing_v1/somepath/logger.cpp.in src/thing_v2/somepath/logger.cpp.in ...could they output like this? build/thing_v1/somepath/logger.cpp build/thing_v2/somepath/logger.cpp

Sure, this may be poor form, but what if it's closed-source, third-party code that you can't change? In that case, this is a brick wall in terms of adopting meson in a legacy codebase.

[rulatir]

Ever since I gave up on meson (it's been 2 years or something), I've been sending a link to this issue to anyone who confides in me that they're considering meson for their project.

[jpakkane]

which takes a single input file (.edl) and outputs enclave_t.h, enclave_t.c, enclave_u.h and enclave_t.h (these names are hardcoded, but I can pick the output directory)

The sad and unfortunate fact is that this is just plain poor design. If you have a system that forces people to generate multiplie headers with the same name within a single project then it is broken because Murphy's law says that they will get mixed up. It is unavoidable. This does not really help you, but I'd figured I'd write it out explicitly so that it is absolutely clear for everybody.

That being said, could you explain why it takes 200 lines of build definition per target? It seems a bit excessive.

Another question is are you using those files in a single executable/build target or does each of them go to a separate executable?

[mkow]

If you have a system that forces people to generate multiplie headers with the same name within a single project

I'm not sure what's your exact definition of "project", but here I'm generating a lot of test binaries - each is a separate application built with that framework. Within a single app these headers have unique names.

The sad and unfortunate fact is that this is just plain poor design. If you have a system that forces people to generate multiplie headers with the same name within a single project then it is broken because Murphy's law says that they will get mixed up. It is unavoidable. This does not really help you, but I'd figured I'd write it out explicitly so that it is absolutely clear for everybody.

1) Whether it is a bad design or not, it's out of my control. And as you see from this thread, this is a common pattern in code generators in many frameworks. 2) Even if the codegen did otherwise (e.g. allowed to prefix the generated file names with something), then it doesn't solve the problem. See the last point of my previous message under "Failed approaches" - the file names are not the problem here, I actually need them in test-specific subdirectories. Also, even if it did solve the problem, that's how the software works right now and I have to somehow live with it. I can upstream a patch, and then after a few months or even a year they'll have it in their release. What should I do in that time? Use another build system? 3) What are you suggesting me to do then? You're complaining about the framework, but that doesn't hint to anything I could practically do to use Meson in my project.

That being said, could you explain why it takes 200 lines of build definition per target? It seems a bit excessive.

There's a bunch of configuration and tools I need to invoke in that process. Each test is actually building both the enclave (.so) and the test app using it (ELF exec), and both of them depend on the generated code. I could refactor out some of the shared Meson code to some higher-level meson.build, but that still leaves me with ~70 lines per test. Even if I golfed this to fewer lines, then it's still multiplied by the number of tests, so, I'll end up with a huge number of copy-pasted lines.

Another question is are you using those files in a single executable/build target or does each of them go to a separate executable?

Each test needs a separate generation of that code (i.e. the auto-gen'd code is test-specific and each test requires this auto-gen'd code).