Design proposal: install files created by generator.process()

[sarum9in]

Problem

Files can be generated by multiple functions, in core meson we have custom_target() and generator.process(). All other functions depend on one of those. custom_target() supports installation of generated files (even though it is poorly documented, #2451), but `generator.process() does not.

Use-case

Protocol Buffers generated headers are consumed by library dependencies. Installing these headers directly without helper install scripts is convenient.

Caveat

While documentation recommends to use custom_target() for anything that is intended to be installed, custom_target() does not support preserve_path_from functionality: the ability to generate files with arbitrary names. This is crucial for code generators like protoc. So the alternative for this proposal would have been to implement this functionality in custom_target() and keep generator's output non-installable. Note that it is not good idea to allow arbitrary filenames creation by users because it is hard to support, @jpakkane can tell more about this.

Proposal

Generator itself is a template that transforms single input file to multiple output files. process() applies this transformation to multiple input files independently. To the best of my understanding given generator X the following is equivalent:

lst = X.process('hello.inp', 'world.inp')

lst2 = X.process('hello.inp')
lst2 += X.process('world.inp')

Similar to custom_target(), specify installation directory per-output file. So for a given generator X that transforms input file into 2 output files we have to specify installation per-argument. This is a simple version, we do not allow user to specify different installation directories for different inputs. It is not a restriction however since user can always just invoke X.process() multiple times if they need different configuration and concatenate returned generated lists.

X = generator(..., output : ['@BASENAME@.out1', '@BASENAME@.out2'])
X.process('file1', 'file2', 'file3',
    install : true,
    install_dir : [
        'dir1',  # file1.out1, file2.out1, file3.out1
        'dir2',  # file1.out2, file2.out2, file3.out2
    ],
)

Same as custom_target()'s install_dir we support boolean arguments here.

If preserve_path_from is set we just install target with a subpath, e.g.

X = generator(..., output : ['@BASENAME@.out'])
X.process('foo/bar/input.inp', preserve_path_from : meson.current_source_dir(),
    install : true,
    install_dir : get_option('includedir'), # /usr/include/foo/bar/input.out
)

Alternative considered

We could allow more flexible implementation to let user specify different per-input-per-output files, somewhat similar to #2416, but this adds complexity for little added value.

X = generator(..., output : ['@BASENAME@.out1', '@BASENAME@.out2'])
X.process('file1', 'file2', 'file3',
    install : true,
    install_dir : [
        ('dir1', 'dir2'), # for file1.out1, file1.out2
        ('dir3', 'dir4'), # for file2.out1, file2.out2
        ('dir5', 'dir6'), # for file3.out1, file3.out2
    ],
)

Another option is to implement preserve_path_from for custom_target() and treat generator()-created files as internal.

Review

Feedback is welcome. I am requesting approval from @jpakkane before I start working on this since it is major change and requires some non-trivial work to be done.

[sarum9in]

Alternative to this is to allow preserve_path_from for custom_target() as pointed out by @nirbheek

@jpakkane what do you think? Is there any reason not to implement preserve_path_from for custom_target()?

[jpakkane]

The main reason is the same as always: targets should not have free rein to put their stuff in random locations in the tree. They should only put them where told. Generator is different because its output goes into the private dir which is free from these limitations, targets can have anything at all in there.

Overall I like the install kwarg for process but it can be confusing due to two reasons. First: if the result is not used anywhere, the files are not generated and thus not installed. Second: if you use the output in two targets, it installs them twice. The latter we can detect and error out on, though.

[sarum9in]

To address first we can print warning if install : true is unused.

To address second we can count number of times it is used and if it more than one we error. User can always call process twice, one with install other without and use same source list.

[sarum9in]

Okay, given conversation in IRC I have alternative proposal.

custom_target() support for arbitrary output filenames

The main argument against this is that we should not allow users to create random files in build directory. However it is fine to do so for generators since they create files in private directory. So what if we allow creating files in private directory with a special flag?

proto_cpp = custom_target('protobuf-sources',
    input : ['hello.proto', 'foo/bar.proto'],
    output : ['hello.pb.h', 'hello.pb.cc', 'foo/bar.pb.h', 'foo/bar.pb.cc'],
    command : [protoc, '--proto_path=path1', '--proto_path=path2', '--cpp_out=@OUTDIR@', '@INPUT@'],
    install : true,
    install_dir : [get_option('includedir'), false, join_paths(get_option('includedir'), 'foo'), false],
    use_private_directory : true,  # this parameter allows arbitrary names
)
lib = shared_library('protolib', proto_cpp, include_directories : proto_cpp.private_dir_include())

[sarum9in]

I read a bit of source code and I understand the problem with this approach. There is a concept of output of a build target, get_outputs() method returns them. It is assumed that output is placed into target directory, something returned by get_target_dir(target). In order to implement proposal above we need to break this assumption and make it conditional, i.e. we need to introduce additional method get_outputs_in_private_dir() and based on that use get_target_private_dir() instead.

This is still doable but introduces additional complexity.

[Salamandar]

I'm really interested in this. For instance, VLC has a tree with lots of lua files in subdirectories. Those files need to be compiled and installed with the same tree. generator would've been the way to go… Now I need a meson.build in each subdir.

[sarum9in]

@jpakkane do you think we should find a way for custom_target() to support arbitrary output filenames and let them go into private directory or we should add install functionality to generators?

[jpakkane]

The latter is already in progress right?

Allowing arbitrary output dirs for top level targets is prohibited by design. It will not be supported.

[sarum9in]

@jpakkane I have not worked on any code here because we did not decide what is the best way to solve this and I am looking for your input.

I am not sure if we are talking about the same thing. I don't propose arbitrary top-level output dirs, I am proposing to allow creation of arbitrary directories inside private dir of custom target, where we allow this behavior for generators already.

some-other-top-level-target
foo@cus/
    subdir/
        message.pb.h
        message.pb.cc

But without generators. This would not require us to change semantics of generators being non-installable and also this would create these files only once.

[sarum9in]

Discussed on IRC that we are going with the first approach.

[nioncode]

Any progress on this?

[rhd]

I'd like a solution for this as well. Using the meson generator with protoc to generate python code is really painful. Any python code that needs to use the generated code needs to know the private directory in the build folder but meson does not provide a way to get it AFAICT.

The best hack/workaround I've seen is here. Is there a better way?

[sarum9in]

It's unlikely I will have time to work on this soon, sorry. Unassigning.

[GabrielGanne]

Hi,

Apologies if this is not the right place to post this. If it's not, please redirect me.

I'm trying to generate and install man pages using rst2man. I'm new to meson, but this is the is way I imagine this could go:

project('install generated files sample project')

rstfiles = files('my-command.rst')  # expect many files here
rst2man = find_program('rst2man', required : true)

man1 = generator(rst2man,
        output : '@BASENAME@.1',
        arguments : ['@INPUT@', '@OUTPUT@']
)

man_pages = man1.process(rstfiles)
install_data(man_pages)

I could not find a way to do this without a generator, and this raises an assertion telling me that man_pages are not of File type. Unless I missed something this is a use case for this proposal, right ?

Best regards,

[Salamandar]

Yeah, that's exactly what this issue is about.

EDIT : the exact syntax would be :

man_pages = man1.process(
    rstfiles,
    install_dir: `the_install_dir`,
)

[valpackett]

The proposed install_* arguments directly in process thing doesn't sound that good. Installing is not the only thing you can do with generated files. Why can't we use generated files everywhere normal files are used? :(

schema_core = gen_schema.process('metadata/core.xml')
settings_schemas = gnome.compile_schemas(build_by_default: true,
    depend_files: [schema_core, …])
install_data([schema_core, …], install_dir: 'share/glib-2.0/schemas')

meson.build:<the compile_schemas line>:0: ERROR: Unknown type 'GeneratedListHolder' in depend_files.

[Salamandar]

It looks like Meson is missing a lot of polymorphism on file types. Any kind of file should be installable, have methods like full_path(), or used as a program.

[nh2]

The best hack/workaround I've seen is here. Is there a better way?

@rhd Is that really working?

I'm having the same problem and commented a question.

[deepbluev7]

I just ran into this too when trying to generate manpages. I wanted to do:

groffc = generator(groff, output: '@BASENAME@.ps', arguments: ['-Tps', '-man', '@INPUT@'], capture: true)
ps2pdfc = generator(ps2pdf, output: '@BASENAME@.pdf', arguments: ['@INPUT@'])
custom_target('pdf', files: ps2pdfc.process(groffc.process(manpages)), install: true, install_dir: get_option('mandir'))

Instead I had to do:

pdfs = []
foreach f : manpages
  tempgroff = custom_target(input: f, output: '@BASENAME@.ps', command: [groff, '-Tps', '-man', '@INPUT@'], capture: true)
  pdfs += custom_target(input: tempgroff, output: '@BASENAME@.pdf', command: [ps2pdf, '@INPUT@'], install: true, install_dir: get_option('mandir'))
endforeach

alias_target('pdf', pdfs)

While that seems to work, it is much harder to read and generates 950 targets in my project (which is a bit of a mess in autocompletion and such).