Using -ffile-prefix-map with meson?

[dwsteele]

We (https://github.com/pgbackrest/pgbackrest) are switching over from autoconf/make to meson, but there is one issue we would like to solve before allowing release builds to be done in meson. This has to do with using the __FILE__ macro. On the old in-tree builds we'd get debug messages like this in the logs:

DEBUG:     common/lock::lockRelease: (failOnNoLock: false)

But with meson we get this:

DEBUG:     ../pgbackrest/src/common/lock::lockRelease: (failOnNoLock: false)

Using -ffile-prefix-map seems to be the solution to this problem, but we're not sure how to get meson to pass it the correct value, e.g. -ffile-prefix-map=../pgbackrest/src=.

Also, if we hard code the value:

add_project_arguments('-ffile-prefix-map=../pgbackrest/src/=', language: 'c')

It works as expected on clang but gcc produces this no matter what gets passed to it (even an empty param):

DEBUG:     ./common/lock::lockRelease: (failOnNoLock: false)

Although even the gcc output is fine for our purposes.

Has anybody gotten this to work?

[dwsteele]

I managed to generate something with the solution from #7485, but not really too satisfied with it.

python = import('python').find_installation()

file_prefix = run_command([
    python,
    '-c',
    'import sys, os; print(os.path.relpath(sys.argv[1], sys.argv[2]))',
    meson.current_source_dir(),
    meson.build_root()
]).stdout().strip()

pgbackrest = executable(
    ...
    c_args: [
        cc.get_supported_arguments('-ffile-prefix-map=@0@/=' . format(file_prefix))
    ],
    ...
)

It's pretty fragile -- where file_prefix is defined and how it is used changes the output. Seems like a built-in solution would be better since reproducible builds are an important feature.

[dcbaker]

I think we should actually just handle this transparently, it’s important for reproducible builds apparently, and needs to be different for generated and static sources

[dwsteele]

Yeah, I'm refining the code for commit right now and I was starting to think it should be the default. We (pgbackrest) need to support meson down to 0.53 so we'll still use this code, but would be great to get this improved going forward.

[eli-schwartz]

It's pretty fragile -- where file_prefix is defined and how it is used changes the output.

There's an open PR to add a relative_to() function to the Fs module, which avoids doing complicated python scripts. Though you probably want meson.project_source_root(), not current source dir.

As far as reproducible builds go, various people doing reproducible builds already set stuff like this in CFLAGS before running Meson.

Additional challenge: you do NOT want to use -ffile-prefix-map, as this is the combination of -fmacro-prefix-map and -fdebug-prefix-map. You want to fix your macros, but you do not want to mess with debuginfo, as people deploying debug info will be remapping this to something like /usr/src/debug and people who are downloading the project to build and run it locally will want to correctly pick up the source files from their git checkout.

[eli-schwartz]

switching over from autoconf/make to meson, but there is one issue we would like to solve before allowing release builds to be done in meson. This has to do with using the __FILE__ macro. On the old in-tree builds we'd get debug messages like this in the logs:

DEBUG:     common/lock::lockRelease: (failOnNoLock: false)

But with meson we get this:

DEBUG:     ../pgbackrest/src/common/lock::lockRelease: (failOnNoLock: false)

BTW: with autoconf/make you might still get that exact problem if you do out of source builds, which autotools does not enforce (but make distcheck verifies that it works) and surprisingly few people actually bother to do.

[dwsteele]

you do NOT want to use -ffile-prefix-map

Ah, yes, thanks. I had gotten the mistaken impression from #7485 that -fmacro-prefix-map did not affect __FILE__. I should have checked the gcc docs to be sure.

with autoconf/make you might still get that exact problem if you do out of source builds

True, but release builds are done in-tree and so in practice it is fine. Since in-tree builds are not an option with meson we need another solution.

[tristan957]

FYI, the wlroots project has a pretty good pattern.

https://gitlab.freedesktop.org/wlroots/wlroots/-/blob/master/meson.build#L75

I have copied this idiom in my project with much success. We use the #define to move a string pointer should -fmacro-prefix-map not exist. See https://github.com/hse-project/hse/blob/master/lib/util/src/hse_err.c#L76=.

@dcbaker This issue has been a huge problem for Rust reproducible builds since panics/unwraps produce debug file paths. See https://doc.rust-lang.org/rustc/command-line-arguments.html#--remap-path-prefix-remap-source-names-in-output.

I think adding a b_reproducible=true/false would be good, where it defaults to true. Then it could be applied behind the scenes pretty easily. Unfortunately not all compilers support this, so user needs a way to know if Meson was successful or not. Maybe cc.is_reproducible()? Then one could work in the workaround as shown above.

[emersion]

For reference, the relative_to() PR mentioned above is: https://github.com/mesonbuild/meson/pull/7908

[eli-schwartz]

Note that fs.relative_to() was added in meson 1.3.0

[emersion]

Nice! The wlroots hack can be cleaned up this way: https://gitlab.freedesktop.org/wlroots/wlroots/-/merge_requests/4724

[detly]

A couple of notes from my efforts today before I forget:

• fs.relative_to() exists since 1.3.0 (as @eli-schwartz mentioned, and thanks for IRC advice!), so you can do something like:

  fs = import('fs')
  source_prefix_path = fs.relative_to(
     meson.project_source_root(),
     meson.project_build_root()
  )
  
  c_args = [
     # ...,
     '-fmacro-prefix-map=' + source_prefix_path + '/=',
  ]

• note the trailing /, because source_prefix_path will just be .. and you may not want the resulting path to start with / (you could also do ..=.)

• I had an utterly bizarre time with previous experiments somehow "persisting". I would try '-fmacro-prefix-map=../=lkfdsfsm/'in myc_args, remove it, _delete the build directory and recreate it_, rebuild from my original meson config with no trace of-fmacro-prefix-mapwhatsoever, and still getlkfdsfsm/` in my paths.

  It was ccache. I had to turn it off or run ccache -C in between builds while experimenting. Just writing that down in case it saves someone else 15 minutes.

[detly]

I'll add a couple more notes to this so it's not lost, but please don't take this as a suggestion to use or not use this.

There is an undocumented option for MSVC to do this: /d1trimfile. So if you're looking to implement this transparently in Meson, that may or may not be helpful. Here's an example of how I use it:

fs = import('fs')
source_prefix_path = fs.relative_to(
    meson.project_source_root(),
    meson.global_build_root()
)

testing_c_args = common_c_args + c_compiler.get_supported_arguments(
    # GCC and LLVM clang
    '-fmacro-prefix-map=' + source_prefix_path + '/=',
    # MSVC
    '/d1trimfile:' + source_prefix_path + '/'
)

The other thing to note is that if you do implement this transparently, be aware that there may be two different use cases: removing the global build root or removing the project build root. When using this option in tests, I want the global build root stripped so that my project's test binaries have the same output whether they're run in the actual project, or in a project that has mine as a subproject. OTOH, a calling library probably wants the project path stripped so that paths point to the source files in the subproject directory.