Meson needs native understanding of symbol version scripts

hughsie commented 6 years ago

Meson doesn't seem to understand much about linkers; most of the projects porting to meson do something like this:

mapfile = 'libgusb.ver'
vflag = '-Wl,--version-script,@0@/@1@'.format(meson.current_source_dir(), mapfile)
gusb = shared_library(
  'gusb',
  sources : [
    'gusb-context.c',
  ],
  soversion : lt_current,
  version : lt_version,
  dependencies : [
    libusb,
  ],
  link_args : vflag,
  link_depends : mapfile,
  install : true
)

Which is fine, unless you discover that OS-X doesn't support --version-script : https://github.com/hughsie/libgusb/issues/11

Rather than have to do evil things to the meson.build file to support OS-X, I would much prefer fr meson to know what a version-script is, rather than just dumping it in a command line. In the case of OS-X the version script could just be ignored. Ideas welcome, thanks!

nirbheek commented 6 years ago

Just for the record, could you explain why you need a version script instead of using symbol visibility prototypes?

hughsie commented 6 years ago

The version script provides a record of the ABI/API over time for each version.

inigomartinez commented 6 years ago

Several gnome and freedesktop packages are also using version scripts, so this feature would be very appreciated.

I would also like to point out that trying to workaround this doesn't seem to be an easy task, given the number of existing linkers and that their behaviour does not seem to be always the same.

npmccallum commented 6 years ago

I'm also using linker scripts in my projects. I'd very much appreciate this.

emersion commented 5 years ago

On macOS, Clang supports -exported_symbols_list, which is a list of exported symbols. Here is an example meson.build that supports both: https://github.com/mkhl/mrsh/blob/f94361fdff16f3a19a764691a418fb6734a634ad/meson.build#L40-L48

I'd like to help with this issue, however I'm not sure what the best design is. Make Meson parse version scripts and convert it to a symbol list when required (not all platforms support symbol versioning)? Or the other way around: make Meson upgrade symbol lists to version scripts if necessary (GCC doesn't support symbol lists), allow users to provide both if they want to have symbol versioning?

jpakkane commented 5 years ago

For symbol exporting the preferred choice is export attributes. If it is at all possible to switch to using those you should. Not only are they the only solution that is properly cross platform (meaning Windows) they also allow the compiler to do optimizations it otherwise would not be able to do.

emersion commented 5 years ago

Export attributes is a GNU extension. I'd rather have two compiler-specific lines in my build file rather than a GNU-specific attribute in all my headers. This also helps having more readable headers.

Windows does support symbol lists (https://docs.microsoft.com/en-us/cpp/build/reference/exports?view=vs-2017).

emersion commented 5 years ago

Anyway, if I understand correctly you'd prefer having Meson use version scripts when supported, and convert to symbol lists otherwise?

npmccallum commented 5 years ago

@emersion My preference is to support a data type in Meson which defines versioned symbol exports. Meson should then do the right thing with this data type. When the target linker doesn't support symbol versioning, Meson should fall back to unversioned symbol exports. Where the target linker doesn't support either versioned or unversioned symbol exports, export all symbols. However, I think all linkers provide some symbol export mechanism.

It should also be possible to detect what the linker/platform supports (similar to meson.get_compiler()). This way developers can override the default behavior.

jpakkane commented 5 years ago

Please bring your comments to #4747.

mon commented 5 years ago

I think the title of this issue should be changed or scope expanded as #4747 specifically refers to symbol exports, whereas linker scripts can be used for other things.

Specifically, I use a linker script for cross compiling to ARM, where it specifies the memory layout. I currently use the (hacky) solution of:

link_args = [
    '-T' + join_paths(meson.current_source_dir(), 'nrf52.ld'),
    '-L' + join_paths(meson.current_source_dir(), SDK_ROOT + 'components/toolchain/gcc'),
]

The -L is due to the linker script using IMPORT and needing references there.

Without very specific args such as link_includes and link_script, folders() or something like .absolute for file paths, this will remain "messy".

phillipjohnston commented 5 years ago

I am also using linker scripts for ARM (very common for embedded). Another benefit of understanding linker scripts is we could trigger a re-link of that file is modified.

(Currently, it will say "no work to do")

mon commented 5 years ago

@phillipjohnston as a workaround, I have had great success with link_depends.

phillipjohnston commented 5 years ago

Thanks for pointing that out. Works with an executable, but not with declare_dependency, which is a bummer for my particular project's organization scheme. (I have linker files organized by embedded platform. You select the target platform for your final executable as a dependency)

blitz commented 5 years ago

I'm trying to build a project with meson that has a linker script that needs to be preprocessed (because it includes headers that define the memory layout). This seems to be way harder to achieve than necessary...

marc-hb commented 4 years ago

I think the title of this issue should be changed or scope expanded as #4747 specifically refers to symbol exports, whereas linker scripts can be used for other things

@hughsie please remove "linker script" from the title of this issue. I landed here and wasted a fair amount of time realizing this issue is really not about https://sourceware.org/binutils/docs-2.32/ld/Scripts.html but something completely different: either "version scripts" or (clearer IMHO) "export maps" but certainly not "linker scripts".

Thanks!

marc-hb commented 4 years ago

Just FYI: Is BUILD_DIR = executable.get_id() OK? For: configure_file('linker_script.ld', BUILD_DIR) #5995

marc-hb commented 4 years ago

New! Add a binutils module #6063

zackw commented 4 years ago

/subscribe

I would like to point out that ELF version scripts are more powerful than symbol visibility attributes or OSX/Windows export lists. They give you the ability to define multiple versions of a single symbol, which is valuable for preserving backward binary compatibility. See https://sourceware.org/binutils/docs/ld/VERSION.html for documentation; see https://github.com/besser82/libxcrypt for a relatively straightforward example of a library that actually uses this.

smcv commented 3 years ago

I would like to point out that ELF version scripts are more powerful than symbol visibility attributes or OSX/Windows export lists. They give you the ability to define multiple versions of a single symbol, which is valuable for preserving backward binary compatibility.

Having versioned symbols is also necessary when two copies of the same library (like libcrypto.so.1.0.2 and libcrypto.so.1.1) might get pulled into the namespace of the same executable, or when two unrelated libraries define symbols with the same naming convention (json-c, jansson and json-glib all have symbols starting with json_, causing crashes when the dependency chain GLib -> libmount -> libcryptsetup -> json-c caused json-c symbols to appear in the namespace of executables that use jansson or json-glib).

There are several possible modes for using ELF version scripts, in increasing order of complexity

to disambiguate between libraries: define a single version that contains all the exported symbols. In at least GNU ld, -Wl,--default-symver will reuse the SONAME as the name of that single version (akheron/jansson#540, https://gitlab.gnome.org/GNOME/json-glib/-/merge_requests/28); or you can use a simple version-script (like IJG libjpeg6b)
to disambiguate between libraries as above, and also have explicit export control over exports as an alternative to visibility attributes (I don't have a good example of this right now)
to disambiguate between libraries and control exports as above, and also track dependencies by the version in which each referenced symbol was added (json-c/json-c#639, util-linux, OpenSSL)
all of the above, and also have older "hidden" versions of the same symbol (glibc is one of the few libraries that does this)

jpakkane commented 3 years ago

Since there are knowledgeable people on this thread, one thing that I have wondered about but never gotten a reasonable answer to is whether "linker scripts" and "symbol version scripts" are the same thing or not? That is, is there ever a case where you would use both for the same output file at the same time?

emersion commented 3 years ago

I don't think they're the same, and I think there are use-cases in which they're both used as the same time (e.g. libc implementations).

zackw commented 3 years ago

@jpakkane They are two different things.

A "linker script" specifies how to put together all the sections of the object files going into the link, to make the executable or shared library or whatever. It specifies things like "the text segment of the executable should be loaded at absolute address 0x800_0000, should be marked readable and executable, and should contain the data from the .text, .init, .fini, etc. sections of the input object files." They're needed for any kind of link operation, but the linker has a built-in script that's correct for "normal" programs; you only need to provide one yourself if you're linking something unusual, like an OS kernel or a bootloader.

A "symbol version script" specifies the names of all the symbols that are to be exported from a shared library, with the version tags to attach to each. They're only needed for shared libraries and for executables that load plugins, but all of those can and probably should have one; it's not nearly as unusual a need.

(Version tags let you offer backward binary compatibility on a per-symbol basis. For instance, suppose the function frobnicate used to take three arguments but now you want to add a fourth. Instead of adding a new name like frobnicate2, you can define frobnicate@MYLIBRARY_1.0 that takes only three arguments, and frobnicate@@MYLIBRARY_1.1 that takes four. The prototype in mylibrary.h would specify four arguments, and newly compiled programs would link against the four-argument version -- that's what the double @@ means -- but old binaries still get the three-argument function that they expect. Internally, the three-argument function would probably tack on a default value for the fourth argument and call the four-argument function. Sadly, the tooling for this is not what it should be; in addition to the version script, you need to put a horrible pile of macros that expand to assembly inserts in your C source files. The compiler devs are working on improvements but I'm not holding my breath.)

GNU ld lets you embed a version script inside a linker script; but it also lets you put each in its own file, and usually that's more convenient for maintenance. I don't know about other linkers.

Ideally, Meson's executable() and shared_library() targets would have separate optional arguments for linker scripts and symbol version scripts (it can make sense to have more than one of either) and would know how to pass each of these to each supported linker. If I were the project manager, though, I'd prioritize the support for symbol version scripts, as this is the more commonly needed feature.

smcv commented 3 years ago

Even though its title mentions linker scripts, this issue report was originally about symbol-version scripts, as introduced by -Wl,--version-script with GNU ld (and so was my recent comment). It would probably be clearer to retitle it to be about symbol-version scripts, which are what the original issue reporter wanted, and have a separate issue for linker scripts.

Some of the people who later commented on this issue genuinely do want linker scripts (for embedded stuff), and not symbol-version scripts.

If I were the project manager, though, I'd prioritize the support for symbol version scripts, as this is the more commonly needed feature.

That's what I'd say too, but perhaps I'm biased by using symbol-version scripts somewhat frequently, and never having wanted a linker script :-)

phillipjohnston commented 3 years ago

If I were the project manager, though, I'd prioritize the support for symbol version scripts, as this is the more commonly needed feature.

That's what I'd say too, but perhaps I'm biased by using symbol-version scripts somewhat frequently, and never having wanted a linker script :-)

This is definitely a "what do you use" bias - I've never used a symbol version script in my life, while I use linker scripts every day. How could I ever agree that's "more commonly needed"? =P

jpakkane commented 3 years ago

For symbol versioning scripts there is already some work in #4747. Visual Studio has a similar thing and so we'd probably want to support both with one syntax.

marc-hb commented 3 years ago

. It would probably be clearer to retitle it to be about symbol-version scripts, which are what the original issue reporter wanted,

@jpakkane or some other admin can you please change "linker scripts" to "version scripts" in the title? I asked the submitter this a few months ago but it doesn't look like he still pays attention to this bug.

and have a separate issue for linker scripts.

There are already a couple links to related discussions if you're interested.

antonysigma commented 3 years ago

I think the title of this issue should be changed or scope expanded as #4747 specifically refers to symbol exports, whereas linker scripts can be used for other things.

Specifically, I use a linker script for cross compiling to ARM, where it specifies the memory layout. I currently use the (hacky) solution of:
link_args = [
    '-T' + join_paths(meson.current_source_dir(), 'nrf52.ld'),
    '-L' + join_paths(meson.current_source_dir(), SDK_ROOT + 'components/toolchain/gcc'),
]
The -L is due to the linker script using IMPORT and needing references there.

Without very specific args such as link_includes and link_script, folders() or something like .absolute for file paths, this will remain "messy".

Hi, I suppose #3047 is still within the scope of "replacing linker script" by -Wl,-T,path/to/linker_script.ld?

The ad hoc solution at https://github.com/mesonbuild/meson/issues/3047#issuecomment-459604971 works for me, except that clang complains about the wrong placement of the -Wl,-T,... argument in the link command. Here's my patch to resolve the bug: https://github.com/mesonbuild/meson/pull/7505. Could you take a look and try it out with clang toolchain?

dcbaker commented 9 months ago

For the issue of joining strings and files for arguments #12287 maybe of help

mesonbuild / meson

Meson needs native understanding of symbol version scripts #3047