split dynamic linker representations from compilers

[dcbaker]

This series is a start toward breaking the representations of the Compiler and Linker objects. It does this by creating a new DynamicLinker class and internally the compiler dispatches to it.

I have tested with all of the compilers I can, I need help from people with weird or proprietary compilers.

I have no way to test with a number of the proprietary compilers like PGI, ccrx, or cuda. I'll start pinging those people as we get closer to being able to merge this.

Fixes #4837 Fixes #4784 Fixes #5595

[ignatenkobrain]

Flake8 detected 14 issues on 0ad3dc7fbc7cf940a3e0a18eeb783b6250d14a14 Visit https://sider.review/gh/repos/19784232/pulls/5681 to review the issues.

_{posted by Sider}

[dcbaker]

Apple clang + the apple linker now works, real gcc (from homebrew) + the apple linker mostly works.

[dcbaker]

I now have ICC, GCC, Clang, and Apple Clang all working on macos along with the Linux compilers.

[ignatenkobrain]

Flake8 detected 7 issues on 590ee56cf5cebc85d5c7be469da664a162a24334 Visit https://sider.review/gh/repos/19784232/pulls/5681 to review the issues.

_{posted by Sider}

[dcbaker]

I've taken a stab at getting PGI, the arm compilers and the ccrx compiler sorted. No idea if they actually work or not. I think I broke apple again.

[ignatenkobrain]

Flake8 detected 6 issues on 6c8292a9d33324aebf0aaa21c01c3a859c5c0165 Visit https://sider.review/gh/repos/19784232/pulls/5681 to review the issues.

_{posted by Sider}

[dcbaker]

I've introduced a few new bugs on Linux, but I have cl.exe/link.exe and clang-cl/ldd-link.exe working, icl/xilink.exe is kinda working, but I don't konw if that's from this series or previous breakage. No idea on Apple. I've made an attempt at cuda, but it's blind since cuda requires the Nviida drivers to even install. I've realized the ccrx linker implementation I've done is completely bogus. That's all for today.

[ignatenkobrain]

Flake8 detected 9 issues on 2a69fd5490e06bba3cc2806daac97852a2ed09fc Visit https://sider.review/gh/repos/19784232/pulls/5681 to review the issues.

_{posted by Sider}

[ignatenkobrain]

Flake8 detected 10 issues on 91d517f861f9fe00eed9d5eeaba652805098cb24 Visit https://sider.review/gh/repos/19784232/pulls/5681 to review the issues.

_{posted by Sider}

[dcbaker]

Getting very, very close to getting all of the compilers I have access to working at least as well as they did before (There are still some problems with using gcc and non-apple clang on mac, but it's still better than the current status of "nothing works at all")

[dcbaker]

@obilaniu I don't know who to ask about cuda, but you've done a bunch of work on it, so maybe you'd be willing to help?

@ftechz, I know that the ccrx support is still really busted but I at least wanted to make you aware of this.

@scivision, @mohdamerkhalidi @sompen @mbedarka @makise-homura

If I could ask you guys to take a look at the compiler's I've tried to get working (PGI, CCRX, Cuda, LCC, ArmCc, and ArmClang), but I either can't get access to or have no way to run that would be fantastic. If things aren't working a list of failing tests and backtraces would be useful. If you're feeling particularly generous and want to send me patches I'd very much appreciate that.

[obilaniu]

@dcbaker Yeah, there are some blowups. I'm running with CUDA Toolkit 10.1 and a Kepler GPU, and I get a few backtraces. The minimum patch to allow the testcases to run is:

diff --git a/mesonbuild/compilers/cuda.py b/mesonbuild/compilers/cuda.py
index a0019bbe..54d2a453 100644
--- a/mesonbuild/compilers/cuda.py
+++ b/mesonbuild/compilers/cuda.py
@@ -166,7 +166,8 @@ class CudaCompiler(Compiler):
         Converts GNU-style arguments -Wl,-arg,-arg
         to NVCC-style arguments -Xlinker=-arg,-arg
         """
-        return [arg.replace('-Wl', '-Xlinker=', 1) if arg.startswith('-Wl') else arg for arg in args]
+        args = [arg.replace('-Wl', '-Xlinker=', 1) if arg.startswith('-Wl') else arg for arg in args]
+        return [arg.replace(' ', '\\') for arg in args]

     def name_string(self):
         return ' '.join(self.exelist)
@@ -189,6 +190,9 @@ class CudaCompiler(Compiler):
     def get_debug_args(self, is_debug):
         return cuda_debug_args[is_debug]

+    def get_output_args(self, target):
+        return ['-o', target]
+
     def get_werror_args(self):
         return ['-Werror=cross-execution-space-call,deprecated-declarations,reorder']

@@ -212,8 +216,8 @@ class CudaCompiler(Compiler):
     def build_rpath_args(self, env: 'Environment', build_dir: str, from_dir: str,
                          rpath_paths: str, build_rpath: str,
                          install_rpath: str) -> typing.List[str]:
-        return self._cook_link_args(super().build_rpath_args(
-            build_dir, from_dir, rpath_paths, build_rpath, install_rpath))
+        return self._cook_link_args(self.linker.build_rpath_args(
+            env, build_dir, from_dir, rpath_paths, build_rpath, install_rpath))

     def linker_to_compiler_args(self, args):
         return args
@@ -223,3 +227,6 @@ class CudaCompiler(Compiler):

     def compute_parameters_with_absolute_paths(self, parameter_list, build_dir):
         return []
+
+    def get_std_exe_link_args(self):
+        return []
diff --git a/mesonbuild/environment.py b/mesonbuild/environment.py
index d16ce480..a1add802 100644
--- a/mesonbuild/environment.py
+++ b/mesonbuild/environment.py
@@ -912,7 +912,7 @@ class Environment:
             # Luckily, the "V" also makes it very simple to extract
             # the full version:
             version = out.strip().split('V')[-1]
-            linker = self._guess_nix_linker(compiler, for_machine, prefix=['-Xlinker='])
+            linker = self._guess_nix_linker(compiler, for_machine, prefix='-Xlinker=')
             return CudaCompiler(ccache + compiler, version, for_machine, exe_wrap, linker=linker)
         raise EnvironmentException('Could not find suitable CUDA compiler: "' + ' '.join(compilers) + '"')

I also get for every single build, CUDA or not, a DEPRECATION warning that seems spurious:

The Meson build system
Version: 0.51.0
Source dir: /Meson/test cases/common/99 threads
Build dir: /Meson/test cases/common/99 threads/build
Build type: native build
DEPRECATION: Duplicated values in array option is deprecated. This will become a hard error in the future.
Project name: threads
Project version: undefined
C compiler for the build machine: cc (gcc 7.4.0 "cc (SUSE Linux) 7.4.0")
C++ compiler for the build machine: c++ (gcc 7.4.0 "c++ (SUSE Linux) 7.4.0")
C compiler for the host machine: cc (gcc 7.4.0 "cc (SUSE Linux) 7.4.0")
C++ compiler for the host machine: c++ (gcc 7.4.0 "c++ (SUSE Linux) 7.4.0")
Build machine cpu family: x86_64
Build machine cpu: x86_64
Run-time dependency threads found: YES 
Build targets in project: 2
Found ninja-1.8.2 at /usr/bin/ninja
[4/4] Linking target cppthreadprog.

This message appeared somewhere between 0.50.1 and 0.51.0 though.

EDIT: The deprecation warning comes while pkg_config_path is being set, and it complains because I have duplicates in my PKG_CONFIG_PATH environment variable. Why is something this harmless about to become a hard error?

[scivision]

Re: PGI, this was perhaps my first PR to Meson--I didn't make sure that all the tests passed, just that all my various projects worked. So in master, probably since I "added" PGI, there are several project tests broken.

One idea is to have a branch off of master just to fix PGI. There will be unique issues for PGI on Windows since PGI doesn't have it's own C++ compiler on Windows. Maybe a strategy for now is just to list the tests that are broken with PGI, and see that this PR #5681 doesn't break anything additional?

[scivision]

Re: PGI, I made a Gist of Linux tests that fail for PGI on Meson master. https://gist.github.com/scivision/6b6c7833751d9f69b340d594459dbd26

Then I made a PR fixing common/13 pch test: #5731

[dcbaker]

@obilaniu. Thanks for the cuda changes, I'll integrate those. RE: the duplicates warning. It's because we've added a command line argument for pkg_config_path, and the environment is used if the command line argument is not passed, so that's a bad warning in that case. Could you open a bug about it and mention me, I'll fix it.

[jpakkane]

Trying to run this with the ARM compiler under Linux produces the error "armlink.exe not found". Then again, master does the same. This is probably because armcc has only been used on Windows thus far.

[dcbaker]

@jpakkane, I vaguely remember that when the arm compilers were enabled that it was only working for windows. I guess we could try removing the .exe and see what happens. Thanks for looking though.

[dcbaker]

I'll get the D thing sorted today. I've installed a debian chroot so I can get a working GDC (it's still broken on ARCH...)

[dcbaker]

@ftechz @mohdamerkhalidi @sompen @mbedarka @makise-homura; All of the compiler/linker combinations I have access to are now working. Can you have a look at the compilers you've enabled and see if things are working? I would really appreciate patches if possible.

@scivision, what's the status of PGI now.

[scivision]

@dcbaker there is something wrong for PGI

common/1 trivial:

ERROR: Linker pgi does not support allow undefined

fortran/1 basic:

File "/usr3/graduate/mhirsch/code/meson/mesonbuild/compilers/compilers.py", line 771, in get_linker_args_from_envvars return self.linker.get_args_from_envvars() AttributeError: 'NoneType' object has no attribute 'get_args_from_envvars'

[scivision]

@dcbaker fixes for those PGI issues in https://github.com/scivision/meson/tree/dcbaker-dynamic-linker-split

[scivision]

However, the Fortran tests all fail as follows. A straightforward fix perhaps?

Traceback (most recent call last): File "\mesonbuild\mesonmain.py", line 129, in run return options.run_func(options) File "\mesonbuild\msetup.py", line 241, in run app.generate() File "\mesonbuild\msetup.py", line 159, in generate self._generate(env) File "\mesonbuild\msetup.py", line 175, in _generate intr = interpreter.Interpreter(b) File "\mesonbuild\interpreter.py", line 2088, in init self.parse_project() File "\mesonbuild\interpreterbase.py", line 397, in parse_project self.evaluate_codeblock(self.ast, end=1) File "\mesonbuild\interpreterbase.py", line 436, in evaluate_codeblock raise e File "\mesonbuild\interpreterbase.py", line 430, in evaluate_codeblock self.evaluate_statement(cur) File "\mesonbuild\interpreterbase.py", line 441, in evaluate_statement return self.function_call(cur) File "\mesonbuild\interpreterbase.py", line 778, in function_call return func(node, posargs, kwargs) File "\mesonbuild\interpreterbase.py", line 143, in wrapped return f(*wrapped_args, *wrapped_kwargs) File "\mesonbuild\interpreterbase.py", line 174, in wrapped return f(wrapped_args, **wrapped_kwargs) File "\mesonbuild\interpreter.py", line 2737, in func_project self.add_languages(proj_langs, True) File "\mesonbuild\interpreter.py", line 2796, in add_languages success = self.add_languages_for(args, required, MachineChoice.BUILD) File "\mesonbuild\interpreter.py", line 2810, in add_languages_for comp = self.environment.detect_compiler_for(lang, for_machine) File "\mesonbuild\environment.py", line 1236, in detect_compiler_for self.coredata.process_new_compiler(lang, comp, self) File "\mesonbuild\coredata.py", line 720, in process_new_compiler for k, o in comp.get_and_default_options(env.properties[comp.for_machine]).items(): File "\mesonbuild\compilers\compilers.py", line 843, in get_and_default_options compile_args, link_args = self.get_args_from_envvars() File "\mesonbuild\compilers\compilers.py", line 800, in get_args_from_envvars env_link_flags = self.get_linker_args_from_envvars() File "\mesonbuild\compilers\compilers.py", line 771, in get_linker_args_from_envvars return self.linker.get_args_from_envvars() AttributeError: 'NoneType' object has no attribute 'get_args_from_envvars'

[scivision]

...Flang Fortran tests work

[scivision]

on INtel for Linux, the Fortran tests all fail. On intel for Windows they're OK

[dcbaker]

@scivision, the ifort thing is a bug that's always existed manifesting in a strange way. It turns out we've never generated .d files with ifort on mac and Linux, I added the GNU interface for that to the GnuLikeMixin which ifort doesn't support. I've fixed that.

I've also pulled your branch, squashed all of the linker commits into the "Add PGI linker" commit, and re-authored that to you (since you wrote all of the code). I've cleaned up the cleanup commits a bit and merged them into the early cleanup part of the series. I've also fixed (I think) the pgi fortran problem, which was not adding a dynamic linker instance for it. I'm still working out a couple of rebase bugs and then I'll push again.

[dcbaker]

@ftechz @mohdamerkhalidi @sompen @mbedarka @makise-homura; ping

@jpakkane, I saw you enabled armcc on linux, could you try again and see if this works for you?

[dcbaker]

I've updated emscripten to use the CompilerIsLinker infrastructure, and fixed a couple of missed methods in the Ccrx family that were shadowing the same methods in the their linker. I still really need to hear back from @ftechz @mohdamerkhalidi @sompen @mbedarka @makise-homura @jpakkane about whether the compilers/linkers they added/have access to work.

[obilaniu]

@dcbaker It has been two weeks since you first pinged us to verify the more exotic compilers work, and only PGI and NVCC were stepped up. The others are: ARMCC/ARMClang, which ship with Keil, an already-complete development kit; and lcc targetting Elbrus CPUs, for which no CI system is available because they are nearly exclusive to the Moscow Center for SPARC Technologies (MCST).

It may be reasonable to obsolete them in 0.52.0 and drop them in 0.53.0 unless ARM, MCST or volunteers step up, and institute some type of loose "maintainership" of the various compilers.

For my part, I'm willing to maintain for some time the CUDA module and keep NVCC running, although currently the GPU project I have retrieves nvcc manually and uses it in custom_target()s to build CUDA kernels. The reason why mostly has to do with the painful variety of different ways the CUDA Toolkit is installed, officially and unofficially, on many Linux distributions and HPC clusters that exist.

[jpakkane]

ARM on Linux fails with:

ERROR: armlink does not support allow undefined

As a reminder I do have license keys for ARM Linux compilers, which I can share with dev team members if they want to run the tests themselves.

[sompen]

@dcbaker Apologies for the delay. I have checked ArmCc and ArmClang compilers with my local test setup and the tests were initially failing due to missing methods - get_allow_undefined_args and get_soname_args, for Arm family. After overriding the two methods for Arm family, the setup works fine.

Patch snippet for Arm family compiler/linker failure fixes:

diff --git a/mesonbuild/linkers.py b/mesonbuild/linkers.py
index 9f0d7309..a0e58d2c 100644
--- a/mesonbuild/linkers.py
+++ b/mesonbuild/linkers.py
@@ -677,6 +677,14 @@ class ArmDynamicLinker(PosixDynamicLinkerMixin, DynamicLinker):
     def get_std_shared_lib_args(self) -> typing.List[str]:
         return []

+    def get_allow_undefined_args(self) -> typing.List[str]:
+        return []
+
+    def get_soname_args(self, env: 'Environment', prefix: str, shlib_name: str,
+                        suffix: str, soversion: str, darwin_versions: typing.Tuple[str, str],
+                        is_shared_module: bool) -> typing.List[str]:
+        return []
+

[dcbaker]

@sopen, thanks! I've integrated your changes for the arm linkers.

@ftechz, I've added the same changes to the rlink that I added to the arm linker, I think might fix all of it, but at least should get us further along.

[dcbaker]

I've also added an implementation of the illumos dynamic linker, I'd originally planned to do this separately but this is taking so long I just added it here.

[jpakkane]

For Armclang executables work but shared libraries error out with ArmClangdynamicLinker missing get_soname_args. In current master it works. Or at least compiles, I don't know if the Arm compiler can really do share libraries, because according to file the files it outputs are static linked rather than shared objects.

The linker also complains about missing entry points when creating the shared libraries. Which probably means that shared linking is not supported. But it would be great if it did not fail with a traceback at least.

[dcbaker]

@jpakkane ,something like that?

Also, I'm looking into the windows/apple regression.

[jpakkane]

Seems reasonable.

[dcbaker]

It looks like msys is just failing for everyone?

[dcbaker]

@makise-homura, any chance you can give this a whirl?

[dcbaker]

@ftechz, does the updated version work?

[philcaonz]

@ftechz, does the updated version work?

@dcbaker Sorry for the delayed reply. Yes it is working now 👍

[dcbaker]

@jpakkane, are we ready to merge?

[jpakkane]

CI had frozen so I restarted it.

[jpakkane]

And it's done. Thanks for getting this done.

[dcbaker]

I am glad to be done with it!