Add compiler-rt settings to support baremetal builds and builtins

[WebDrake]

This should improve the cross-compiler support, for example if we want to build for ARM.

Part of https://github.com/ldc-developers/ldc2.snap/issues/97.

@denizzzka FYI. Let's see if this addresses things.

[denizzzka]

Thank you, but I can’t check compilation yet: my computer (enterprise-grade server, hehe!) is not able to build biggest LLVM part due to random hangs.

What about libunwind? This is important because ldc2 currently produces code what depends on it.

[denizzzka]

Idea: maybe it is possible to create "pre-release" snap available from official snap storage?

[WebDrake]

I'll test myself, but once it's done and merged, there will be a copy published to the edge channel. I'll ping you when that's out so you can install and test it.

[denizzzka]

Ok, thanks!

[WebDrake]

What about libunwind? This is important because ldc2 currently produces code what depends on it.

Can you explain in more detail?

In any case, that will be a separate PR, but happy to do it.

[kinke]

I don't see how this would include the desired compiler-rt for an ancient ARM arch - the compiler-rt libs are built for the same (host) arch, so someone would have to build the LDC snap package for that arch (and then most likely only use the libs because a native compiler/LLVM is way too resource-hungry).

I'd look into cross-compiling the compiler-rt libs alone, or use some prebuilt libs (e.g., package libclang-common for Debian) instead. I.e., not touch the snap package at all.

[WebDrake]

I don't see how this would include the desired compiler-rt for an ancient ARM arch

I wasn't sure about that either. Is it a positive change in its own right, though? Or not worth keeping?

[denizzzka]

I don't see how this would include the desired compiler-rt for an ancient ARM arch

Also this is one of most modern arch of MCUs.

• the compiler-rt libs are built for the same (host) arch, so someone would have to build the LDC snap package for that arch (and then most likely only use the libs because a native compiler/LLVM is way too resource-hungry).

I did not complete build process completely, but it seems to me that the build system builds all needed RT libraries and gives names to these libraries that includes name of architecture.

I'd look into cross-compiling the compiler-rt libs alone

Is not have sence because it is almost impossible to build for architectures without this appropriate rt libraries. That is, if there is architecture supported by compiler, then for it you need a compiler runtime library for this architecture too.

or use some prebuilt libs (e.g., package libclang-common for Debian) instead.

This case is better, but won't there be inconspicuous inconsistencies in case of different versions of llvm backend?

I.e., not touch the snap package at all.

Why is this so important?

[denizzzka]

What about libunwind? This is important because ldc2 currently produces code what depends on it.

Can you explain in more detail?

libunwind is part of LLVM project, it is library what provides support of exceptions handling. It is needed by result binaries if compiled without -betterC switch. Situation with this library is same as with compiler RTs - distros provide it only for host architectures.

It is small and it is much easier to build it separately than a LLVM backend, but there is no particular reason not to put it into the bundle.

[denizzzka]

If the snap size is become too large, it can be divided it into "standard" and "full".

[kinke]

I did not complete build process completely, but it seems to me that the build system builds all needed RT libraries and gives names to these libraries that includes name of architecture.

From the CMake output:

Compiler-RT supported architectures: x86_64;i386

Not to mention that (compiling) the libs would obviously require a cross-compiler, and e.g. clang isn't built as part of building LLVM.

[denizzzka]

Not to mention that (compiling) the libs would obviously require a cross-compiler, and e.g. clang isn't built as part of building LLVM.

yep :-(

[denizzzka]

...Then for what are these lines was added?

        -DLLVM_TARGETS_TO_BUILD='AArch64;ARM;Mips;MSP430;NVPTX;PowerPC;RISCV;WebAssembly;X86' \
        -DLLVM_EXPERIMENTAL_TARGETS_TO_BUILD='AVR' \

These targets also unusable without compiler RT. Each user will be forced to download and build LLVM with libraries to use it.

[kinke]

Those lines enable the respective backends for codegen. And no, you don't need a whole LLVM for each target; for most targets, the builtins library from compiler-rt is optional too; for that ARM arch, you apparently need that lib when using 64 bit integers, and cross-compiling that lib with clang shouldn't be rocket science and not take too long to compile.

[WebDrake]

@kinke perhaps it would be good to touch base some time on exactly what needs to be done to ensure the snap package can cross-compile effectively, and what is supported by the current official binary release packages downloadable from GitHub.

That would include being clear about what needs to be in the package, and what should be assumed to be present on the host system where the cross-compile build is happening. (For example: the snap package already relies on an external gcc by default, and requires gcc-multilib to be installed on the host system to support ldc2 -m32.)

Would you be happy to take some time to discuss this? This is an area where I feel inexperienced and would therefore welcome some clear guidance. No worries if you don't have the time or feel that it's my responsibility to RTFM a bit more ;-)

I'm planning to be at DConf this year, so we could also consider making this a hackathon project there.

[denizzzka]

the builtins library from compiler-rt is optional too; for that ARM arch, you apparently need that lib when using 64 bit integers

Are 64bit integers is part of the language? In my opinion - yes. Division of 64 bit integers? Same answer.

And this applies not only to ARM. For each architecture there are these nuances what covered by compiler RT.

So I think it is need to provide this library as we provide compiler itself. And if some special case happens, then user will able disable this library just by exclude it from linking.

[kinke]

perhaps it would be good to touch base some time on exactly what needs to be done to ensure the snap package can cross-compile effectively, and what is supported by the current official binary release packages downloadable from GitHub. That would include being clear about what needs to be in the package, and what should be assumed to be present on the host system where the cross-compile build is happening. (For example: the snap package already relies on an external gcc by default, and requires gcc-multilib to be installed on the host system to support ldc2 -m32.)

AFAIK, the snap package should be identical to the official prebuilt packages in that regard (external C compiler + libs required too for Posix targets). The general approach is described here: https://wiki.dlang.org/Cross-compiling_with_LDC [Adam Ruppe has recently implemented a little tool to simplify this (downloading prebuilt package for the desired target, extracting the libs and extending ldc2.conf), but it's limited to Android targets.]

Would you be happy to take some time to discuss this?

Not really. ;)

[kinke]

And this applies not only to ARM. For each architecture there are these nuances what covered by compiler RT.

As I said, most other targets don't require the builtins lib; sometimes, required builtins (not just for 64/128-bit integers, but also soft-float implementations) are also provided by the C runtime. Edit: E.g., not required for Linux/AArch64, Android/ARMv7-A and Android/AArch64.

So I think it is need to provide this library as we provide compiler itself.

A normal x86 clang package doesn't ship with compiler-rt for the dozens of supported targets either, which would be required when cross-compiling to your ARM arch. And clang isn't a hobby project maintained by a couple of people.

[WebDrake]

AFAIK, the snap package should be identical to the official prebuilt packages in that regard (external C compiler + libs required too for Posix targets).

Yes, this seems like the right division. I think what I was implicitly wanting to know was whether anything extra had to be done to ensure druntime and phobos support for all those architectures (to the extent that the upstream code supports it). But I guess that doesn't need to be pre-bundled into the compiler package: instead the user can use ldc-build-runtime as described here: https://wiki.dlang.org/Building_LDC_runtime_libraries

Would you be happy to take some time to discuss this?

Not really. ;)

No worries. I think my roadmap from here is roughly as follows:

• a couple of small pieces of cleanup on the 1.18 package (ldc2.conf, stage-packages)

• getting 1.19 and 1.20 releases out (long overdue!)

• address the LLD issue via the 1.20 ldc2.conf

... and then return to cross-compiler concerns after this. Main thing is just to get a clear documentation of requirements out there.

[denizzzka]

A normal x86 clang package doesn't ship with compiler-rt for the dozens of supported targets either, which would be required when cross-compiling to your ARM arch. And clang isn't a hobby project maintained by a couple of people.

clang isn't a hobby project maintained by a couple of people => A normal x86 clang package doesn't ship with compiler-rt for the dozens of supported targets either, which would be required when cross-compiling to your ARM arch.

In our D case we cannot afford to configure build for different architectures and distros and package files and separate RT libraries, etc, because community is too small for this. And this is an cause not to do for now as they do in clang.

[kinke]

@denizzzka: I don't get your point and I'm not sure you got mine, but anyway: you're obviously welcome to figure out how to cross-compile the compiler-rt libraries (ideally the whole set, but the builtins library alone would be a start as well) - probably just finding a suited CMake cmdline in the compiler-rt directory, similar to the ones emitted by ldc-build-runtime - and then extend the Azure Pipelines CI script in our LLVM fork to build & package them for other guys targeting your ARM arch baremetal (=> GitHub download of prebuilt libs). Future tools could then use this package to automatically set up such baremetal targets.

[WebDrake]

Closing as superseded by more recent changes to snap package (which now just wraps upstream builds).