Linux ARM port discussion

[jkotas]

• [ ] Build the right flavor of cross-targeting RuyJIT. Linux x64 hosted, arm targeting? https://github.com/dotnet/corert/issues/2381#issuecomment-268010021 talks about issues to watch for.
• [ ] Add command line switch to specify target architecture and platform to ilc (the CoreRT managed->native compiler). The cross targeting support in the CoreRT compiler should be wired in internally for the most part already: https://github.com/dotnet/corert/blob/master/src/Common/src/TypeSystem/Common/TargetDetails.cs
• [ ] Add cross-targeting support to ObjectWriter https://github.com/dotnet/llilc/tree/master/lib/ObjWriter (add a new argument to Initialize method to specify target platform)
• [ ] Port assembly helpers as needed.
• [ ] Implement the R2R helpers as needed. There are some implement already: https://github.com/dotnet/corert/tree/master/src/ILCompiler.Compiler/src/Compiler/DependencyAnalysis/Target_ARM
• [ ] Infrastructure / CI work at least some things work. We can use similar setup as what is used for CoreCLR.

[jkotas]

@sergign60 @Dmitri-Botcharnikov Here are some of the big items I can think of. You are welcomed to add more or ask questions here.

[jkotas]

Also check issues from the earlier ARM port effort for some of the problems you may run into:

https://github.com/dotnet/corert/issues/609 https://github.com/dotnet/corert/issues/458 https://github.com/dotnet/corert/issues/428

[sergign60]

@jkotas thanks!

[jkotas]

• [ ] FP convention differences: RyuJIT is using R11 as FP; CoreRT code is using R7 as FP. Switch RyuJIT to R7 when generating code for CoreRT ABI?

[alpencolt]

@jkotas you wrote:

Add cross-targeting support to ObjectWriter https://github.com/dotnet/llilc/tree/master/lib/ObjWriter (add a new argument to Initialize method to specify target platform)

Still it be needed now for cross build (x86 -> arm)?

[jkotas]

Either that, or build specific objectwriter for each combination (ie one for x86->x86, and one for x86->arm). Similar to how RyuJIT build flavors work.

[alpencolt]

@jkotas I configure llvm on x86 host with:

cmake ../ -DWITH_CORECLR=../../Linux.x86.Debug/ -DLLVM_TARGET_ARCH="ARM;X86" -DLLVM_TARGETS_TO_BUILD="ARM;X86" -DLLVM_DEFAULT_TARGET_TRIPLE=thumbv7-linux-gnueabi -DCMAKE_BUILD_TYPE=Debug -DLLVM_BUILD_LLVM_DYLIB=1 -DLLVM_LINK_LLVM_DYLIB=1 -DLLVM_OPTIMIZED_TABLEGEN=1 -DHAVE_POSIX_SPAWN=0 -DLLVM_ENABLE_PIC=1 -DLLVM_BUILD_TESTS=0 -DLLVM_ENABLE_DOXYGEN=0 -DLLVM_INCLUDE_DOCS=0 -DLLVM_INCLUDE_TESTS=0 -DLLVM_BINUTILS_INCDIR=/usr/include

And objectwriter got correct target triple thumbv7-linux-gnueabi. But throw assertion in ELFObjectWriter::recordRelocation():661:

if (&SecB != &FixupSection) {
      Ctx.reportError(Fixup.getLoc(),
                      "Cannot represent a difference across sections");
      return;
}

Where:

      SecB:
<MCSection Fragments:[
      <MCDataFragment<MCFragment 0xa165e30 LayoutOrder:0 Offset:0 HasInstructions:0 BundlePadding:0>
        Contents:[] (0 bytes)>]>

      FixupSection:
<MCSection Fragments:[
      <MCDataFragment<MCFragment 0xa1672c0 LayoutOrder:0 Offset:0 HasInstructions:0 BundlePadding:0>
        Contents:[] (0 bytes)>,
      <MCFillFragment<MCFragment 0xa1b7838 LayoutOrder:1 Offset:0 HasInstructions:0 BundlePadding:0> Value:� Size:4>,
      <MCDataFragment<MCFragment 0xa1f10d0 LayoutOrder:2 Offset:4 HasInstructions:0 BundlePadding:0>
        Contents:[00,00,00,00,00,00,00,00,00,00,00,00] (12 bytes),
        Fixups:[<MCFixup Offset:0 Value:.text Kind:2>,
                <MCFixup Offset:8 Value:(.Lsec_end5-.text)-0 Kind:2>]>,
      <MCFillFragment<MCFragment 0xa1f0e68 LayoutOrder:3 Offset:16 HasInstructions:0 BundlePadding:0> Value:� Size:4>,
      <MCDataFragment<MCFragment 0xa1f1170 LayoutOrder:4 Offset:20 HasInstructions:0 BundlePadding:0>
        Contents:[00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00] (20 bytes),
        Fixups:[<MCFixup Offset:0 Value:__managedcode Kind:2>,
                <MCFixup Offset:8 Value:(.Lsec_end2-__managedcode)-0 Kind:2>]>]>

It looks I'm losing sight of something. May be you know what to check first?

[jkotas]

No idea

[xoofx]

@jkotas As I wanted to output an ELF file from ILCompiler Windows directly, I have modified objwriter locally on my machine, related to this I believe:

Add cross-targeting support to ObjectWriter https://github.com/dotnet/llilc/tree/master/lib/ObjWriter (add a new argument to Initialize method to specify target platform)

By modifying the objwriter::Init to accept a triple (and if null, it use the default triple as previously). Do you want a PR for that or you had something different in mind?

[jkotas]

PR sounds good. Note that we have moved ObjWriter sources to CoreRT repo - https://github.com/dotnet/corert/tree/master/src/Native/ObjWriter

[xoofx]

Build the right flavor of cross-targeting RuyJIT. Linux x64 hosted, arm targeting? #2381 (comment) talks about issues to watch for.

Ah I missed that one... that could be really bad. I did compile RyuJIT x64-arm (by modifying CoreCLR build manually) and my simple case of codegen was working and the code was running fine but my work case was only using unmanaged code so it was quite simple...

Not sure how to proceed with this... use a bootstrap ilc that would redirect to ilc32 or ilc64, and then load the proper RyuJIT? (that would require to compile ObjWriter to 32/64bits, not sure if we have anything else to rebuild? apart all the Native?)

Also, how do you want the RyuJIT library to be shipped/named: clrjitilc_x64.dll clrjitilc_arm.dll clrjitilc_arm64.dll and loading them on the fly?

[alpencolt]

@xoofx as I remember CoreCLR still not support cross bitness. So for ARM32 you can compile under x86 only. In case of Linux you can use rootfs and in case of Windows it should work as is.

Cross-platfrom (Win to Linux) compilation is another issue:

1. Cpp codegen works well
2. RuyJIT doesn't work. But you can use docker with Linux and it works=)

[jkotas]

We have prior art for building crosstargeting JITs, e.g.:

https://dotnet.myget.org/feed/dotnet-core/package/nuget/runtime.linux-arm.Microsoft.NETCore.Jit/3.0.0-preview1-27101-02 has Linux x64 hosted / arm targeting flavor and Linux x86 hosted / arm targeting flavor.

https://dotnet.myget.org/feed/dotnet-core/package/nuget/runtime.linux-arm64.Microsoft.NETCore.Jit/3.0.0-preview1-27101-02 has Linux x64 hosted / arm64 targeting flavor.

so the compiler packaging work is to bundle multiple flavors into ilc under different names and then load the right one at runtime as you have suggested.

not sure if we have anything else to rebuild

CoreLib needs to be rebuild as well. It is build with platform-specific ifdefs today.

[xoofx]

We have prior art for building crosstargeting JITs, e.g.

Good to know, that's what I was able to do for ARM from a x64 process on Windows in my sample prototype, so the comment https://github.com/dotnet/corert/issues/2381#issuecomment-268010021 still stand but we should try to support cross-targeting with different bitness?

[jkotas]

support cross-targeting with different bitness?

Right. x64 hosted / arm targeting config of the compiler should work fine. The x64 hosted / arm targeting RyuJIT is used in the shipping CoreCLR, so it is pretty well tested.

[xoofx]

Right. x64 hosted / arm targeting config of the compiler should work fine. The x64 hosted / arm targeting RyuJIT is used in the shipping CoreCLR, so it is pretty well tested.

Awesome

so the compiler packaging work is to bundle multiple flavors into ilc under different names and then load the right one at runtime as you have suggested.

Good. So all flavors are already packaged or there are a few left like Windows x64 -> arm? If we need to add a missing package, do you have a quick pointer where I could start with?

I will try to make a PR for getting multiple packages working out-of-the-box with ilc

[jkotas]

all flavors are already packaged

No. We have done the packaging for just the combination we needed.

do you have a quick pointer where I could start with?

https://github.com/dotnet/coreclr/pull/16859/ is a PR that added some of them.

You may want to open a discussion issue on this in CoreCLR repo. I do not know what the right way to do this would be. Also, we are migrating the CI and builds to Azure DevOps, and so the right way to do this in Azure DevOps may be different from the right way to do this in the old system.