Support cross-building for ARM64

[qmfrederik]

Long story short, I'm trying to use NerdBank.GitVersioning in a .NET Core project which runs on a Raspberry Pi, which means I'd need an ARM64/aarch64 version of libgit2, too.

This PR adds:

• A CMakeLists file which configures CMake to cross-compile for arm64
• Updated Dockerfiles (Ubuntu 18.04 for now) which install the cross-compile toolchains and run the build.

After this:

$ file nuget.package/runtimes/ubuntu.18.04-arm64/native/libgit2-*.so 
nuget.package/runtimes/ubuntu.18.04-arm64/native/libgit2-.so:        ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, BuildID[sha1]=e5023c1d76c94853db4600b13094016f2ae1c356, not stripped

[bording]

@qmfrederik This does look interesting, but I have a few questions, which will likely lead to a few additional changes here.

1. You're running on a Raspberry Pi, which distro is that running?
2. What version of .NET Core are you using?
3. Is that an official Microsoft release of .NET Core?

If you look at the supported OS documents for .NET Core: https://github.com/dotnet/core/blob/master/release-notes/2.1/2.1-supported-os.md https://github.com/dotnet/core/blob/master/release-notes/2.2/2.2-supported-os.md

Microsoft is only currently supporting ARM32 for Debian 9 and Ubuntu 16.04+. I'm not sure I'd want to add ARM64 binaries at this point since they don't actually support it.

They are adding ARM64 support for .NET Core 3.0: https://github.com/dotnet/core/blob/master/release-notes/3.0/3.0-supported-os.md

But we don't know a release date for that yet.

If we're going to be start providing ARM binaries at all, I would think we'd want ARM32 as well here if possible.

One of the other reasons I've not done this before is that I'm not currently aware of a way to actually test the binaries. Are you aware of any CI systems that provide ARM cpus to run on?

[bording]

There's also the question of which RIDs we actually have to build for to get the full coverage needed here, which will require some investigation.

[qmfrederik]

@bording Sure, here you go:

1. The host OS is Ubuntu 18.04. You can download official ARM64 images for the Raspberry Pi here: https://wiki.ubuntu.com/ARM/RaspberryPi. I deploy my software to ARM64 using Docker containers, and there the guest is the public ubuntu/bionic image, which has ARM64 support, too.
2. The ARM64 version of .NET Core 2.2 and .NET Core 3.0 Preview.
3. For .NET Core 2.2 yes; for .NET Core 3.0 Preview no. IIIRC .NET Core 3.0 is scheduled for CY2019H2, with the ship date announced at the build conference.

At the time .NET Core 2.2 shipped, the story was that there was support for ARM64 although you'd ideally cross compile from x86 to ARM64. The current story is that ARM64 is only being maintained in master (.NET Core 3.0).

I can definitively extend this PR/create a new PR which cross compiles for ARM32, too, if that helps. Just keep in mind that you can't as easily run an ARM32 binary on an ARM64 version of Linux as you can mix 32-bit and 64-bit apps on Intel architectures.

Regarding to CI, I currently rely on https://cloud.drone.io which provides free access (for open source projects) to some very beefy ARM64 hardware (and x64 as well, for that matter). Again, I can help out here as well if needed.

Regarding the RIDs, I've more or less copied what you've done for x64, but limiting myself for the Debian/Ubuntu flavors. Personally I avoid shipping native libraries for the linux-* RID as you always can find one corner case or another.

[bording]

I find it strange that Microsoft is providing ARM64 binaries for 2.1 and 2.2 given that they say they only support ARM32 and that ARM64 support is only for 3.0.

I also noticed that it looks like ASP.NET Core binaries are ARM32-only. Take a look at the download links here: https://dotnet.microsoft.com/download/dotnet-core/2.2

So, ARM32 seems to be the actual architecture we'd want to support, for shipping versions of .NET Core.... but given that ARM64 support is coming soon with .NET Core 3.0, both ARM32 and ARM64 makes sense to me.

[bording]

For the existing set of RIDs, those were chosen based on trying to have the minimum number of separate binaries possible to cover the supported platforms. The considerations are around these things

1. libc type (glibc vs. musl)
2. libc version (can't use a binary with a newer glibc than the one that ships in the distro
3. OpenSSL version that ships in the distro
4. RID graph compatibility

The existing choices for RIDs likely won't be right to just include ARM32/ARM64 versions, so we'll need to figure out the actual set we'll need.

[qmfrederik]

So, here are the dependencies for arm64 (I've added the Dockerfiles I've used for reference purposes):

$ readelf -d nuget.package/runtimes/debian.9-arm64/native/libgit2-.so 

Dynamic section at offset 0xf5b70 contains 29 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [librt.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libssl.so.1.1]
 0x0000000000000001 (NEEDED)             Shared library: [libcrypto.so.1.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]

$ readelf -d nuget.package/runtimes/ubuntu.16.04-arm64/native/libgit2-.so 

Dynamic section at offset 0xfdb70 contains 29 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libssl.so.1.0.0]
 0x0000000000000001 (NEEDED)             Shared library: [libcrypto.so.1.0.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [ld-linux-aarch64.so.1]

$ readelf -d nuget.package/runtimes/ubuntu.18.04-arm64/native/libgit2-.so 

Dynamic section at offset 0x109b70 contains 29 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libssl.so.1.1]
 0x0000000000000001 (NEEDED)             Shared library: [libcrypto.so.1.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [ld-linux-aarch64.so.1]

$ readelf -d nuget.package/runtimes/ubuntu.18.10-arm64/native/libgit2-.so 

Dynamic section at offset 0x111b70 contains 29 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libssl.so.1.1]
 0x0000000000000001 (NEEDED)             Shared library: [libcrypto.so.1.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [ld-linux-aarch64.so.1]

You could try to go with the builds for Debian 9 as linux-arm64, they should work on Ubuntu 18.04 and 18.10, and a dedicated file for Ubuntu 18.06.

The only other distro which is/will be supported for arm64 is Alpine Linux. It looks like Alpine doesn't ship with a cross-compilation toolchain (at least I couldn't find the required packages), and the only option is to cross-compile for Alpine arm64 from another distro such as Ubuntu - e.g. https://github.com/dotnet/coreclr/issues/8927 describes how we did it for CoreCLR itself.

I don't have a need for Alpine arm64 at the moment; would it be OK to leave that for later/someone else?

[bording]

I don't have a need for Alpine arm64 at the moment; would it be OK to leave that for later/someone else?

Sure, we can always work on that later.

Similar to my comment on the ARM32 PR: https://github.com/libgit2/libgit2sharp.nativebinaries/pull/92#issuecomment-479587618

I think we could try for debian-arm64 and ubuntu.16.04-arm64 builds, but I'd want to test that out to verify everything works.

I don't have access to any ARM hardware myself, so would you be able to help test this? I'm trying to think of what would be the easiest way to actually try using the binaries to verify they work.

At some point, I'd like to get the LibGit2Sharp repo running tests on all the relevant repos (https://github.com/libgit2/libgit2sharp/issues/1634), but that's not currently in place.

[bording]

@qmfrederik Can you update the PR so that we build the following binaries:

• debian-arm64 using your debian 9 docker file
• ubuntu.16.04-arm64 using your Dockerfile.ubuntu.16.04-arm64 file

If you do that, I should be able to build a preview version of the binaries package that you could use to run the LibGit2Sharp tests.

I'm hoping the debian binary will also work fine on Ubuntu 18.04 and 18.10.

[qmfrederik]

Yes, let me take care of that today. Sorry, lost track of this during the Easter holiday.

[qmfrederik]

@bording Yes, I think that would work - the Debian image would be used for Debian 9, Ubuntu 18.04 and Ubuntu 18.10. They all use the same version of OpenSSL so we should be good there; and then there's a special case for Ubuntu 16.04.

I rebased on top of master and pushed an additinal commit which incorporates your feedback. You'll probably want to squash all commits if you merge this.

If you want me to squash first, let me know.

[bording]

@qmfrederik I have uploaded LibGit2Sharp.NativeBinaries 2.0.280-armpreview01. You should be able to use that with the LibGit2Sharp repo to verify that the binaries do in fact work like we think they will.

Once you've done that, I'll merge this (most likely squashing things when I do), and then we can do the same thing for the arm32 PR.

Once both of them are in, I'll push up a real package update and then I'll create a LibGit2Sharp 0.27 prerelease that uses it.

[qmfrederik]

@bording Thanks, I ran a test and it appears the libraries are indeed working correctly.

Here's the full test output:

ubuntu@ubuntu:~/git/libgit2sharp/LibGit2Sharp.Tests$ dotnet test
Test run for /home/ubuntu/git/libgit2sharp/bin/LibGit2Sharp.Tests/netcoreapp3.0/LibGit2Sharp.Tests.dll(.NETCoreApp,Version=v3.0)
Microsoft (R) Test Execution Command Line Tool Version 16.0.1
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
[xUnit.net 00:00:33.65]     LibGit2Sharp.Tests.DiffTreeToTreeFixture.RenameDetectionObeysConfigurationSetting [SKIP]
Skipped  LibGit2Sharp.Tests.DiffTreeToTreeFixture.RenameDetectionObeysConfigurationSetting
[xUnit.net 00:01:01.17]     LibGit2Sharp.Tests.StatusFixture.CanIncludeStatusOfUnalteredFiles [FAIL]
Failed   LibGit2Sharp.Tests.StatusFixture.CanIncludeStatusOfUnalteredFiles
Error Message:
 Assert.Equal() Failure
Expected: String[] ["1.txt", "1/branch_file.txt", "branch_file.txt", "new.txt", "README"]
Actual:   String[] ["1.txt", "1/branch_file.txt", "README", "branch_file.txt", "new.txt"]
Stack Trace:
   at LibGit2Sharp.Tests.StatusFixture.CanIncludeStatusOfUnalteredFiles() in /home/ubuntu/git/libgit2sharp/LibGit2Sharp.Tests/StatusFixture.cs:line 643
[xUnit.net 00:01:05.86]     LibGit2Sharp.Tests.CloneFixture.CanInspectCertificateOnClone(url: "git@github.com:libgit2/TestGitRepository.git", hostname: "github.com", certType: typeof(LibGit2Sharp.CertificateSsh)) [SKIP]
Skipped  LibGit2Sharp.Tests.CloneFixture.CanInspectCertificateOnClone(url: "git@github.com:libgit2/TestGitRepository.git", hostname: "github.com", certType: typeof(LibGit2Sharp.CertificateSsh))
[xUnit.net 00:01:12.95]     LibGit2Sharp.Tests.CloneFixture.CanCloneWithCredentials [SKIP]
Skipped  LibGit2Sharp.Tests.CloneFixture.CanCloneWithCredentials
[xUnit.net 00:02:04.35]     LibGit2Sharp.Tests.BlobFixture.CanGetBlobAsFilteredText(autocrlf: "true", expectedText: "hey there\r\n") [SKIP]
Skipped  LibGit2Sharp.Tests.BlobFixture.CanGetBlobAsFilteredText(autocrlf: "true", expectedText: "hey there\r\n")
[xUnit.net 00:02:04.85]     LibGit2Sharp.Tests.BlobFixture.CanReadBlobFilteredStream(autocrlf: "true", expectedContent: "hey there\r\n") [SKIP]
Skipped  LibGit2Sharp.Tests.BlobFixture.CanReadBlobFilteredStream(autocrlf: "true", expectedContent: "hey there\r\n")
[xUnit.net 00:02:37.28]     LibGit2Sharp.Tests.FetchFixture.CanFetchIntoAnEmptyRepositoryWithCredentials [SKIP]
Skipped  LibGit2Sharp.Tests.FetchFixture.CanFetchIntoAnEmptyRepositoryWithCredentials
[xUnit.net 00:02:57.13]     LibGit2Sharp.Tests.StageFixture.CanStageANewFileWithAFullPath(ignorecase: True) [SKIP]
Skipped  LibGit2Sharp.Tests.StageFixture.CanStageANewFileWithAFullPath(ignorecase: True)
[xUnit.net 00:03:04.11]     LibGit2Sharp.Tests.GlobalSettingsFixture.LoadFromSpecifiedPath(architecture: "x86") [SKIP]
[xUnit.net 00:03:04.13]     LibGit2Sharp.Tests.GlobalSettingsFixture.LoadFromSpecifiedPath(architecture: "x64") [SKIP]
[xUnit.net 00:03:04.16]     LibGit2Sharp.Tests.GlobalSettingsFixture.CanRetrieveValidVersionString [FAIL]
Skipped  LibGit2Sharp.Tests.GlobalSettingsFixture.LoadFromSpecifiedPath(architecture: "x86")
Skipped  LibGit2Sharp.Tests.GlobalSettingsFixture.LoadFromSpecifiedPath(architecture: "x64")
Failed   LibGit2Sharp.Tests.GlobalSettingsFixture.CanRetrieveValidVersionString
Error Message:
 The following version string format is enforced:Major.Minor.Patch[-previewTag]+{LibGit2Sharp_abbrev_hash}.libgit2-{libgit2_abbrev_hash} (x86|x64 - features). But found "1.0.0.2 (x64 - Threads, Https, NSec)" instead.
Expected: True
Actual:   False
Stack Trace:
   at LibGit2Sharp.Tests.GlobalSettingsFixture.CanRetrieveValidVersionString() in /home/ubuntu/git/libgit2sharp/LibGit2Sharp.Tests/GlobalSettingsFixture.cs:line 42
[xUnit.net 00:03:04.37]     LibGit2Sharp.Tests.RepositoryFixture.CanListRemoteReferencesWithCredentials [SKIP]
Skipped  LibGit2Sharp.Tests.RepositoryFixture.CanListRemoteReferencesWithCredentials
[xUnit.net 00:03:19.02]     LibGit2Sharp.Tests.NetworkFixture.CanListRemoteReferencesWithCredentials [SKIP]
Skipped  LibGit2Sharp.Tests.NetworkFixture.CanListRemoteReferencesWithCredentials

Total tests: 1382. Passed: 1369. Failed: 2. Skipped: 11.
Test Run Failed.
Test execution time: 3.7021 Minutes

• CanIncludeStatusOfUnalteredFiles seems to be a collation-related issue
• CanRetrieveValidVersionString fails because NB.GV is not working on ARM64, and I manually created a ThisAssembly class, with a random version string.

So as far as I can see, things look good...

[bording]

To clarify, which version of Ubuntu was this?

CanIncludeStatusOfUnalteredFiles seems to be a collation-related issue

Hmm, that seems strange. Something to figure out later I suppose.

CanRetrieveValidVersionString fails because NB.GV is not working on ARM64, and I manually created a ThisAssembly class, with a random version string.

Of course, since it uses LibGit2Sharp! Will have to think about how to handle this for the first release so NB.GV can update!

[qmfrederik]

To clarify, which version of Ubuntu was this?

Ubuntu 18.04, so it seems like building building for Debian 9 and packaging it as debian-arm64 works.