search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.29k stars 4.74k forks source link

Enabling Linux ARM32 for .NET Core #2548

Closed hqueue closed 4 years ago

hqueue commented 7 years ago

Host: Ubuntu 14.04 x64 Target: Ubuntu 14.04 or(and) 16.04 ARM, Ubuntu 14.04 ARM Softfp and Tizen

To enable core-setup for Linux ARM32, we are going to follow steps suggested in PR dotnet/core-setup#712.

ARM hardfp

Common

CoreCLR

CoreFX

core-setup For core-setup, we will follow steps below.

For example, we want to build core-setup for Linux/ARM using following command.

$ ./build.sh  --env-vars DISABLE_CROSSGEN=1,TARGETPLATFORM=arm,TARGETRID=ubuntu.14.04-arm,CROSS=1,ROOTFS_DIR=/home/jyoung/git/dotnet/rootfs-coreclr/arm/

ASP.NET

ARM softfp

We will use armel for arm-softfp through out the dotnet.

Common

Debian.8

CoreFX - Part1

CoreCLR

CoreFX - Part2

core-setup For core-setup, we will follow steps below.

./build.sh --skiptests --env-vars DISABLE_CROSSGEN=1,TARGETPLATFORM=armel,TARGETRID=debian.8-armel,CROSS=1,ROOTFS_DIR=/home/jyoung/git/dotnet/rootfs/armel

Tizen 4.0.0

CoreFX - Part1

CoreCLR

CoreFX - Part2

core-setup For core-setup, we will follow steps below.

dotnet-ci

./build.sh -c Release --skiptests --env-vars DISABLE_CROSSGEN=1,TARGETPLATFORM=armel,TARGETRID=tizen.4.0.0-armel,CROSS=1,ROOTFS_DIR=/home/jyoung/git/dotnet/rootfs/armel-tizen

Results

ubuntu.14.04-arm (Latest build is available at https://github.com/dotnet/core-setup#daily-builds) dotnet-ubuntu.14.04-arm.1.2.0-beta-001291-00.tar.gz (Last updated on Jan 19) dotnet-sdk-ubuntu.14.04-arm.1.0.0-preview5-004431.tar.gz ubuntu.16.04-arm (Latest build is available at https://github.com/dotnet/core-setup#daily-builds) dotnet-ubuntu.16.04-arm.1.2.0-beta-001291-00.tar.gz (Last updated on Jan 19) dotnet-sdk-ubuntu.16.04-arm.1.0.0-preview5-004431.tar.gz debian.8-armel dotnet-debian.8-armel.1.2.0-beta-001271-00.tar.gz tizen.4.0.0-armel dotnet-tizen.4.0.0-armel.1.2.0-beta-001273-00.tar.gz

gkhanna79 commented 7 years ago

CC @schellap @ramarag @janvorli

janvorli commented 7 years ago

As I have mentioned in the discussion in dotnet/core-setup#724, I believe that we should be consistent in the way we do cross build over our repos. That means to use rootfs and to do the cross build using clang and not GCC. The same way we do it in CoreCLR and CoreFX. I hope we can reuse the logic from the build.sh in CoreCLR here.

gkhanna79 commented 7 years ago

@hqueue @hseok-oh Is Ubuntu 14.04 an intended target for Arm32?

hqueue commented 7 years ago

@gkhanna79 AFAIK default rid is ubuntu.14.04-arm for coreclr ARM and default ARM rootfs is trusty which is ubuntu 14.04.

hseok-oh commented 7 years ago

@hqueue @gkhanna79 RaspberryPI2 used ubuntu 14.04. (but no longer maintained) https://wiki.ubuntu.com/ARM/RaspberryPi

gkhanna79 commented 7 years ago

@hseok-oh If Ubuntu 14.04 is not maintained and supported, is there a specific reason to support it for Linux Arm32? If not, it will simplify bunch of things (e.g. libcoreclrtraceptprovider.so builds just fine for 16.04 but needs additional work for 14.04).

What do you think?

gkhanna79 commented 7 years ago

AFAIK default rid is ubuntu.14.04-arm for coreclr ARM and default ARM rootfs is trusty which is ubuntu 14.04

I don't know who chose those defaults or why. Could it be cause Tizen OS is based upon Ubuntu 14.04? do we build libcoreclrtraceptprovider.so for Tizen?

gkhanna79 commented 7 years ago

@kouvel This is the Linux Arm32 main work-list issue.

gkhanna79 commented 7 years ago

CC @schellap

hqueue commented 7 years ago

AFAIK default rid is ubuntu.14.04-arm for coreclr ARM and default ARM rootfs is trusty which is ubuntu 14.04

I don't know who chose those defaults or why. Could it be cause Tizen OS is based upon Ubuntu 14.04? do we build libcoreclrtraceptprovider.so for Tizen?

@gkhanna79 I don't think it's related to Tizen OS. The most possible explanation will be that ubuntu 14.04 is selected because of devices (e.g. ARM Emulator, Raspberry pi 2 and etc.) which were used to bring up CoreCLR ARM. Please check out https://github.com/dotnet/coreclr/issues/3805 for ARM emulator which is used at the very early stage of CoreCLR ARM32 bring up.

gkhanna79 commented 7 years ago

Thanks for the explanation @hqueue. For now, we can continue to build for both 14.04 and 16.04 and see how far we will need the 14.04 implementation.

With regards to the emulator, I noticed that in CoreCLR repo, we bootup the emulator and setup the rootfs within it and then perform the build. This is different from the build instructions we have at https://github.com/dotnet/coreclr/blob/master/Documentation/building/cross-building.md where it comprises of two steps:

  1. Build rootfs for an architecture
  2. Perform a cross build of the repo

We strive hard to ensure that CI builds the same way as any developer would. Since the above steps are how any developer in the community would be expected to build for Arm32, what do you think of updating the CI script to do just that and not rely on emulator to be present for performing the build?

Secondly, the emulator should only be required to deploy binaries and execute tests. I notice that we only run 22 tests within the emulator, even though the repo has 11K+ tests. Do these 22 tests add any value from the CI perspective (e.g. here is a log of a recent PR that shows the 22 tests that ran - https://ci.dot.net/job/dotnet_coreclr/job/master/job/arm_emulator_cross_debug_ubuntu_prtest/1080/consoleFull) since they dont seem to cover key scenarios where breaks maybe introduced?

hqueue commented 7 years ago

@gkhanna79 For the first question, I will call @sjsinju @wateret, since I'm not aware of the latest of ARM CI.

@sjsinju @wateret Can you please answer the first question above regarding CI and ARM emulator ?

For the second one, I also think tests are not sufficient to cover key sccenarios. We will look into them.

wateret commented 7 years ago

@gkhanna79 CI cross-builds the same way as any developers would. You can check out arm32_ci_script.sh#L227. The emulator is only for running tests. Of course, we still need pre-built rootfs to cross-build though.

One more thing that may be confusing is that the CI has arm-softfp rootfs only which cannot be created by 'any developers'. There is my work on CoreFX for enabling arm-hardfp(ubuntu 14.04). After it's done I will apply it to CoreCLR as well.

jyoungyun commented 7 years ago

@gkhanna79 I couldn't find where libuv.so is built. Could you give me an advice?

qmfrederik commented 7 years ago

@jyoungyun not sure if it's the way libuv is built for .NET, but for raspberry pi the instructions so far have been to download it from the libuv project & compile it:

See the these comments from @leemgs related to dotnet/coreclr#6321

rpi2@arm# sudo apt-get install gyp
rpi2@arm# wget http://dist.libuv.org/dist/v1.0.0-rc1/libuv-v1.0.0-rc1.tar.gz 
(The latest version is v1.9.1: http://dist.libuv.org/dist/v1.9.1/libuv-v1.9.1.tar.gz)
rpi2@arm# tar -xvf libuv-v1.0.0-rc1.tar.gz
rpi2@arm# cd libuv-v1.0.0-rc1/
rpi2@arm# ./gyp_uv.py -f make -Duv_library=shared_library
rpi2@arm# make -C out
rpi2@arm# sudo cp out/Debug/lib.target/libuv.so /usr/lib/libuv.so.1.0.0-rc1
rpi2@arm# sudo ln -s libuv.so.1.0.0-rc1 /usr/lib/libuv.so.1

That worked for me to get Kestrel up & running on Raspberry Pi; but note that newer versions of libuv are now available at http://dist.libuv.org/dist/

gkhanna79 commented 7 years ago

CI cross-builds the same way as any developers would.

@wateret My point is that we should decouple cross building in CI from emulator. The nuance is that in CI, we setup rootfs from emulator and then perform cross-build while a regular developer will not. My suggestion is to fix the script to do the following:

1) Without the emulator, perform the cross build (e.g. cross/build-rootfs.sh arm) 2) Load the emulator only for running tests. Thus, if we were to decide that emulator may no longer be needed for testing for some reason, then cross-build continues to function as it is expected.

gkhanna79 commented 7 years ago

@hqueue did you find anything interesting about running the 22 tests in CI?

jyoungyun commented 7 years ago

@qmfrederik Thank you for your answer. What you are talking about is how to download libuv.so directly. But I want to know where the libuv.nupkg is generated. The NETCoreApp is dependent on libuv module so I wanna generate libuv.nupkg for arm.

wateret commented 7 years ago

@gkhanna79

we setup rootfs from emulator and then perform cross-build while a regular developer will not.

Just make things clear, the emulator image is nothing but pre-built rootfs. Of course the emulator also needs it for running tests.

For the first one you suggested, basically I agree with you. However it takes pretty long to build rootfs with build-rootfs.sh. As far as I remember it took 1-2 hours on my linux machine. I'm not sure if it is acceptable. For the second one, the emulator is loaded only for running tests already. Build does not run on the emulator. It cross-builds.

hqueue commented 7 years ago

did you find anything interesting about running the 22 tests in CI?

@gkhanna79 Those 22 tests are TCs where we observed regression frequently when brining-up CoreCLR for Linux ARM32. Therefore the coverage of 22 tests does not cover the code which is not modified freqeuently. However we cannot run all TCs (11K+) with ARM emulator becasue it took very long time and we faced timeout in CI environment.

I also think we need more TCs for CI, but not sure about how to choose candidates. Can you suggest any options ?

cc: @jyoungyun

qmfrederik commented 7 years ago

@jyoungyun I think the libuv NuGet package is located here: https://github.com/aspnet/libuv-package?files=1

jyoungyun commented 7 years ago

@qmfrederik Thank you! I will check that repository.

Finally, I succeeded in obtaining a dotnet-ubuntu-arm.1.2.0-beta-001206-00.tar.gz file, which confirmed that it works well on Raspberry PI2. I built the core-setup repository using the local nupkg source and these are the results built from CoreCLR and CoreFX(including https://github.com/dotnet/corefx/pull/14655 patch). If CoreCLR and CoreFX nupkgs are uploaded in the NugetServer or anywhere, we can generate the .NET Core Runtime for Ubuntu ARM. I have tested only on Ubuntu 14.04 and will be testing on 16.04 later.

jyoung@DXL-Workstation:~/git/dotnet/core-setup-jy/artifacts/ubuntu.14.04-arm/packages$ ls
total 33M
-rwxrw---- 1 jyoung jyoung  28K Dec 21 18:18 dotnet-deb-tool.1.0.1-t-beta-001206.nupkg
-rw-rw---- 1 jyoung jyoung 226K Dec 21 18:19 dotnet-hostfxr-ubuntu-arm.1.2.0-beta-001206-00.tar.gz
-rw-rw---- 1 jyoung jyoung  17M Dec 21 18:19 dotnet-sharedframework-ubuntu-arm.1.2.0-beta-001206-00.tar.gz
-rw-rw---- 1 jyoung jyoung  17M Dec 21 18:19 dotnet-ubuntu-arm.1.2.0-beta-001206-00.tar.gz
drwxrwx--- 2 jyoung jyoung 4.0K Dec 21 17:04 intermediate
pi@raspberrypi:~/Downloads/24919/shared/Microsoft.NETCore.App/1.2.0-beta-001206-00 $ ./dotnet hello.exe 
Hello World
qmfrederik commented 7 years ago

Awesome! If there's any way for you to share the NuGet packages/tarballs, I'd be happy to try them out on a RPi 2 & 3, and see how far I get with our .NET Core-based product :)

gkhanna79 commented 7 years ago

I succeeded in obtaining a dotnet-ubuntu-arm.1.2.0-beta-001206-00.tar.gz file

This is excellent news @jyoungyun! Thank you all for driving this through :)

The LibUV package does come from ASP.NET repository. @Eilon Can you please advise which repo to enlist to build this package from?

gkhanna79 commented 7 years ago

If CoreCLR and CoreFX nupkgs are uploaded in the NugetServer or anywhere

This is the work I am tracking to get done as part of the E2E pipeline build support across the three .NET Core repos (CoreCLR, CoreFX, Core-Setup).

gkhanna79 commented 7 years ago

@wateret

As far as I remember it took 1-2 hours on my linux machine

That is interesting. For me, creating a new rootfs takes about 7 or so mins on my USB 3.0 SSD for both Trusty and Xenial builds. Everything else is cost of a regular cross build from there on. Hence, my question around why we need to pickup rootfs from emulator and not just fetch it everytime. Building upon this, to enable CI build for Ubuntu 16.04 arm, are you planning to add another emulator that will contain rootfs?

@hqueue

Can you suggest any options ?

Do you have any automation around tests on real HW? My current thinking is to evaluate doing so in favor of emulator int he foreseeable future.

janvorli commented 7 years ago

I can also confirm that creating a new rootfs takes a couple of minutes on my Linux box. It is probably related to the speed of the internet connection, since the process fetches all the necessary packages.

Eilon commented 7 years ago

The libuv source code is 3rd party and is here: https://github.com/libuv/libuv

The ASP.NET team's build process for compiling libuv is in our own repo here: https://github.com/aspnet/libuv-build/

And it references the libuv source code via a Git submodule: https://github.com/aspnet/libuv-build/tree/dev/submodules

And finally, the ASP.NET team has a repo for taking the libuv binaries and packaging them into a NuGet package: https://github.com/aspnet/libuv-package

Eilon commented 7 years ago

cc @moozzyk who knows the most about the libuv build.

moozzyk commented 7 years ago

What @Eilon said. Packages for different OS/architectures (currently Windows Win32/x64/arm, Linux (64-bit), macOS) are built from this repo https://github.com/aspnet/libuv-build/ (which pulls libuv code as a submodule) and pushed to the myget feed as OS specific Microsoft.AspNetCore.Internal.libuv-* packages. The libuv-package repo is responsible for pulling the build packages and creating the über libuv package that contains bits for all supported architectures.

jyoungyun commented 7 years ago

@qmfrederik I attached a .NET Core Runtime tarball for Ubuntu 14.04 ARM. When I am ready for Ubuntu 16.04, I will share the file here. (Updated on 12/26 : I attached a .NET Core Runtime tarball for Ubuntu 16.04 ARM.) dotnet-ubuntu-arm.1.2.0-beta-001206-00.tar.gz (Ubuntu 14.04 ARM) dotnet-ubuntu-arm.1.2.0-beta-001206-00.tar.gz (Ubuntu 16.04 ARM)

@gkhanna79

This is the work I am tracking to get done as part of the E2E pipeline build support across the three .NET Core repos (CoreCLR, CoreFX, Core-Setup).

Thank you for your support!

@Eilon @moozzyk @qmfrederik Thank you for comments. In fact, we do not need libuv.so at present. However, it seems that libuv for Linux arm is required to release .NET Core Runtime for Ubuntu ARM. Should I add libuv package for Linux arm? What do you think?

moozzyk commented 7 years ago

@jyoungyun - the only thing that currently requires libuv is the Asp.NET Core Http Server - Kestrel. If you don't use Asp.NET Core you can get by without libuv. Currently we use kind of a homebrew way to build libuv for different platforms (one of the reasons was that their scripts did not support Windows ARM) but eventually it would be good just to build libuv using scripts from their repo which would make enabling new platforms easier.

hqueue commented 7 years ago

FYI. I've ran above dotnet tarball at Raspberry Pi3 which is ubuntu.16.04-arm. At least, it works well with HelloWorld.exe and other simple applications. :) Thanks @jyoungyun !

hqueue commented 7 years ago

Do you have any automation around tests on real HW? My current thinking is to evaluate doing so in favor of emulator int he foreseeable future.

@gkhanna79 Unfortunately we don't have automated CoreCLR/CoreFX tests on HW and it also takes several hours to perform full CoreCLR unit tests on ARM device, e.g. Rpi2 or Rpi3. We are also interested in the test infra but I'm afraid that full CoreCLR/CoreFX tests may not be feasible for ARM HW due to timeout, e.g. CoreFX tests run much longer than CoreCLR. Therefore I think we may have to choose tests even when the infra is available.

SteveDesmond-ca commented 7 years ago

Looks like dotnet/core-setup#712 has been merged, so we can check that one off!

gkhanna79 commented 7 years ago

Update build script to enable cross building (PR will be posted soon.)

@jyoungyun I believe you have completed this for Core-Setup, right? If so, can you please add PR details above and check this off :) ?

gkhanna79 commented 7 years ago

Update build script to enable downloading and packaging for Linux/ARM.

@hqueue What is this item about (it is listed under Core-Setup section)?

gkhanna79 commented 7 years ago

Should I add libuv package for Linux arm? What do you think?

@jyoungyun I think it will be a good idea to get the changes ready to build lib-uv for arm32 (as tracked by https://github.com/aspnet/libuv-build/issues/19).

moozzyk commented 7 years ago

@gkhanna79 @jyoungyun - is Linux/arm RID sufficient though? How about hard float vs. soft float? The linux-arm RID does not convey this information....

SteveDesmond-ca commented 7 years ago

Convention in the dotnet repos seems to be that arm=armhf, and softp must be specified explicitly. Perhaps an arm-softp rid could be added if necessary

On Dec 27, 2016 7:01 PM, "Pawel Kadluczka" notifications@github.com wrote:

@gkhanna79 https://github.com/gkhanna79 @jyoungyun https://github.com/jyoungyun - is Linux/arm RID sufficient though? How about hard float vs. soft float? The linux-arm RID does not convey this information....

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/dotnet/core-setup/issues/725#issuecomment-269400424, or mute the thread https://github.com/notifications/unsubscribe-auth/AAL9OFobHMUWCNgwxEvC8nLJMdvM1PJtks5rMabfgaJpZM4LD3ey .

hqueue commented 7 years ago

I believe you have completed this for Core-Setup, right? If so, can you please add PR details above and check this off :) ? What is this item about (it is listed under Core-Setup section)?

@gkhanna79 At first, we thought they were another tasks and I was planning to prepare seperate PR for each task. But as you already noticed, PR dotnet/core-setup#712 has enabled them altogether. I will checked them off. Thank you for notifying us :)

hqueue commented 7 years ago

is Linux/arm RID sufficient though? How about hard float vs. soft float? The linux-arm RID does not convey this information....

@moozzyk @stevedesmond-ca @gkhanna79 @jyoungyun I also have same question since when this task was started and for example there is no distinction in Ubuntun rids among different float ABI (e.g. hardfp, soft, softfp). This may affect all other dotent repos inclduing CoreCLR and CoreFX.

jyoungyun commented 7 years ago

Convention in the dotnet repos seems to be that arm=armhf, and softp must be specified explicitly. Perhaps an arm-softp rid could be added if necessary

@stevedesmond-ca @moozzyk @gkhanna79 @hqueue Linux RID does not contain the float ABI information, but we already use in combination with RID and float ABI information as a buildarch. As @stevedesmond-ca mentioned above, if necessary, we can add an arm-softfp rid like CoreCLR and CoreFX repos. I would like to keep the existing structure about RID and float ABI information in core-setup and libuv-build. And we are going to add arm-softfp to core-setup soon for supporting Tizen OS. Tizen only supports arm-softfp.

@gkhanna79 All of work for building core-setup for Ubuntu ARM are already merged via dotnet/core-setup#712. And I will post PR soon to disable GenerateDebs target regarding dotnet/core-setup#849.

hqueue commented 7 years ago

@gkhanna79 I'm also thinking about adding Tizen to Linux ARM32 and it will be started bottom up, i.e. starting from CoreCLR and CoreFX as we did it for ubuntu 14.04 and 16.04. I think we can follow similar steps. What do you think of it? I can build CoreCLR for Tizen right now and I can add rootfs for Tizen whenever approved.

gkhanna79 commented 7 years ago

@hqueue @jyoungyun I must admit I am not very familiar with the difference between soft and hard fp. Can you please elaborate on them before we go down the path of determining whether they require new RIDs or not?

gkhanna79 commented 7 years ago

adding Tizen to Linux ARM32 and it will be started bottom up

@hqueue In general, the concept sounds fine to me. however, before proceeding ahead with it, I would like to understand the difference between Tizen (built for Linux Arm32) and a regular Linux Arm32 .NET Core - can you share more details on it?

hqueue commented 7 years ago

Can you please elaborate on them before we go down the path of determining whether they require new RIDs or not?

@gkhanna79 GNU defines three floating ABIs for ARM, i.e. soft, softfp and hard. (see -mfloat-abi=name part at https://gcc.gnu.org/onlinedocs/gcc/ARM-Options.html) The major difference between softfp and hard is as follows.

Therefore you can not use two ABIs together, because calling conventions are different.

In dotnet, we use hard and softfp for ARM. (no soft ABI.)

And all binaries in arm rootfs have hard ABI and binaries in arm-softfp rootfs have softfp ABI.

hqueue commented 7 years ago

I would like to understand the difference between Tizen (built for Linux Arm32) and a regular Linux Arm32 .NET Core - can you share more details on it?

@gkhanna79 Basically you can consider Tizen as another distribution of Linux/arm-softfp. The major difference is version of libraries/kernel and composion of packages in each distribution, for example version of libicu is different. The difference is similar to the difference between ubuntu.14.04 and ubuntu.16.04 in Linux Arm32 .NET Core.

In short,

FYI. In CoreCLR, we have total 5 rootfs for Linux/arm (hardfp), i.e. jessie, vivid, trusty, sily and xenial.

I'm trying to explain in short to avoid confusion. If you have any question, let me know :)

gkhanna79 commented 7 years ago

And all binaries in arm rootfs have hard ABI and binaries in arm-softfp rootfs have softfp ABI

Thanks for the explanation @hqueue. I concur that we lazily enable soft-fp if/once the need arises.

gkhanna79 commented 7 years ago

Thanks for the brief explanation of Tizen @hqueue. Aside from the platform binaries/kernel version differences, is Tizen carrying the same set of binaries as .NET Core usually does? That is, when building a Tizen app, do you basically target .NET Core (and thus, have a shared FX based of Microsoft.NETCore.App) OR do you have (or plan to have) a different targeting mechanism (e.g. Samsung.Tizen package) that could carry a different set of binaries?

Also, how do Tizen apps get activated? Do they activate in the same manner as .NET Core apps (using dotnet.exe) or some other way?