search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.27k stars 4.73k forks source link

Enabling Linux ARM32 for .NET Core #2548

Closed hqueue closed 4 years ago

hqueue commented 7 years ago

Host: Ubuntu 14.04 x64 Target: Ubuntu 14.04 or(and) 16.04 ARM, Ubuntu 14.04 ARM Softfp and Tizen

To enable core-setup for Linux ARM32, we are going to follow steps suggested in PR dotnet/core-setup#712.

ARM hardfp

Common

CoreCLR

CoreFX

core-setup For core-setup, we will follow steps below.

For example, we want to build core-setup for Linux/ARM using following command.

$ ./build.sh  --env-vars DISABLE_CROSSGEN=1,TARGETPLATFORM=arm,TARGETRID=ubuntu.14.04-arm,CROSS=1,ROOTFS_DIR=/home/jyoung/git/dotnet/rootfs-coreclr/arm/

ASP.NET

ARM softfp

We will use armel for arm-softfp through out the dotnet.

Common

Debian.8

CoreFX - Part1

CoreCLR

CoreFX - Part2

core-setup For core-setup, we will follow steps below.

./build.sh --skiptests --env-vars DISABLE_CROSSGEN=1,TARGETPLATFORM=armel,TARGETRID=debian.8-armel,CROSS=1,ROOTFS_DIR=/home/jyoung/git/dotnet/rootfs/armel

Tizen 4.0.0

CoreFX - Part1

CoreCLR

CoreFX - Part2

core-setup For core-setup, we will follow steps below.

dotnet-ci

./build.sh -c Release --skiptests --env-vars DISABLE_CROSSGEN=1,TARGETPLATFORM=armel,TARGETRID=tizen.4.0.0-armel,CROSS=1,ROOTFS_DIR=/home/jyoung/git/dotnet/rootfs/armel-tizen

Results

ubuntu.14.04-arm (Latest build is available at https://github.com/dotnet/core-setup#daily-builds) dotnet-ubuntu.14.04-arm.1.2.0-beta-001291-00.tar.gz (Last updated on Jan 19) dotnet-sdk-ubuntu.14.04-arm.1.0.0-preview5-004431.tar.gz ubuntu.16.04-arm (Latest build is available at https://github.com/dotnet/core-setup#daily-builds) dotnet-ubuntu.16.04-arm.1.2.0-beta-001291-00.tar.gz (Last updated on Jan 19) dotnet-sdk-ubuntu.16.04-arm.1.0.0-preview5-004431.tar.gz debian.8-armel dotnet-debian.8-armel.1.2.0-beta-001271-00.tar.gz tizen.4.0.0-armel dotnet-tizen.4.0.0-armel.1.2.0-beta-001273-00.tar.gz

gkhanna79 commented 7 years ago

Basically, @hqueue, I am trying to understand if Tizen is just another distro equivalent for .NET Core or completely new targeting/AppModel experience.

hqueue commented 7 years ago

@gkhanna79 I thinks there are several questions here.

is Tizen carrying the same set of binaries as .NET Core usually does?

For CoreCLR and CoreFX, right. We will use exactly same set of binaries in Linux/arm-softfp, and AFAIK there is no difference between Linux/ARM and Linux/arm-softfp by now.

Tizen itself use RPM packaging system for platform. CoreCLR and CoreFX for Tizen are already being built with Tizen build system (https://build.tizen.org) at Tizen community (https://www.tizen.org). CoreCLR for Tizen: https://build.tizen.org/package/show?package=coreclr&project=Tizen%3AMobile CoreFX for Tizen: https://build.tizen.org/package/show?package=corefx&project=Tizen%3AMobile

That is, when building a Tizen app, do you basically target .NET Core (and thus, have a shared FX based of Microsoft.NETCore.App) OR do you have (or plan to have) a different targeting mechanism (e.g. Samsung.Tizen package) that could carry a different set of binaries? Also, how do Tizen apps get activated? Do they activate in the same manner as .NET Core apps (using dotnet.exe) or some other way?

There are serveral app model in Tizen and .NET Core will be one of them. I think this can be explained by @lemmaa and @idkiller. @lemmaa @idkiller Please look into this inquiries.

I am trying to understand if Tizen is just another distro equivalent for .NET Core or completely new targeting/AppModel experience.

@gkhanna79 I don't think it provide completely new experience. Because Tizen will act as another distro of Linux, at least I think you can consider it as another distro equivalent for .NET Core. @lemmaa @idkiller what do you think of it?

I think we can continue the discussion next year for more detail :)

jyoungyun commented 7 years ago

@gkhanna79 @hqueue @hseok-oh What do you think of naming the arm-softfp as armsf for nupkgs? I think it does not look so pretty to have a - between buildarch name. Please give me good idea. :)

hqueue commented 7 years ago

@jyoungyun How about using armel for our arm softfp ? It seems that armel is used for ARM softfp binaries in Ubuntu and it would be great if we can follow conventions in Linux community.

See https://wiki.ubuntu.com/ToolChain

armel: soft-float ABI / hard-float ABI, defaulting to arm-linux-gnueabi. armhf: hard-float ABI / soft-float ABI, defaulting to arm-linux-gnueabihf.

idkiller commented 7 years ago

@gkhanna79 If target means "base framework library", current Tizen use Microsoft.NETCore.App as a base framework for developing User apps. but we only use it as a "base framework", Tizen Apps does not executed like other dotnet application. Tizen has not used dotnet.exe or another tool like corerun, coreconsole. Tizen has own original launcher and internally it use interface of libcoreclr.so directly. In this model, binary and dlls in coreclr and Microsoft.NETCore.App should pre-installed to device and app should be run like Framework-dependent deployments. actual App package only has dll or exe of app or app's dependency. but these days, we prepare another model like Self-contained deployments. it should use in device that does not has a runtime. As for the Packaging, Tizen apps are not packaged with nupkg. (it use tpk) there are some APIs for Tizen apps (https://git.tizen.org/cgit/?q=csapi) internal build for these APIs in GBS and build for user app in VisualStudio use nupkg but app is not packed to nupkg.

gkhanna79 commented 7 years ago

@jyoungyun I concur with @hqueue on following the convention. Given that we already have an "arm" RID (at https://github.com/dotnet/corefx/blob/master/pkg/Microsoft.NETCore.Platforms/runtime.json), we will use it for hardfp (and synonym for armhf above). For softfp support, armel sounds good to me and something that we should add at https://github.com/dotnet/corefx/blob/master/pkg/Microsoft.NETCore.Platforms/runtime.json.

CC @janvorli

janvorli commented 7 years ago

It makes sense to me.

jyoungyun commented 7 years ago

@hqueue @gkhanna79 @janvorli I also agree with @hqueue opinion! Sounds good to me too. I will post a PR to support armel build on core-setup soon.

hqueue commented 7 years ago

@gkhanna79 @janvorli Cool! We prepared PR for armel in CoreCLR and CoreFX and updated the agenda of this issue releated to armel. We are working on Core-Setup too. Also I have updated Tizen topic in arm-softfp (armel) part. Please also take a look :)

hqueue commented 7 years ago

We have one ubuntu.14.04 rootfs for Linux/arm-softfp now and we can build for ubuntu.14.04 using ARM softfp ABI.

@gkhanna79 I've made a mistake in previous answers. For ARM softfp, we use debian 8 (jessie) rootfs not ubuntu14.04. Therefore rid will be debian.8-armel if we use armel for ARM softfp.

gkhanna79 commented 7 years ago

@hqueue i have cleanedup/reordered the workitems for armel and Tizen.

moozzyk commented 7 years ago

@gkhanna79 https://dotnet.myget.org/feed/aspnetcore-ci-dev/package/nuget/Libuv/1.10.0-preview1-22036 contains arm versions of Libuv for Linux. Let me know once the package has been replicated and I can update https://github.com/dotnet/core-setup/blob/master/pkg/projects/Microsoft.NETCore.App/project.json#L43

gkhanna79 commented 7 years ago

@moozzyk I have uploaded it to myget and should be visible as soon as myget will make it visible in :)

moozzyk commented 7 years ago

@gkhanna79 - updated. I guess you can mark "Build LibUV for Arm32 (Issue aspnet/libuv-build#19)." checked.

gkhanna79 commented 7 years ago

@moozzyk Done!

jyoungyun commented 7 years ago

@gkhanna79 Should we provide sdk to release arm and armel officially? And.. should the sdk tarball support to resolve the NETCoreApp1.1(it will be NETCoreApp2.0) for the official release? Can you tell us what we need to prepare for an official release? At present, the sdk tarball what I made from cli project does not resolve NETCoreApp1.1 due to Unable to resolve dependency 'Microsoft.CodeAnalysis.CSharp' error. So I am wondering if I have to solve this problem for the official release and if I have to fix roslyn to fix it.

Ramon-Balaguer commented 7 years ago

Please @jyoungyun , can you upload your HellowWorld project and the bins? I have: Raspberry Pi3 + ubuntu.16.04-arm + dotnet-ubuntu-arm.1.2.0-beta-001206-00.tar.gz.

Could you please tell me how to compile a netcore project to run on Raspberry? I'm using windows.

Thanks

jyoungyun commented 7 years ago

@scrambler86 Hi, I attached sdk tarball for ubuntu.14.04-arm. Even this file is for ubuntu.14.04-arm but i think it's okay to run on ubuntu.16.04-arm. After installing sdk tarball, you can use dotnet new command for creating a HelloWorld project. I usually build core-setup and cli project in the following order.

  1. Build CoreCLR for ubuntu.14.04-arm
  2. Build CoreFX for ubuntu.14.04-arm
  3. Copy result nupkgs of CoreCLR and CoreFX to specific directory and add below line to NuGet.Config file in core-setup. 
    <add key="Local" value="The speicific directory path" />
  4. Build Core-setup with ./build.sh --skiptests --env-vars DISABLE_CROSSGEN=1,TARGETPLATFORM=arm,TARGETRID=ubuntu.14.04-arm,CROSS=1,ROOTFS_DIR=/home/jyoung/git/dotnet/rootfs/arm command.
  5. Copy results nupkgs of core-setup to specific directory and add upper line to NuGet.Config in cli.
  6. Build cli with ./build.sh --env-vars DOTNET_RUNTIME_ID=ubuntu.14.04-arm -c Release /p:CLIBUILD_SKIP_TESTS=true /p:DISABLE_CROSSGEN=1 command. But to use --env-vars option, cli needs https://github.com/dotnet/cli/pull/5290 PR.

dotnet-sdk-ubuntu.14.04-arm.1.0.0-preview5-004431.tar.gz

Gonkers commented 7 years ago

@jyoungyun Thank you for your help it's much appreciated! @scrambler86 I just found this post as well that could be helpful https://github.com/dotnet/core/blob/master/samples/RaspberryPiInstructions.md

jyoungyun commented 7 years ago

I updated the shared runtime tarballs for ubuntu.14.04 arm and ubuntu.16.04 arm. These tarballs contain libuv.so and CodeAnalysis dlls that were missing from the last tarballs.

And I also uploaded the sdk tarballs for rpi3. You can use it in the following order.

  1. Extract the matching shared runtime and sdk tar.gz to the same folder on your Pi and go into that folder in the terminal. dotnet-ubuntu.14.04-arm.1.2.0-beta-001291-00.tar.gz (shared runtime) dotnet-sdk-ubuntu.14.04-arm.1.0.0-preview5-004431.tar.gz (sdk) dotnet-ubuntu.16.04-arm.1.2.0-beta-001291-00.tar.gz (shared runtime) dotnet-sdk-ubuntu.16.04-arm.1.0.0-preview5-004431.tar.gz (sdk) 
    mkdir dotnet
tar xvzf dotnet-ubuntu.16.04-arm.1.2.0-beta-001291-00.tar.gz -C ./dotnet
tar xvzf dotnet-sdk-ubuntu.16.04-arm.1.0.0-preview5-004431.tar.gz -C ./dotnet
  2. Make a symbolic link from dotnet to /usr/bin/dotnet. ln -sf /usr/bin/dotnet /yourpath/dotnet/dotnet
  3. Create a folder named helloworld and go into it.
  4. Run dotnet new then dotnet restore then dotnet publish to publish your helloworld app. But, in my case, I couldn't success for resolving NETCoreApp1.1 yet even though I have the related nupkg files.
Petermarcu commented 7 years ago

I just pushed some simple instructions to dotnet/core master. They point at some older tarballs. I will adjust it to point to these. Looking forward to when these can be part of the daily build. :)

https://github.com/dotnet/core/blob/master/samples/RaspberryPiInstructions.md

Petermarcu commented 7 years ago

@scrambler86 , in the next couple weeks, we should be able to get all the packages published in order to actually build a standalone app on Windows that that targets arm32 linux. This is what we need to have the developer experience on Windows targeting Raspberry Pi complete. For now, if the app you write doesn't carry the runtime with it and can just run on the shared runtime, then you should be able to publish on Windows and copy the published output to your Pi. Give it a try and let us know.

Petermarcu commented 7 years ago

@jyoungyun, do you have a nuget.config that points to a local folder containing your packages? Without that, I dont think it would know how to find them.

gkhanna79 commented 7 years ago

@jyoungyun All master branches will be updated to resolve .NETCoreApp 2.0. However, before that happens, we need to have the nuget packages published that I am working through since the build-rootfs.sh fails in our official build infrastructure.

JensNordenbro commented 7 years ago

So all this nice work is aiming for 2.0, or is it 1.2 release?

SteveDesmond-ca commented 7 years ago

1.2 is in the process of being renamed 2.0

hqueue commented 7 years ago

However, before that happens, we need to have the nuget packages published that I am working through since the build-rootfs.sh fails in our official build infrastructure.

@gkhanna79 Do you have problem with buiild-rootfs.sh for arm and armel, i.e. ./build-rootfs.sh arm and ./build-rootfs.sh armel? What kind of problem are you suffering ? I think we can also look into the porblem if possible.

Petermarcu commented 7 years ago

It was something about building in Docker which is how we build many of our Linux legs of our build.

jyoungyun commented 7 years ago

@Petermarcu Yes, I updated a NuGet.Config file to point to a local folder containing our nupkgs. But it was still failed to resolve NETCoreApp1.1. The error message is like below.

Retrying 'FindPackagesByIdAsync' for source 'https://api.nuget.org/v3-flatcontainer/system.globalization/index.json'.
  A task was canceled.

I doubt if it is an issue of the corporate network. Have you ever seen such an error?

Petermarcu commented 7 years ago

I haven't. @ericstj any thoughts?

ericstj commented 7 years ago

NuGet will usually fail if it has any feed in its list that can't be resolved. NuGet/dotnet restore will usually echo all the feeds its using as well as all the config sources to the console when you run it. Double check that you aren't using any nuget feeds that are unresolvable in your private network.

JensNordenbro commented 7 years ago

Ok, 2.0 it is. Thanks @stevedesmond-ca. So the the core-setup effort for arm will debute at 2.0 release?

hqueue commented 7 years ago

It was something about building in Docker which is how we build many of our Linux legs of our build.

@Petermarcu @gkhanna79 FYI. Among arm rootfs, rootfs for armel(Tizen) (constructed from build-rootfs.sh armel tizen) has more simpler structure than other rootfs (ubuntu and debian) and requires less effort. For example, armel(Tizen) rootfs just contains only required components for build (about 100MB), but other rootfs, i.e. arm (Ubuntu) and armel (Debian), have all Linux components of default Linux distro (about 700MB) and requires debootstrap operation with sudo previledge during build-rootfs.sh. Therefore requires more resources including network bandwidth, storage and computation. We also experience failure sometime in host x64 Linux box when constructing rootfs for ubuntu-arm and debian-armel due to various reasons including network problem and etc. Therefore if you are experiencing trouble to construct ARM rootfs in docker environment, I would suggest trying simpler Tizen rootfs.

gkhanna79 commented 7 years ago

@hqueue The fact that Docker images are a container and not a virtualized environment created problems - it had nothing to do with the rootfs creation.

The specific issue was that I was able to build LinuxArm32 in Docker image on my dev machine but couldn't do so in our official build. After investigating various options, I took a different approach to compare differences between the success and failing case. The issue is that, even when attempted within a Docker container, the cross build expects (in the second stage invocation that qemu-debootstrap kicks off) that the host machine has the cross build dependencies also installed. Since they were not present in the host OS image in our official build, and only present in the Docker container, cross build within Docker container would fail.

I took one of our official build machines offline yesterday, got the missing packages installed and got a clean Linux Arm32 build in a Docker image finally! :) I will get our official machines updated and will soon get this going.

With the above said, we need to produce .NET Core for both armhf and armel. Thus, debian.8-armel or Tizen armel build will be an additive and not alternative to armhf build.

hqueue commented 7 years ago

@gkhanna79 Glad to hear the progress and thank you for all your efforts :)

gkhanna79 commented 7 years ago

Updating the primary thread - CoreCLR pipeline build is now publishing Arm32 (armhf) packages for Ubuntu 14.04 and Ubuntu 16.04 on myget! Here are the first set of packages that got published today from the repo:

https://dotnet.myget.org/feed/dotnet-core/package/nuget/runtime.ubuntu.14.04-arm.Microsoft.NETCore.Runtime.CoreCLR

https://dotnet.myget.org/feed/dotnet-core/package/nuget/runtime.ubuntu.16.04-arm.Microsoft.NETCore.Runtime.CoreCLR

hqueue commented 7 years ago

@gkhanna79 First of all great thanks for publishing packages for CoreCLR :)

We tested with above packages, but unfortunately, I think it has bug due to clang which is used when building CoreCLR.

Current llvm-3.8 and below can not build CoreCLR and CoreFX correctly with -O3 optimization for ARM, because they have bug related to TLS. You can find details at https://github.com/dotnet/coreclr/blob/master/Documentation/building/linux-instructions.md#how-to-enable--o3-optimization-level-for-armlinux (There are lots of issues related to this. One of them is https://github.com/dotnet/coreclr/issues/6530 )

Therefore I think we have to consider upgrading llvm version for ARM release and we will also discuss this issue with issue dotnet/core-setup#790 (arm32 CI). Let's move to dotnet/core-setup#790 and I will continue discussion there.

BTW Most simple workaround is using "-O1" instead of "-O3" when building CoreCLR with current infra structures.

gkhanna79 commented 7 years ago

Very well @hqueue - I will continue the discussion with you on dotnet/core-setup#790.

SteveDesmond-ca commented 7 years ago

Heads up, in case anyone else wants them, I've got automated builds of the .NET Core Runtime for ubuntu.16.04-arm working now, via a bunch of bash scripts.

https://github.com/stevedesmond-ca/dotnet-arm/releases/

I'm planning on publishing nightlies (including the SDK within the next week) there until official CI builds can be set up.

gkhanna79 commented 7 years ago

Excellent @stevedesmond-ca! I am working on the CoreFX support for official builds right now.

BTW, are your builds for Ubuntu 14.04 or 16.04? And I take it you are consuming the packages we publish for CoreCLR already, right?

SteveDesmond-ca commented 7 years ago

16.04 only right now. I'm building CoreCLR and CoreFX from scratch, then feeding artifacts from those into core-setup, very similar to how @jyoungyun described here.

Once CoreCLR on myget is built with -O1 (currently I get the -O3 TLS segfault from what's pulled down) it should be easier to use official builds. I do think there's value in being able to do the whole thing offline, to decouple the build process from the current CI publish process, but that's just my personal preference.

bjoernbusch commented 7 years ago

@stevedesmond-ca I tried running your built in a docker container on my RPi3 with the following dockerfile:

FROM armv7/armhf-ubuntu:16.04

ENV VERSION 2.0.0-beta-001455-00

RUN apt-get update -q -y && \
apt-get upgrade -q -y && \
apt-get install -q -y wget

RUN wget https://github.com/stevedesmond-ca/dotnet-arm/releases/download/$VERSION/dotnet-ubuntu-arm.$VERSION.tar.gz && \
mkdir -p /usr/share/dotnet && \
tar -xf dotnet-ubuntu-arm.$VERSION.tar.gz -C /usr/share/dotnet/ && \
ln -sf /usr/share/dotnet/dotnet /usr/bin/dotnet

RUN mkdir dotnet
COPY dotnet-arm/hello-world/hello-world.csproj dotnet/hello-world.csproj
COPY dotnet-arm/hello-world/hello-world.deps.json dotnet/hello-world.deps.json
COPY dotnet-arm/hello-world/hello-world.dll dotnet/hello-world.dll
COPY dotnet-arm/hello-world/hello-world.pdb dotnet/hello-world.pdb
COPY dotnet-arm/hello-world/hello-world.runtimeconfig.json dotnet/hello-world.runtimeconfig.json
COPY dotnet-arm/hello-world/Program.cs dotnet/Program.cs

I then start the container with docker run -it core /bin/bash and execute in dotnet folder dotnet hello-world.dll, but then I get an error:

Failed to load /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0-beta-001455-00/libcoreclr.so, error: libunwind.so.8: cannot open shared object file: No such file or directory
Failed to bind to CoreCLR at '/usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0-beta-001455-00/libcoreclr.so'

Any hint on what stupid mistake I'm making?

SteveDesmond-ca commented 7 years ago

@bjoernbusch check out https://github.com/dotnet/core/blob/master/samples/RaspberryPiInstructions.md -- specifically the "prereq packages" part -- chances are those aren't in the base image

bjoernbusch commented 7 years ago

Yes! facepalm that was it, now it's working! Thanks!

jyoungyun commented 7 years ago

@stevedesmond-ca Thank you for sharing your tarball :) It works well on my raspberry PI3 too. Now I'm trying to build core-setup using docker. Does your CI system use docker for releasing, doesn't it? But when I tried to build with docker, I was faced with a little problem that all arguments that match the docker argument were removed by https://github.com/dotnet/core-setup/blob/master/build.sh#L36L37 lines. Have you ever built core-setup for Ubuntu.16.04-arm in docker? My build command line is like below. After parsing on build.sh, some arguments are removed. 

./build.sh --docker ubuntu.16.04 --env-vars "DISABLE_CROSSGEN=1,TARGETPLATFORM=arm,TARGETRID=ubuntu.16.04-arm,CROSS=1,ROOTFS_DIR=/opt/code/cross/rootfs/arm"
->
Argumtens~~~~:--env-vars DISABLE_CROSSGEN=1,TARGETPLATFORM=arm,TARGETRID=-arm,CROSS=1,ROOTFS_DIR=/opt/code/cross/rootfs/arm

I will make a PR soon. If I'm wrong, let me know.

SteveDesmond-ca commented 7 years ago

I'm just doing my builds from a xenial VM, no Docker involved.

ThadHouse commented 7 years ago

Thanks for sharing the debian softfp tarball! I was able to get it working on one of the embedded platforms I normally work with (The NI roboRIO for FRC). Worked fantastically, and even was able to run a fairly complex socket based program with no issues I could find. Going to be so nice not to have to custom compile Mono anymore.

hqueue commented 7 years ago

Worked fantastically, and even was able to run a fairly complex socket based program with no issues I could find

@ThadHouse Glad to hear it helps. And also goot to hear that real complex applications run on dotnet for debian (softfp) in real device :)

jyoungyun commented 7 years ago

@ThadHouse Actually I tested the operation for debian.8-armel tarball on DockerImage by using rpi3. It is really good news that it works well on real device. :)

jyoungyun commented 7 years ago

@gkhanna79 Could you let me know the progress of nupkg uploading for arm in core-setup? AFAIK this should be preceded, CoreFX nupkgs can upload to MyGet Feed. Thank you for your continued support.