Closed bbusacker closed 5 years ago
@bbusacker we don't support Linux arm64 yet, and i don't know the timeline to support it. I made up a user branch which should create the agent package that runnable in linux-arm64. https://github.com/Microsoft/vsts-agent/tree/users/tihuang/linuxarm64 You can follow the instruction here to build the agent package.
@chrisrpatterson FYI,
we've talked about it. marking as enhancement for consideration
@TingluoHuang I need an arm64 build agent so I checked out your branch, but it seems like there isn't a built version .net sdk for arm64:
-----------------------------------------
Ensure Dotnet SDK
-----------------------------------------
Download dotnetsdk into /root/vsts-agent/src/../_dotnetsdk/2.0.3
dotnet-install: Downloading link: https://dotnetcli.azureedge.net/dotnet/Sdk/2.0.3/dotnet-sdk-2.0.3-linux-arm64.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
dotnet-install: Cannot download: https://dotnetcli.azureedge.net/dotnet/Sdk/2.0.3/dotnet-sdk-2.0.3-linux-arm64.tar.gz
dotnet-install: Downloading legacy link: https://dotnetcli.azureedge.net/dotnet/Sdk/2.0.3/dotnet-dev-ubuntu.16.04-arm64.2.0.3.tar.gz
curl: (22) The requested URL returned error: 404 Not Found
dotnet_install: Error: Could not download .NET Core SDK version 2.0.3
Failed: dotnet-install.sh Failed with return code 1
Any pointers on getting this working? Is there an alternate download for the SDK?
@jkleinsc i tweak the script a little bit, try git pull that branch and try run dev.sh layout
again.
@TingluoHuang I'm hitting this issue:
Configuring...
-------------------
A command is running to initially populate your local package cache, to improve restore speed and enable offline access. This command will take up to a minute to complete and will only happen once.
Decompressing 100% 10497 ms
Expanding 100% 20798 ms
Microsoft (R) Build Engine version 15.6.82.30579 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.
2cbcf4bac192d1da6155331a99c128add035972b
Building 2cbcf4bac192d1da6155331a99c128add035972b --- linux-arm64
Microsoft (R) Build Engine version 15.6.82.30579 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.
Restoring packages for /Users/tylerleonhardt/Downloads/vsts-agent/src/Microsoft.VisualStudio.Services.Agent/Microsoft.VisualStudio.Services.Agent.csproj...
Installing Microsoft.IdentityModel.Logging 1.1.5.
Installing Microsoft.IdentityModel.Tokens 5.1.5.
Installing Microsoft.Bcl 1.1.9.
Installing System.IdentityModel.Tokens.Jwt 5.1.5.
Installing Microsoft.Net.Http 2.2.22.
Installing System.ComponentModel.Annotations 4.4.1.
Installing YamlDotNet.Signed 4.2.1.
Installing System.Threading.Tasks.Dataflow 4.8.0.
Installing Microsoft.IdentityModel.Clients.ActiveDirectory 3.17.2.
Installing System.Security.Cryptography.OpenSsl 4.4.0.
Installing System.Security.Cryptography.ProtectedData 4.4.0.
Installing HtmlAgilityPack 1.6.5.
Installing Microsoft.AspNet.WebApi.Client 5.2.3.
Installing System.Configuration.ConfigurationManager 4.4.1.
Installing vss-api-netcore 0.5.61-private.
Generating MSBuild file /Users/tylerleonhardt/Downloads/vsts-agent/src/Microsoft.VisualStudio.Services.Agent/obj/Microsoft.VisualStudio.Services.Agent.csproj.nuget.g.props.
Generating MSBuild file /Users/tylerleonhardt/Downloads/vsts-agent/src/Microsoft.VisualStudio.Services.Agent/obj/Microsoft.VisualStudio.Services.Agent.csproj.nuget.g.targets.
Restore completed in 11.97 sec for /Users/tylerleonhardt/Downloads/vsts-agent/src/Microsoft.VisualStudio.Services.Agent/Microsoft.VisualStudio.Services.Agent.csproj.
/Users/tylerleonhardt/.nuget/packages/microsoft.netcore.app/2.0.0/build/netcoreapp2.0/Microsoft.NETCore.App.targets(19,5): error : Project is targeting runtime 'linux-arm64' but did not resolve any runtime-specific packages for the 'Microsoft.NETCore.App' package. This runtime may not be supported by .NET Core. [/Users/tylerleonhardt/Downloads/vsts-agent/src/Microsoft.VisualStudio.Services.Agent/Microsoft.VisualStudio.Services.Agent.csproj] [/Users/tylerleonhardt/Downloads/vsts-agent/src/dir.proj]
/Users/tylerleonhardt/Downloads/vsts-agent/src/dir.proj(34,5): error MSB3073: The command "dotnet publish Microsoft.VisualStudio.Services.Agent/Microsoft.VisualStudio.Services.Agent.csproj --output "/Users/tylerleonhardt/Downloads/vsts-agent/src/../_layout/bin" --runtime linux-arm64 -c Debug" exited with code 1.
Failed: build
on macOS & Ubuntu.
On Windows it fails before that:
-----------------------------------------
Ensure Dotnet SDK
-----------------------------------------
Download dotnetsdk into /e/projects/vsts-agent/src/../_dotnetsdk/2.0.3
dotnet_install: Error: OS name could not be detected: UName = MINGW64_NT-10.0
Failed: dotnet-install.sh Failed with return code 1
/e/projects/vsts-agent/src
And then on my raspberry pi running Raspbian Stretch:
pi@raspberrypi:~/Desktop/vsts-agent/src $ ./dev.sh l
~/Desktop/vsts-agent/src ~/Desktop/vsts-agent/src
-----------------------------------------
Ensure Dotnet SDK
-----------------------------------------
Download dotnetsdk into /home/pi/Desktop/vsts-agent/src/../_dotnetsdk/2.0.3
dotnet-install: Downloading link: https://dotnetcli.azureedge.net/dotnet/Sdk/2.1.101/dotnet-sdk-2.1.101-linux-arm.tar.gz
curl: (22) The requested URL returned error: 404
dotnet-install: Cannot download: https://dotnetcli.azureedge.net/dotnet/Sdk/2.1.101/dotnet-sdk-2.1.101-linux-arm.tar.gz
dotnet_install: Error: Could not download .NET Core SDK version 2.1.101
Failed: dotnet-install.sh Failed with return code 1
~/Desktop/vsts-agent/src
I published 2 branch for ARM agent for people who want's to try out.
users/tihuang/linuxarm
-> 32 bits ARM @tylerl0706 helped me tested out this one, seems to work.
users/tihuang/linuxarm64
-> 64 bits ARM <- don't use this one since dotnet core only support arm32
It's awesome 🎉
@TingluoHuang I tested arm64 and it fails with an illegal instruction error executing:
"dotnet msbuild"
This is the same issue I had before and seems to be some defect in the dotnet cli itself. Can anyone else confirm or deny this problem on an arm64 system?
@bbusacker are you building on the arm device?
@bbusacker what kind of device you are using? what's the OS?
Exynos-based phone development board. Unbuntu 16.04
Sent from my iPhone
On May 25, 2018, at 8:52 PM, Ting notifications@github.com<mailto:notifications@github.com> wrote:
@bbusackerhttps://github.com/bbusacker what kind of device you are using? what's the OS?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/Microsoft/vsts-agent/issues/1521#issuecomment-392230682, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ADwCYaXfAGtU1hLwT72SMiu-_X53MB6Cks5t2MN8gaJpZM4TfGuQ.
@tylerl0706 are you building on the arm device? Yes
Sent from my iPhone
On May 25, 2018, at 8:36 PM, Tyler James Leonhardt notifications@github.com<mailto:notifications@github.com> wrote:
@bbusackerhttps://github.com/bbusacker are you building on the arm device?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/Microsoft/vsts-agent/issues/1521#issuecomment-392229697, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ADwCYeGHEeWLY8vX-vLmvpsplUM7YEtWks5t2L-HgaJpZM4TfGuQ.
@bbusacker can you share the full output of ./dev.sh layout
Here is the output to stderr:
./dev.sh layout > output.log ./dev.sh: line 77: 19753 Illegal instruction dotnet msbuild /t:layout /p:PackageRuntime=${RUNTIME_ID} /p:BUILDCONFIG=${BUILD_CONFIG} Failed: build
and attached is output.log output.log
@bbusacker try use dotnet sdk under /home/ufs/vsts-agent-users-tihuang-linuxarm64/_dotnetsdk/2.1
run:
dotnet new
dotnet run
I want to make sure whether dotnet core is supported in your device.
@TingluoHuang
dotnet new fails with illegal instruction on ubuntu 16.04 arm64 so it's pretty clear it's a dotnet cli issue. Can you think of any workaround for this issue?
@bbusacker i would suggest you to seek help from dotnet core people, you can log an issue at https://github.com/dotnet/cli
@bbusacker base on https://github.com/dotnet/core/blob/master/release-notes/2.1/2.1-supported-os.md#linux ARM64 is not listed as supported in 2.1, you might want to check with dotnet core guys, since the CLI allows us create a arm64 package. :(
@TingluoHuang I tried on arm64 and I get the following error running dev.sh layout
:
/root/vsts-agent/src/dir.proj(34,5): error MSB3073: The command "dotnet publish
Microsoft.VisualStudio.Services.Agent/Microsoft.VisualStudio.Services.Agent.csproj --output
"/root/vsts-agent/src/../_layout/bin" --runtime linux-arm64 -c Debug" exited with code 1.```
@jkleinsc what's the full log looks like? i would hope dotnet provide some usefully output before it exits.
@TingluoHuang oops... I forgot a line. Also, I tried running as non root, but still same error:
/home/builduser/vsts-agent/_dotnetsdk/2.1/sdk/2.1.400-preview-008942/Roslyn/Microsoft.CSharp.Core.targets(52,5):
error MSB6006: "csc.dll" exited with code 139.
[/home/builduser/vsts-agent/src/Microsoft.VisualStudio.Services.Agent/Microsoft.VisualStudio.Services.Agent.csproj] [/home/builduser/vsts-agent/src/dir.proj]
/home/builduser/vsts-agent/src/dir.proj(34,5): error MSB3073: The command "dotnet publish
Microsoft.VisualStudio.Services.Agent/Microsoft.VisualStudio.Services.Agent.csproj --output
"/home/builduser/vsts-agent/src/../_layout/bin" --runtime linux-arm64 -c Debug" exited with code 1.```
Here is the full log:
Prepend /root/vsts-agent/src/../_dotnetsdk/2.1 to %PATH%
-----------------------------------------
Dotnet SDK Version
-----------------------------------------
2.1.400-preview-008942
-----------------------------------------
Pre-cache external resources for linux platform ...
-----------------------------------------
Download exists: TEE-CLC-14.0.4.zip
Download exists: vso-task-lib.tar.gz
Download exists: node-v6.10.3-linux-arm64.tar.gz
-----------------------------------------
Create layout ...
-----------------------------------------
Microsoft (R) Build Engine version 15.8.60.28450 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.
e3506ce69ece31e213fa9a60e7990a7e98be0d90
Building e3506ce69ece31e213fa9a60e7990a7e98be0d90 --- linux-arm64
Microsoft (R) Build Engine version 15.8.60.28450 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.
Restoring packages for /root/vsts-agent/src/Microsoft.VisualStudio.Services.Agent/Microsoft.VisualStudio.Services.Agent.csproj...
Generating MSBuild file /root/vsts-agent/src/Microsoft.VisualStudio.Services.Agent/obj/Microsoft.VisualStudio.Services.Agent.csproj.nuget.g.props.
Generating MSBuild file /root/vsts-agent/src/Microsoft.VisualStudio.Services.Agent/obj/Microsoft.VisualStudio.Services.Agent.csproj.nuget.g.targets.
Restore completed in 2.87 sec for /root/vsts-agent/src/Microsoft.VisualStudio.Services.Agent/Microsoft.VisualStudio.Services.Agent.csproj.
You are working with a preview version of the .NET Core SDK. You can define the SDK version via a global.json file in the current project. More at https://go.microsoft.com/fwlink/?linkid=869452
/root/vsts-agent/_dotnetsdk/2.1/sdk/2.1.400-preview-008942/Roslyn/Microsoft.CSharp.Core.targets(52,5): error MSB6006: "csc.dll" exited with code 139. [/root/vsts-agent/src/Microsoft.VisualStudio.Services.Agent/Microsoft.VisualStudio.Services.Agent.csproj] [/root/vsts-agent/src/dir.proj]
/root/vsts-agent/src/dir.proj(34,5): error MSB3073: The command "dotnet publish Microsoft.VisualStudio.Services.Agent/Microsoft.VisualStudio.Services.Agent.csproj --output "/root/vsts-agent/src/../_layout/bin" --runtime linux-arm64 -c Debug" exited with code 1.
Failed: build
@jkleinsc i found something similar online https://github.com/dotnet/roslyn/issues/26505 looks like a CSC issue.
@jkleinsc can you also try use dotnet sdk under /root/vsts-agent/_dotnetsdk/2.1 run: dotnet new dotnet run
@jkleinsc Are you building on an arm64 system directly? My build seems to be taking a different path than yours. My path is invoking msbuild to build while yours seems to be invoking publish. Are you just doing ./dev.sh layout on the command-line?
@bbusacker yes building on arm64 directly. And I am running ./dev.sh layout on the command-line.
@TingluoHuang dotnet new and run seem to work. Output here:
root@arm64-ci:~/vsts-agent/_dotnetsdk/2.1# ./dotnet new
Usage: new [options]
Options:
-h, --help Displays help for this command.
-l, --list Lists templates containing the specified name. If no name is specified, lists all templates.
-n, --name The name for the output being created. If no name is specified, the name of the current directory is used.
-o, --output Location to place the generated output.
-i, --install Installs a source or a template pack.
-u, --uninstall Uninstalls a source or a template pack.
--nuget-source Specifies a NuGet source to use during install.
--type Filters templates based on available types. Predefined values are "project", "item" or "other".
--force Forces content to be generated even if it would change existing files.
-lang, --language Filters templates based on language and specifies the language of the template to create.
Templates Short Name Language Tags
----------------------------------------------------------------------------------------------------------------------------
Console Application console [C#], F#, VB Common/Console
Class library classlib [C#], F#, VB Common/Library
Unit Test Project mstest [C#], F#, VB Test/MSTest
xUnit Test Project xunit [C#], F#, VB Test/xUnit
Razor Page page [C#] Web/ASP.NET
MVC ViewImports viewimports [C#] Web/ASP.NET
MVC ViewStart viewstart [C#] Web/ASP.NET
ASP.NET Core Empty web [C#], F# Web/Empty
ASP.NET Core Web App (Model-View-Controller) mvc [C#], F# Web/MVC
ASP.NET Core Web App razor [C#] Web/MVC/Razor Pages
ASP.NET Core with Angular angular [C#] Web/MVC/SPA
ASP.NET Core with React.js react [C#] Web/MVC/SPA
ASP.NET Core with React.js and Redux reactredux [C#] Web/MVC/SPA
Razor Class Library razorclasslib [C#] Web/Razor/Library/Razor Class Library
ASP.NET Core Web API webapi [C#], F# Web/WebAPI
global.json file globaljson Config
NuGet Config nugetconfig Config
Web Config webconfig Config
Solution File sln Solution
Examples:
dotnet new mvc --auth Individual
dotnet new xunit
dotnet new --help
and:
./dotnet run
Couldn't find a project to run. Ensure a project exists in /root/vsts-agent/_dotnetsdk/2.1, or pass the path to the project using --project.
@jkleinsc do ./dotnet new console then do ./dotnet run.
Also could you post the output from ./dotnet --info. I'd like to compare it to what I have.
@bbusacker here you go:
root@arm64-ci:~/vsts-agent/_dotnetsdk/2.1# ./dotnet new console
The template "Console Application" was created successfully.
Processing post-creation actions...
Running 'dotnet restore' on /root/vsts-agent/_dotnetsdk/2.1/2.1.csproj...
Restoring packages for /root/vsts-agent/_dotnetsdk/2.1/2.1.csproj...
/root/vsts-agent/_dotnetsdk/2.1/2.1.csproj : error NU1102: Unable to find package Microsoft.NETCore.App with version (>= 2.1.0-rtm-26515-03)
/root/vsts-agent/_dotnetsdk/2.1/2.1.csproj : error NU1102: - Found 32 version(s) in nuget.org [ Nearest version: 2.1.0-rc1 ]
Generating MSBuild file /root/vsts-agent/_dotnetsdk/2.1/obj/2.1.csproj.nuget.g.props.
Generating MSBuild file /root/vsts-agent/_dotnetsdk/2.1/obj/2.1.csproj.nuget.g.targets.
Restore failed in 1.87 sec for /root/vsts-agent/_dotnetsdk/2.1/2.1.csproj.
Restore failed.
Post action failed.
Description: Restore NuGet packages required by this project.
Manual instructions: Run 'dotnet restore'
root@arm64-ci:~/vsts-agent/_dotnetsdk/2.1# ./dotnet run
/root/vsts-agent/_dotnetsdk/2.1/2.1.csproj : error NU1102: Unable to find package Microsoft.NETCore.App with version (>= 2.1.0-rtm-26515-03)
/root/vsts-agent/_dotnetsdk/2.1/2.1.csproj : error NU1102: - Found 32 version(s) in nuget.org [ Nearest version: 2.1.0-rc1 ]
The build failed. Please fix the build errors and run again.
root@arm64-ci:~/vsts-agent/_dotnetsdk/2.1# ./dotnet --info
.NET Core SDK (reflecting any global.json):
Version: 2.1.400-preview-008942
Commit: de407deff0
Runtime Environment:
OS Name: ubuntu
OS Version: 16.04
OS Platform: Linux
RID: ubuntu.16.04-arm64
Base Path: /root/vsts-agent/_dotnetsdk/2.1/sdk/2.1.400-preview-008942/
Host (useful for support):
Version: 2.1.0-rtm-26515-03
Commit: caa7b7e2ba
.NET Core SDKs installed:
2.1.400-preview-008942 [/root/vsts-agent/_dotnetsdk/2.1/sdk]
.NET Core runtimes installed:
Microsoft.NETCore.App 2.1.0-rtm-26515-03 [/root/vsts-agent/_dotnetsdk/2.1/shared/Microsoft.NETCore.App]
To install additional .NET Core runtimes or SDKs:
https://aka.ms/dotnet-download
FWIW the arm64 hardware I am trying to build on is a Hisilicon D05.
@TingluoHuang I am now trying the arm build instead of the arm64 one and I am running into this error:
Couldn't find a valid ICU package installed on the system. Set the configuration flag System.Globalization.Invariant to true if you want to run with no globalization support.
at System.Environment.FailFast(System.String)
at System.Globalization.GlobalizationMode.GetGlobalizationInvariantMode()
at System.Globalization.GlobalizationMode..cctor()
at System.Globalization.CultureData.CreateCultureWithInvariantData()
at System.Globalization.CultureData.get_Invariant()
at System.Globalization.CultureInfo..cctor()
at System.StringComparer..cctor()
at System.AppDomain.InitializeCompatibilityFlags()
at System.AppDomain.CreateAppDomainManager()
at System.AppDomain.Setup(System.Object)
./dev.sh: line 178: 32203 Aborted dotnet --version
That I can tell there aren't icu packages that will work on arm, so do you know what I need to do to get past this error? I see directions here: https://github.com/dotnet/core/blob/master/Documentation/build-and-install-rhel6-prerequisites.md to use a file called "*.runtimeconfig.json", but I'm not sure where that file is and/or where it should exist.
@jkleinsc you can export CORECLR_GLOBAL_INVARIANT=1 as well
@TingluoHuang that doesn't help. When I run ./dotnet --version I see this error:
root@arm64-ci:~/vsts-agent-arm/_dotnetsdk/2.1# echo $CORECLR_GLOBAL_INVARIANT
1
root@arm64-ci:~/vsts-agent-arm/_dotnetsdk/2.1# ./dotnet --version
FailFast:
Couldn't find a valid ICU package installed on the system. Set the configuration flag System.Globalization.Invariant to true if you want to run with no globalization support.
at System.Environment.FailFast(System.String)
at System.Globalization.GlobalizationMode.GetGlobalizationInvariantMode()
at System.Globalization.GlobalizationMode..cctor()
at System.Globalization.CultureData.CreateCultureWithInvariantData()
at System.Globalization.CultureData.get_Invariant()
at System.Globalization.CultureInfo..cctor()
at System.StringComparer..cctor()
at System.AppDomain.InitializeCompatibilityFlags()
at System.AppDomain.CreateAppDomainManager()
at System.AppDomain.Setup(System.Object)
Aborted
@jkleinsc what about "_dotnetsdk\2.0.3\sdk\2.0.3\dotnet.runtimeconfig.json" ?
@TingluoHuang that worked. It looks like dev.sh layout
and dev.sh build
worked! Dumb question -- where do I find the executable(s) from the build?
it's under _layout
dir under agent root directory. ~/vsts-agent-arm/_layout/config.sh
Is there anything we can do to move this forward? It appears that this is a blocker for Electron development on arm64.
https://github.com/dotnet/core/blob/master/release-notes/2.1/2.1-supported-os.md#linux dotnet core haven't support ARM64 yet, but i think you can run the 32 bit app on ARM64 device.
@TingluoHuang There are some arm64 devices that also support running 32-bit codes; that's pretty common in the embedded and single-board computer world. For data center applications, though, the trend has been for silicon makers to strip out 32-bit support and go 64-bit only. See e.g. systems from Cavium (ThunderX, ThunderX2) and Qualcomm (Centriq) which are both 64-bit only.
@vielmetti understand, but that problem is not scope to this repo, we rely on dotnet core on OS support. there are lot of issues at dotnet repo around ARM64, https://github.com/dotnet/core/search?q=arm64&type=Issues From what i know, the fully ARM64 support work in dotnet core haven't finished yet.
Since this issue is about arm64, should there also be a issue for arm32?
The documentation is really lacking that vsts-agent is only support for linux x86-64 and nothing else. Now it says it supports several linux distributions. And because .Net Core 2.1 support arm32, I would expect that VSTS also supported deploying to it.
@NiekGr - part of supporting arm is moving to net core 2.1. Its currently on 2.0.
@bryanmacfarlane thank you
We add ARM32 support in 2.141.0, here is the final build we are about to release. https://github.com/Microsoft/azure-pipelines-agent/releases/tag/v2.141.0
Thanks for getting this done @TingluoHuang! I am really looking forward to using it!
@TingluoHuang refers to arm32 support, which is great, and the issue refers to arm64 support, which is different.
@vielmetti dotnet core haven't support ARM64 yet, we will produce ARM64 agent once dotnet core support them, i will update the issue title.
@vielmetti btw the arm client works fine on the Hisilicon D05 "Type 2A2" hardware. And I think I can build arm64 builds with in.
@jkleinsc There are some Arm SoCs that are server-class and also 32-bit capable (that Hisilicon Hi1616). Many (most) of the server-class chips are 64-bit only (Cavium, Qualcomm, Ampere).
https://github.com/dotnet/announcements/issues/82 we will add ARM64 after dotnet core 3.0 release.
Arm64 support is now available at the tip of the .Net Core libraries and I'm seeing some of its support trickling into vsts-agent. Do you know if building vsts-agent has been tested yet on arm64 systems running ubuntu and whether or not it should work? If not, do you know if you plan on supporting this platform in the future? And if you are when that would be. If not, could you give me some pointers on what it would take to get the library building? I can get the .NET core library downloaded an installed for the build but I'm not sure if there are a bunch of nuget packages that are also needed that would need to also be rebuilt for arm64. Just let me know. I have an arm64 system running Ubuntu 16.04 that I would like to run the build agent on.