search

dotnet / source-build

A repository to track efforts to produce a source tarball of the .NET Core SDK and all its components
MIT License
265 stars 130 forks source link

Building the .NET Core SDK on FreeBSD #1139

Closed joperator closed 7 months ago

joperator commented 5 years ago

I'm trying to build the .NET Core SDK on FreeBSD but I'm having trouble with the documentation. The documentation says I just have to initialize the submodules in the source-build repo and run ./build.sh. But this produces only the following output:

Unsupported OS 'FreeBSD' detected. Downloading linux-x64 tools.
Installing dotnet cli...
Installing dotnet 3.0 preview1 cli...
Restoring BuildTools version 3.0.0-preview4-04104-01...

Later in the documentation it says that

git submodule update --init --recursive [...] will initialize the submodules and clone the repos under the src folder if they haven't been initialized.

But after running this command, my src folder contains only the subfolders netcorecli-fsc and reference-assemblies, which are not all required repos I think. As the documentation says, I expected at least core-setup, standard, corefx and coreclr to be cloned into the src folder. This would also explain why there is no bin folder in my source-build repo after running ./build.sh as described in the documentation. If the scripts don't work under FreeBSD, I expect at least an error, otherwise I assume that everything has been built, which is not the case here.

So, am I doing something wrong or is the documentation incorrect or out of date?

wfurt commented 5 years ago

Issues with LibGit2Sharp should be fixed by https://github.com/dotnet/sourcelink/pull/288 (it may not make it to 3.0 branches)

joperator commented 5 years ago

... I added link to download updated cli if somebody wants to skip all the steps for now.

This sounds great @wfurt. But actually I think I found some bugs when trying to build a .NET Core app targeting netcoreapp3.0 with your updated CLI:

Microsoft (R) Build Engine version 16.3.0-preview-19325-02+eca7818b1 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

~/ConsoleAppCore.csproj : error NU1102: Unable to find package Microsoft.NETCore.App.Ref with version (= 3.0.0-preview7-27826-04)
~/ConsoleAppCore.csproj : error NU1102:   - Found 6 version(s) in nuget.org [ Nearest version: 3.0.0-preview7-27912-14 ]
~/ConsoleAppCore.csproj : error NU1102: Unable to find package Microsoft.NETCore.App.Host.linux-x64 with version (= 3.0.0-preview7-27826-04)
~/ConsoleAppCore.csproj : error NU1102:   - Found 4 version(s) in nuget.org [ Nearest version: 3.0.0-preview7-27912-14 ]
~/ConsoleAppCore.csproj : error NU1102: Unable to find package Microsoft.AspNetCore.App.Ref with version (= 3.0.0-preview7.19325.7)
~/ConsoleAppCore.csproj : error NU1102:   - Found 6 version(s) in nuget.org [ Nearest version: 3.0.0-preview7.19365.7 ]
  Restore failed in 728.67 ms for ~/ConsoleAppCore.csproj.

Build FAILED.

~/ConsoleAppCore.csproj : error NU1102: Unable to find package Microsoft.NETCore.App.Ref with version (= 3.0.0-preview7-27826-04)
~/ConsoleAppCore.csproj : error NU1102:   - Found 6 version(s) in nuget.org [ Nearest version: 3.0.0-preview7-27912-14 ]
~/ConsoleAppCore.csproj : error NU1102: Unable to find package Microsoft.NETCore.App.Host.linux-x64 with version (= 3.0.0-preview7-27826-04)
~/ConsoleAppCore.csproj : error NU1102:   - Found 4 version(s) in nuget.org [ Nearest version: 3.0.0-preview7-27912-14 ]
~/ConsoleAppCore.csproj : error NU1102: Unable to find package Microsoft.AspNetCore.App.Ref with version (= 3.0.0-preview7.19325.7)
~/ConsoleAppCore.csproj : error NU1102:   - Found 6 version(s) in nuget.org [ Nearest version: 3.0.0-preview7.19365.7 ]
    0 Warning(s)
    3 Error(s)

Replacing all occurrences of not found versions with the specified nearest versions in your updated CLI seems to fix these bugs and let me build (and run) the .NET core app again.

wfurt commented 5 years ago

oh, I probably forget to mention this in note. We publish to Nuget only certain version and rest goes to transient blob storage. I have this in my NuGet config

   <add key="dotnet-core" value="https://dotnetfeed.blob.core.windows.net/dotnet-core/index.json" />
   <add key="aspnet-core" value="https://dotnetfeed.blob.core.windows.net/aspnet-aspnetcore/index.json" />

Or you would need to pass it in as extra restore sources. That should not be problem once we sync up to official releases.

For CoreCLR I use -skipmanagedtools so it does not need ASP.NET at all. But it may be needed in some other places.

Also note that on current platforms, we bundle ASP in SDK but we do not on FreeBSD as there was no runtime as well as it was hard to build ASP without stable cli. There may be still some odd ends.

daydrim commented 5 years ago

Glad to hear about the changes... So even with that keys there are errors with .build.sh It looks like we need to disable .netcoreapp22 OS: FreeBSD 11.3 p3

Making all .sh files executable under Tools.
  Restoring packages for /usr/home/daydrim/source-build/Tools/crossgen/crossgen.csproj...
/usr/home/daydrim/source-build/Tools/crossgen/crossgen.csproj : error NU1202: Package Microsoft.NETCore.App 3.0.0-preview-27218-01 is not compatible with netcoreapp2.2 (.NETCoreApp,Version=v2.2). Package Microsoft.NETCore.App 3.0.0-preview-27218-01 supports: netcoreapp3.0 (.NETCoreApp,Version=v3.0)
/usr/home/daydrim/source-build/Tools/crossgen/crossgen.csproj : error NU1202: Package Microsoft.NETCore.App 3.0.0-preview-27218-01 is not compatible with netcoreapp2.2 (.NETCoreApp,Version=v2.2) / linux-x64. Package Microsoft.NETCore.App 3.0.0-preview-27218-01 supports: netcoreapp3.0 (.NETCoreApp,Version=v3.0)
  Generating MSBuild file /usr/home/daydrim/source-build/Tools/crossgen/obj/crossgen.csproj.nuget.g.props.
  Generating MSBuild file /usr/home/daydrim/source-build/Tools/crossgen/obj/crossgen.csproj.nuget.g.targets.
  Restore failed in 438.31 ms for /usr/home/daydrim/source-build/Tools/crossgen/crossgen.csproj.
joperator commented 5 years ago

@daydrim: Did you notice the following steps from the instructions?

and apply following patch (to get https://github.com/dotnet/source-build/pull/631)

Set variables and assemble SKD without crossgen. (set DropSuffix=true to strip preview6 from version). export DISABLE_CROSSGEN=true

daydrim commented 5 years ago

Did you notice the following steps from the instructions?

Yeap. I've tried export DISABLE_CROSSGEN="true" and change core-setup.proj <BuildArguments Condition="'$(TargetOS)' == 'FreeBSD'">$(BuildArguments) -DisableCrossgen=true</BuildArguments>

The result of build.sh from source-build is the same

Making all .sh files executable under Tools.
  Restoring packages for /usr/home/daydrim/source-build/Tools/crossgen/crossgen.csproj...
/usr/home/daydrim/source-build/Tools/crossgen/crossgen.csproj : error NU1202: Package Microsoft.NETCore.App 3.0.0-preview-27218-01 is not compatible with netcoreapp2.2 (.NETCoreApp,Version=v2.2). Package Microsoft.NETCore.App 3.0.0-preview-27218-01 supports: netcoreapp3.0 (.NETCoreApp,Version=v3.0)
/usr/home/daydrim/source-build/Tools/crossgen/crossgen.csproj : error NU1202: Package Microsoft.NETCore.App 3.0.0-preview-27218-01 is not compatible with netcoreapp2.2 (.NETCoreApp,Version=v2.2) / linux-x64. Package Microsoft.NETCore.App 3.0.0-preview-27218-01 supports: netcoreapp3.0 (.NETCoreApp,Version=v3.0)
  Generating MSBuild file /usr/home/daydrim/source-build/Tools/crossgen/obj/crossgen.csproj.nuget.g.props.
  Generating MSBuild file /usr/home/daydrim/source-build/Tools/crossgen/obj/crossgen.csproj.nuget.g.targets.
  Restore failed in 476.24 ms for /usr/home/daydrim/source-build/Tools/crossgen/crossgen.csproj.
joperator commented 5 years ago

That's weird @daydrim because I do almost the same things there as you do. The only difference I see at the moment is that I'm building in a FreeBSD 12.0 VM while you're building on FreeBSD 11.3 - but I don't think this is the cause of the problem. Maybe @wfurt knows more.

I wrote some patches to make source-build work as much as possible using your updated .NET Core SDK for FreeBSD as bootstrap CLI, as you recommended @wfurt: source-build-freebsd-patches.zip

With different command line switches I now get the following results when running ./build.sh:

crummel commented 5 years ago

If you haven't yet, you might want to rebase your work on the latest release/3.0. We started building the runtime as portable - this is the only way SkipRepoReferences=true will work since you won't be building the non-portable assets that core-setup and corefx expect. bin/logs/corefx.log and bin/logs/core-setup.log should have more details, or there should be even more detail in bin/src/<repo>.<hash>/artifacts/log.

wfurt commented 5 years ago

I made some progress over the weekend but I did not have time to update this issue. I'm building on 11.3 and I can build whole runtime (up to core-setup) if I skip Roslyn as @crummel suggested.

As was planning to post PR with some improvements. I will review and incorporate posted patches as I see fit @joperator

As far as the Microsoft.AspNetCore.App.Ref error: remove reference to it in Microsoft.NETCoreSdk.BundledVersions.props. As we get updated SDK from Linux it assumes it is bundled in but it is not on FreeBSD. I will post updated cli but I wanted to do more testing before that.

As far as the the 2.x vs 3.0 targeting: there may be some tricks we can play. I'm still trying to sort this out and talking few experts around. We may just fake old version to keep build happy for now.

joperator commented 5 years ago

I'm trying to run ./build.sh /p:RootRepo=core-setup /p:SkipRepoReferences=true on FreeBSD 12.0 without Roslyn using the release/3.0 branch, but that doesn't seem to work:

Found 2.x SDK 3.0.100-preview7-012630, 3.x SDK 3.0.100-preview7-012821, Arcade bootstrap 1.0.0-beta.19359.6
warning: submodule tools-local/arcade-services has uncommitted changes

Tools are already initialized
Microsoft (R) Build Engine version 16.3.0-preview-19325-02+eca7818b1 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

/root/git/dotnet/source-build/Tools/dotnetcli/sdk/3.0.100-preview7-012630/MSBuild.dll /bl:build.binlog /clp:v=m /flp:v=diag /p:RootRepo=core-setup /p:SkipRepoReferences=true /root/git/dotnet/source-build/build.proj
  core-setup using package version properties saved at /root/git/dotnet/source-build/bin/obj/x64/Release/PackageVersions.props.pre.core-setup.xml 
  Checked cache for conflicts with source-built nupkgs.
  Building 'core-setup'
  Running command:
    /root/git/dotnet/source-build/bin/src/core-setup.d01b2fb7bc6bd4911e157fbd51353059a3ba1a6c//build.sh --restore --build --configuration Release --ci /p:DisableCrossgen=true /p:BuildDebPackage=false /p:BuildAllPackages=true /p:RestoreAllBuildRids=false /p:PreReleaseLabel="preview8" /p:DotNetOutputBlobFeedDir=/root/git/dotnet/source-build/bin/obj/x64/Release/blob-feed/ --verbosity minimal --binaryLog  /p:DotNetPackageVersionPropsPath=/root/git/dotnet/source-build/bin/obj/x64/Release/PackageVersions.props /p:DotNetRestoreSourcePropsPath=/root/git/dotnet/source-build/bin/obj/x64/Release/RestoreSources.props
    Log: /root/git/dotnet/source-build/bin/logs/core-setup.log
    With Environment Variables:
      DOTNET_TOOL_DIR=/root/git/dotnet/source-build/Tools/dotnetcli3/
      DotNetToolPath=/root/git/dotnet/source-build/Tools/dotnetcli3/
      DotNetTool=/root/git/dotnet/source-build/Tools/dotnetcli3/dotnet
      BUILD_TOOLS_TOOL_DIR=/root/git/dotnet/source-build/Tools/
      BUILDTOOLS_SKIP_CROSSGEN=1
      DotNetBuildFromSource=true
      DotNetRestorePackagesPath=/root/git/dotnet/source-build/packages/
      DotNetCoreSdkDir=/root/git/dotnet/source-build/Tools/dotnetcli3/
      DotNetBuildToolsDir=/root/git/dotnet/source-build/Tools/source-built
      DotNetRoot=/root/git/dotnet/source-build/Tools/dotnetcli3/
      DOTNET_PATH=/root/git/dotnet/source-build/Tools/dotnetcli3/
      DOTNET_HOST_PATH=/root/git/dotnet/source-build/Tools/dotnetcli3/dotnet
      _InitializeDotNetCli=/root/git/dotnet/source-build/Tools/dotnetcli3/
      _DotNetInstallDir=/root/git/dotnet/source-build/Tools/dotnetcli3/
      _InitializeToolset=/root/git/dotnet/source-build/Tools/source-built/Microsoft.DotNet.Arcade.Sdk/tools/Build.proj
      BuildToolsPackageDir=/root/git/dotnet/source-build/Tools/source-built
      StabilizePackageVersion=false
      PB_IsStable=false
      DotNetUseShippingVersions=true
      ContinuousIntegrationBuild=true
      MSBUILDDEBUGPATH=/root/git/dotnet/source-build/bin/msbuild-debug/
      MSBUILDDEBUGCOMM=1
      MSBUILDDEBUGSCHEDULER=1
      MSBUILDDEBUGFORCECACHING=1
      MSBUILDDEBUG=1
      MSBUILDDEBUGEVALUATION=1
      MSBUILDTARGETOUTPUTLOGGING=1
      MSBUILDLOGTASKINPUTS=1
      MSBUILDEMITSOLUTION=1
      MSBUILDLOGVERBOSERARSEARCHRESULTS=1
      RoslynCommandLineLogFile=/root/git/dotnet/source-build/bin/roslyn-debug/
      MSBUILDDISABLENODEREUSE=1
      OfficialBuildId=20280805.7
      BUILD_BUILDNUMBER=20280805.7
      GitCommitCount=6217
      GitCommitHash=d01b2fb7bc6bd4911e157fbd51353059a3ba1a6c
      GitInfoCommitHash=d01b2fb7bc6bd4911e157fbd51353059a3ba1a6c
      SourceRevisionId=d01b2fb7bc6bd4911e157fbd51353059a3ba1a6c
      RepositoryCommit=d01b2fb7bc6bd4911e157fbd51353059a3ba1a6c
      COMMIT_SHA=d01b2fb7bc6bd4911e157fbd51353059a3ba1a6c
      GIT_COMMIT=d01b2fb7bc6bd4911e157fbd51353059a3ba1a6c
      PreReleaseVersionLabel=preview8
      PackageVersionStamp=preview8
      PB_VersionStamp=preview8
      RepositoryType=Git
      DeterministicSourcePaths=true
      SOURCE_BUILT_SDK_ID_ARCADE=Microsoft.DotNet.Arcade.Sdk
      SOURCE_BUILT_SDK_VERSION_ARCADE=1.0.0-beta.19381.1
      SOURCE_BUILT_SDK_DIR_ARCADE=/root/git/dotnet/source-build/Tools/source-built/Microsoft.DotNet.Arcade.Sdk/
/root/git/dotnet/source-build/repos/dir.targets(234,5): error MSB3073: The command "/root/git/dotnet/source-build/bin/src/core-setup.d01b2fb7bc6bd4911e157fbd51353059a3ba1a6c//build.sh --restore --build --configuration Release --ci /p:DisableCrossgen=true /p:BuildDebPackage=false /p:BuildAllPackages=true /p:RestoreAllBuildRids=false /p:PreReleaseLabel="preview8" /p:DotNetOutputBlobFeedDir=/root/git/dotnet/source-build/bin/obj/x64/Release/blob-feed/ --verbosity minimal --binaryLog  /p:DotNetPackageVersionPropsPath=/root/git/dotnet/source-build/bin/obj/x64/Release/PackageVersions.props /p:DotNetRestoreSourcePropsPath=/root/git/dotnet/source-build/bin/obj/x64/Release/RestoreSources.props >> /root/git/dotnet/source-build/bin/logs/core-setup.log 2>&1" exited with code 1. [/root/git/dotnet/source-build/repos/core-setup.proj]
  MSBUILD : error MSB1009: Project file does not exist.
  Switch: /root/git/dotnet/source-build/Tools/source-built/Microsoft.DotNet.Arcade.Sdk/tools/Build.proj
  ##vso[task.logissue type=error]Build failed (exit code '1').

  'core-setup' failed during build.
  See '/root/git/dotnet/source-build/bin/logs/core-setup.log' for more information.

Contrary to what is claimed, core-setup.log does not provide more information. It only contains the same message about the missing project file. bin/src/core-setup.<hash>/artifacts/log contains only an empty folder named Release. As the missing project file is supposed to be inside the Tools folder, I assume that it should be created while running init-tools.sh. But the folder Tools/source-built/Microsoft.DotNet.Arcade.Sdk does not exist after that. Which command or script should create this folder and the project file? The same problem also applies to ./build.sh /p:RootRepo=corefx /p:SkipRepoReferences=true.

wfurt commented 5 years ago

You can try to add -x to core-setup's build.sh to make it more verbose. You can also try to execute it directly to get more feedback. For now, I'm focusing on 11.3. I did not find enough time to cleanup my changes and turn them to PR.

crummel commented 5 years ago

Ah, you're not getting Arcade. Could you try ./build.sh /p:RootRepo=arcade /p:SkipRepoReferences=true first, then the core-setup version?

joperator commented 5 years ago

Thanks for your advice @crummel. After building arcade first, building core-setup continues, but now I get the following error for several projects:

"/root/git/dotnet/source-build/Tools/source-built/Microsoft.DotNet.Arcade.Sdk/tools/Build.proj" (default target) (1) ->
"/root/git/dotnet/source-build/bin/src/core-setup.d01b2fb7bc6bd4911e157fbd51353059a3ba1a6c/src/managed/Microsoft.NET.HostModel/Microsoft.NET.HostModel.csproj" (Build target) (9:7) ->
(InitializeSourceControlInformationFromSourceControlManager target) -> 
  /root/git/dotnet/source-build/packages/microsoft.build.tasks.git/1.0.0-beta2-18618-05/build/Microsoft.Build.Tasks.Git.targets(20,5): error : The type initializer for 'LibGit2Sharp.Core.NativeMethods' threw an exception. [/root/git/dotnet/source-build/bin/src/core-setup.d01b2fb7bc6bd4911e157fbd51353059a3ba1a6c/src/managed/Microsoft.NET.HostModel/Microsoft.NET.HostModel.csproj]
/root/git/dotnet/source-build/packages/microsoft.build.tasks.git/1.0.0-beta2-18618-05/build/Microsoft.Build.Tasks.Git.targets(20,5): error :    at LibGit2Sharp.Core.NativeMethods.git_buf_free(GitBuf buf) [/root/git/dotnet/source-build/bin/src/core-setup.d01b2fb7bc6bd4911e157fbd51353059a3ba1a6c/src/managed/Microsoft.NET.HostModel/Microsoft.NET.HostModel.csproj]
/root/git/dotnet/source-build/packages/microsoft.build.tasks.git/1.0.0-beta2-18618-05/build/Microsoft.Build.Tasks.Git.targets(20,5): error :    at LibGit2Sharp.Core.Proxy.git_buf_free(GitBuf buf) [/root/git/dotnet/source-build/bin/src/core-setup.d01b2fb7bc6bd4911e157fbd51353059a3ba1a6c/src/managed/Microsoft.NET.HostModel/Microsoft.NET.HostModel.csproj]
/root/git/dotnet/source-build/packages/microsoft.build.tasks.git/1.0.0-beta2-18618-05/build/Microsoft.Build.Tasks.Git.targets(20,5): error :    at LibGit2Sharp.Core.Handles.GitBuf.Dispose() [/root/git/dotnet/source-build/bin/src/core-setup.d01b2fb7bc6bd4911e157fbd51353059a3ba1a6c/src/managed/Microsoft.NET.HostModel/Microsoft.NET.HostModel.csproj]
/root/git/dotnet/source-build/packages/microsoft.build.tasks.git/1.0.0-beta2-18618-05/build/Microsoft.Build.Tasks.Git.targets(20,5): error :    at LibGit2Sharp.Core.Proxy.ConvertPath(Func`2 pathRetriever) [/root/git/dotnet/source-build/bin/src/core-setup.d01b2fb7bc6bd4911e157fbd51353059a3ba1a6c/src/managed/Microsoft.NET.HostModel/Microsoft.NET.HostModel.csproj]
/root/git/dotnet/source-build/packages/microsoft.build.tasks.git/1.0.0-beta2-18618-05/build/Microsoft.Build.Tasks.Git.targets(20,5): error :    at LibGit2Sharp.Core.Proxy.git_repository_discover(FilePath start_path) [/root/git/dotnet/source-build/bin/src/core-setup.d01b2fb7bc6bd4911e157fbd51353059a3ba1a6c/src/managed/Microsoft.NET.HostModel/Microsoft.NET.HostModel.csproj]
/root/git/dotnet/source-build/packages/microsoft.build.tasks.git/1.0.0-beta2-18618-05/build/Microsoft.Build.Tasks.Git.targets(20,5): error :    at LibGit2Sharp.Repository.Discover(String startingPath) [/root/git/dotnet/source-build/bin/src/core-setup.d01b2fb7bc6bd4911e157fbd51353059a3ba1a6c/src/managed/Microsoft.NET.HostModel/Microsoft.NET.HostModel.csproj]
/root/git/dotnet/source-build/packages/microsoft.build.tasks.git/1.0.0-beta2-18618-05/build/Microsoft.Build.Tasks.Git.targets(20,5): error :    at Microsoft.Build.Tasks.Git.GitOperations.LocateRepository(String directory) in /_/src/Microsoft.Build.Tasks.Git.Operations/GitOperations.cs:line 26 [/root/git/dotnet/source-build/bin/src/core-setup.d01b2fb7bc6bd4911e157fbd51353059a3ba1a6c/src/managed/Microsoft.NET.HostModel/Microsoft.NET.HostModel.csproj]
/root/git/dotnet/source-build/packages/microsoft.build.tasks.git/1.0.0-beta2-18618-05/build/Microsoft.Build.Tasks.Git.targets(20,5): error :    at Microsoft.Build.Tasks.Git.RepositoryTasks.LocateRepository(LocateRepository task) in /_/src/Microsoft.Build.Tasks.Git.Operations/RepositoryTasks.cs:line 58 [/root/git/dotnet/source-build/bin/src/core-setup.d01b2fb7bc6bd4911e157fbd51353059a3ba1a6c/src/managed/Microsoft.NET.HostModel/Microsoft.NET.HostModel.csproj]

I've fixed a similar error before by installing libgit2 and creating a symlink for /usr/local/lib/libgit2.so, but now I have no idea what is causing the current exception.

crummel commented 5 years ago

Hmm, sourcelink isn't supposed to have a dependency on libgit anymore. I'll look into this after we finish up preview9, we might just have to update our sourcelink hash.

FeodorFitsner commented 4 years ago

Do you guys keep any records (like summary of the conversation above) with the progress of building SDK 3.0 RTM? Thank you for your efforts!

crummel commented 4 years ago

Hmm, not really, the closest thing is probably the commit history and comments on that PR. If you have specific questions I'm probably the one that can answer them.

Also, @joperator, I'll be updating sourcelink in https://github.com/dotnet/source-build/pull/1237 so hopefully that'll fix that issue.

joperator commented 4 years ago

That sounds good @crummel. I've already tried to build your arcadeUpgrade branch in the crummel/dotnet_source-build repo, but without success because I can't install the bootstrap CLI since init-tools.sh has been removed. I copied the bootstrap CLI into the .dotnet folder, but that didn't work either due to the following error:

Found bootstrap SDK 3.0.100, bootstrap Arcade 1.0.0-beta.19359.6
grep: /proc/1/cgroup: No such file or directory
+++ dirname ./build.sh
++ cd -P .
++ pwd
+ scriptroot=/root/git/crummel/dotnet_source-build
+ '[' false == true ']'
+ /root/git/crummel/dotnet_source-build/eng/common/build.sh --restore --build -c Release --warnaserror false /p:Projects=/root/git/crummel/dotnet_source-build/build.proj
dotnet-install: Downloading link: https://dotnetcli.azureedge.net/dotnet/Sdk/3.0.100/dotnet-sdk-3.0.100-freebsd-x64.tar.gz
curl: (22) The requested URL returned error: 404 
dotnet-install: Cannot download: https://dotnetcli.azureedge.net/dotnet/Sdk/3.0.100/dotnet-sdk-3.0.100-freebsd-x64.tar.gz
dotnet_install: Error: Could not find/download: `.NET Core SDK` with version = 3.0.100
dotnet_install: Error: Refer to: https://aka.ms/dotnet-os-lifecycle for information on .NET Core support
Failed to install dotnet SDK (exit code '1').

Are there any updates on how to install the bootstrap CLI now?

jasonpugsley commented 4 years ago

If you're looking for a recent SDK for FreeBSD 12 you can take a look here. Ensure you install the packages in DotNet3.0.0_BasePackages_FreeBSD.12.tar https://github.com/jasonpugsley/core-sdk/wiki/.Net-Core-3.0.0-for-FreeBSD-12

joperator commented 4 years ago

Thanks for the guide you provided, @jasonpugsley. I'm able to build coreclr and core-setup as you described, but I had / have problems building corefx and core-sdk:

And I think I found some small mistakes in the patches that are included in your guide:


Maybe you could adjust that in the guide.
jasonpugsley commented 4 years ago

Thank you for the feedback. The missing packages you're seeing with corefx probably means it's not looking in your local package staging directory first. Try clearing all of the caches, the global one is ~/.nuget/packages/ Then each repo may have one too. e.g .packages/ for coreclr and core-setup .nuget/packages/ for core-sdk

The error you're seeing with core-sdk probably requires you to clear the caches too.

Ensure the local pkg source has been configured correctly.

Oh, and use this guide now https://github.com/jasonpugsley/core-sdk/wiki/.Net-Core-3.0.0-for-FreeBSD

joperator commented 4 years ago

Thank you very much for updating your guide, @jasonpugsley. Building corefx now works without adjustments. I still had the same issues when building core-sdk even after clearing the NuGet cache and ensuring that the local package source was configured correctly. After renaming the local package source, trying to build core-sdk again, re-renaming the local package source and clearing all NuGet caches between each step, the build suddenly works. @wfurt, @crummel: Can we use this guide to build more of source-build?

wfurt commented 4 years ago

I wanted to try instructions and execute all corefx tests once again. What is problem with SDK @joperator ? Roslyn once again? I asked few colleagues but I did not get good answer how to approach that.

jasonpugsley commented 4 years ago

Aside from the building of the SDK that is installed from the dotnet-sdk-3.0.100-freebsd-x64.tar.gz archive, there are a set of nuget packages that the builds create. Some of these are important for building the SDK itself, some may be needed for building other software.

We need to decide where to host these packages so they're easily available during a "restore" operation. It may also require some already-published official parent packages to be updated to have references to our packages - though I haven't done an audit to see which, if any, these would be. I expect the packages for release 3.0 are frozen anyway, but maybe 3.1 could have FreeBSD packages officially hosted if the testing that @wfurt mentions meets the required pass-rate?

Otherwise, we should look at hosting them somewhere semi-official and put instructions together to edit the user's nuget configuration so it first looks there for packages during restore.

I'm suggesting this in order to have FreeBSD appear, as much as possible, to be part of the .Net Core community during the lifetime of 3.1. I feel it would be a bit of a shame to have to wait on the outskirts for another year before joining in with .Net 5.

The goal should be to make it easy for newcomers to start using .Net Core on FreeBSD. If we can remove any barriers to adoption then it should be easier to gauge real interest in this project and so how much extra effort it warrants putting in.

Maybe this discussion should move to a separate issue rather than go off topic?

joperator commented 4 years ago

The current problem with the release/3.0 branch in source-build is that it does not find the bootstrap CLI anymore, as I mentioned here. So I'm not sure if work should continue in the release/3.0 branch or in master where it's still possible to initialize the bootstrap CLI. Maybe you know, @wfurt. If it's master, I would give the self-built .NET Core SDK a try to build source-build perhaps with Roslyn.

Another point of interest would be whether it might again be possible to provide a daily build of the .NET Core SDK for FreeBSD using this guide and share the created NuGet packages as recommended by @jasonpugsley.

wfurt commented 4 years ago

I need to find some time and give it another try @joperator. The bootstrap should work AFAIK as long as it is (or pretends to) be the same version as build expects. I got in change for new FreeBSD RIDs but I need to figure out if we would be allowed to bring that to 3.x.

omajid commented 4 years ago

@wfurt, I was told by @danmosemsft that backporting RIDs for newer versions of platforms to existing release branches is a "tell mode" feature and not an "ask mode". Didn't sound like ship room would have any objections:

We agreed in shiproom that updates of this sort to maintain our OS support do not need approval.

From https://github.com/dotnet/corefx/pull/41605

danmoseley commented 4 years ago

FreeBSD is not in the official support matrix - therefore it's a little different to just continuing to maintain our existing support commitments. It's not necessarily a problem, it would just need to have a formal conversation. That would probably start with putting a PR up.

omajid commented 4 years ago

Thanks for correcting me!

wfurt commented 4 years ago

That link points to different OS @danmosemsft. We don't even have one for FreeBSD :) However, we do have RID already in 3.0 and #40865 was to make update to reflect version updates. I'll try to clarify willingness to take change and I'll follow-up on this thread later.

danmoseley commented 4 years ago

You're right @wfurt, bad link. Yep, not pushing back, just responding to question about process.

jasonpugsley commented 4 years ago

What else is required to reenable official builds of the SDK for FreeBSD? Do we actually need all of source-build or can we get away with just coreclr, corefx, core-setup & core-sdk?

dotnet/core-setup#5083

am11 commented 4 years ago

Long term; fingers crossed

I think the smoothest path would be if AzDO team proceed with https://github.com/microsoft/azure-pipelines-agent/pull/1906. Then runtime and sdk repos will get the proper CI/CD setup for FreeBSD. Note that FreeBSD was enabled in coreclr, corefx, core-setup back in Jenkins days (2015 to late-2018), as the CIaaS was switched to AzDO, FreeBSD leg was lost due to the lack of FreeBSD agent in public pool.

(aside from thumbing up https://github.com/microsoft/azure-pipelines-agent/pull/1906 to show our supports)

Short term; right now

For comparison, CirrusCI uses Google cloud for public FreeBSD jobs since Dec 4, 2018. e.g. corefx/.cirrus.yml. If it makes sense; we can, meanwhile, make it publish releases to your fork (@jasonpugsley). I have the pluming done elsewhere, which builds the binaries, creates github release and uploads artifacts to github release: example. The limit is 2 GB per file and we can have many assets and distribute them reliably. #it-scales

wfurt commented 4 years ago

Instructions @jasonpugsley put together worked reasonably well for me. There are again some changes but I was able to build all interesting repos and even run corefx tests after long time. There are some issues we should fix but as bonus I got Filesystem.Watcher working as well. (or at least pass all existing tests)
I think next step really is stabilizing source-build and getting fixes to master. Repo reorg is near by and after it is done we should focus on bringing support again to master. I'll try to get in as much as I can.

wfurt commented 4 years ago

There is still hope for .NET. He needs time... and safe passage across the plains of Gorgoroth. We can give him that! Certainty of death, small chance of success... What are we waiting for?

Let's march to the Black Gate!

It would be great if @jasonpugsley can update notes and rebuild final 3.0 bits. All 3.x branches as well as master need that so we should get good milage out of it.

.NET repose as we know them will disappear tomorrow. I think we should have separate issue to address 5.0/master. It should be easier IMHO but it will be radically different.

It would be great if @joperator can give it another try. The build seems fragile and I can get different results without any changes but I was able to move on with persistence.

wfurt commented 4 years ago

I pushed more changes last nigh and aspnetcore should build now as well. core-sdk builds if requested without dependencies after building at least up to core-setup prior. Full source-build currently breaks on fsharp.
I'm not sure how much we care at the moment. It seems like getting just c# would be still useful. Would it make sense to skip fsharp for now @crummel ?

crummel commented 4 years ago

I don't know that we have the ability right now to skip F# entirely - I think toolset will try to bring it in as a prebuilt, so I think we might still run into issues with F# not having BSD bits available. If we can figure out a way to skip it for BSD only I'd be fine with that, we can disable the F# tests and prebuilt auditing on BSD if necessary.

wfurt commented 4 years ago

ok. I'll give it some try over the weekend. Back then when we had FreeBSD CI build we updated core-sdk not to expect asp.net and 2.1 runtime as it was not available. Conveniently, there was already some property for that. I don't remember anything about fsharp but we probably got prebuilt from Windows.

crummel commented 4 years ago

Are there any dotnet repos with BSD CI running? It'd be good if we could get that set up.

am11 commented 4 years ago

Maybe @brettfo can confirm, but by the looks of it, fsharp uses the similar scripts call order as corefx, so it might be possible to share some updated code from those scripts? i can give it shot (to make fsharp build) if that's worth the effort at this point @wfurt.

wfurt commented 4 years ago 
Unhandled exception. System.DllNotFoundException: Unable to load shared library 'kernel32.dll' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: Shared object "libkernel32.dll" not found, required by "dotnet" (TaskId:93)
11:12:46.265 13:43>CoreCompile: (TargetId:188)
                        at FSharp.Compiler.Lib.UnmanagedProcessExecutionOptions.GetProcessHeap() (TaskId:93)
                        at FSharp.Compiler.Lib.UnmanagedProcessExecutionOptions.EnableHeapTerminationOnCorruption() in /mnt/github/fsharp/src/fsharp/lib.fs:line 534 (TaskId:93)
                        at FSharp.Compiler.CommandLineMain.main(String[] argv) in /mnt/github/fsharp/src/fsharp/fscmain.fs:line 81 (TaskId:93)
11:12:46.471  18:6>Project "/mnt/github/fsharp/src/fsharp/FSharp.Compiler.nuget/Microsoft.FSharp.Compiler.csproj" (18:6) is building "/mnt/github/fsharp/src/fsharp/fsc/fsc.fsproj" (15:15) on node 1 (default targets).

I did not figure out how it works on OSX and Linux. fsharp repo seems to have almost no platform code and this chunk seems clearly Windows specific.

wfurt commented 4 years ago

maybe. But it works on OSX and Linux somehow or we avoid getting here.

brettfo commented 4 years ago

@am11 and @wfurt I think your assumption is correct and the version number check was only accidentally working on Linux. On an Ubuntu 18 box System.Environment.OSVersion.Version.Major is 4 and System.Environment.Version.Major is 3.

@am11 if you send a PR to dotnet/fsharp I'll try to do a quick review and merge, but I don't know how the submodule pointers get updated, maybe @crummel can answer that.

wfurt commented 4 years ago

You may be right @brettfo On Ubuntu 16 I get 4.4.0.145 and that corresponds to kernel version from /proc/version. However on OSX 10.14 I get 18.7.0.0 -> once again corresponds to kern.version

If we feel this is right change I can add patch to #1362

am11 commented 4 years ago

There is also a HeapSetInformation condition after that one, maybe on macOS that is the one preventing the accidental call? would be nice if you could confirm whether the patch works. I have prepared a PR: https://github.com/dotnet/fsharp/pull/7858.

wfurt commented 4 years ago

With proposed change, fsharp builds for me. (quick test outside of source-build)

wfurt commented 4 years ago 
[furt@daemon /mnt/github/source-build]$ time ./build.sh
...
...
...
ls-local/prebuilt-baseline-online.xml' [/mnt/github/source-build/repos/known-good.proj]
    2701 Warning(s)
    0 Error(s)

Time Elapsed 00:59:53.44

real    59m54.934s
user    52m7.936s
sys 21m48.115s    2701 Warning(s)
    0 Error(s)

I can finally build everything. I did hit some weird issues while trying to build various repos. It turned out I was running of out memory and 4G is not enough. After moving to 8G in my VM build is more stable even if it still aborts randomly once in a while.

FeodorFitsner commented 4 years ago

Are ASP.NET Core 3.0 apps supported? I'm trying to run a very basic ASP.NET Core app with just pre-built SDK/packages from this guide and getting the error:

It was not possible to find any compatible framework version
The specified framework 'Microsoft.AspNetCore.App', version '3.0.0' was not found.
  - No frameworks were found.

You can resolve the problem by installing the specified framework and/or SDK.

The .NET Core frameworks can be found at:
  - https://aka.ms/dotnet-download

Steps to reproduce:

$ mkdir web1
$ cd web1
$ ~/dotnet-sdk/dotnet new web
The template "ASP.NET Core Empty" was created successfully.

Processing post-creation actions...
Running 'dotnet restore' on /usr/home/user1/web1/web1.csproj...
  Restore completed in 115.68 ms for /usr/home/user1/web1/web1.csproj.

Restore succeeded.

$ ~/dotnet-sdk/dotnet run
It was not possible to find any compatible framework version
The specified framework 'Microsoft.AspNetCore.App', version '3.0.0' was not found.
  - No frameworks were found.

You can resolve the problem by installing the specified framework and/or SDK.

The .NET Core frameworks can be found at:
  - https://aka.ms/dotnet-download
wfurt commented 4 years ago

did you publish packages and updated your nuget config? I don't think the bootstrap sdk had ASP.NET.

FeodorFitsner commented 4 years ago

I've tried that web app example with bootstrap SDK. So, in order to have ASP.NET Core packages I have to go through the entire build process from the article, i.e. bootstrap SDK has only minimum to build "normal" SDK, correct?

wfurt commented 4 years ago

yes, at least for now. The bootstrap @jasonpugsley put together is based on pre-3.0. If he (or somebody else) makes update, it would be possible to consume package directly without full build. I have not done any testing personally with ASP.NET. However I still should have my build somewhere. I can publish my packages if you want to give it try.

We may have enough now that it may be worth of crafting real package for ports collection.