Closed tmds closed 1 month ago
This issue is for building and including the Runtime package with the sdk.
Runtime packs are similar to targeting packs, so IMO they should be included in e.g. dotnet-runtime-pack-x-y
packages, which the SDK package would have a dependency on.
This makes it possible for the maintainer to apply patches during source-build, and the applications built this way using the sdk can be considered source-buildable.
A note about this from https://github.com/dotnet/source-build/issues/1202, to put it in context: the runtime pack has limitations on what it supports, due to glibc compatibility and portable vs. non-portable builds. An SCD publish for linux-x64
would need to bring down the extra-compatible prebuilt, but e.g. fedora.30-x64
would be source-built.
I believe the SDK and perhaps Core-SDK repos will need to add support for this. The SDK currently locates targeting packs with TargetingPackRoot
($(NetCoreTargetingPackRoot)
) which points to e.g. /usr/bin/dotnet/packs/
. The runtime pack resolver task doesn't seem to have anything similar.
The definition of NetCoreTargetingPackRoot
is over in Core-SDK.
The apphost pack resolver uses TargetingPackRoot
too, despite the name, so I guess it might be fine to use it for runtime packs too.
It makes sense to make it work similar to the apphost package.
Can we set a target milestone for this?
I can probably implement this based on these references.
Adding @dsplaisted
@dsplaisted @marcpopMSFT @MichaelSimons @dseefeld this issue is popping up as part of the s390x support because without it, users cannot create self-contained deployments.
Single-file publish is not possible either, but that is not a feature with the mono runtime (yet).
Can we look at including the runtime package in all source-built SDKs for .NET 7?
cc @omajid @uweigand
The current .NET 6 SDK already checks for runtime packs in the packs/
subdirectory, it just doesn't find any because none were installed. I've been able to do using the following straightforward patch to the installer:
Index: installer/src/redist/targets/GenerateLayout.targets
===================================================================
--- installer.orig/src/redist/targets/GenerateLayout.targets
+++ installer/src/redist/targets/GenerateLayout.targets
@@ -134,6 +134,13 @@
<RelativeLayoutPath>packs/%(PackageName)/%(PackageVersion)</RelativeLayoutPath>
</BundledLayoutPackage>
+ <BundledLayoutPackage Include="MicrosoftNetCoreAppRuntimePackNupkg">
+ <PackageName>Microsoft.NETCore.App.Runtime.$(SharedFrameworkRid)</PackageName>
+ <PackageVersion>$(MicrosoftNETCoreAppRefPackageVersion)</PackageVersion>
+ <TargetFramework>$(TargetFramework)</TargetFramework>
+ <RelativeLayoutPath>packs/%(PackageName)/%(PackageVersion)</RelativeLayoutPath>
+ </BundledLayoutPackage>
+
<BundledLayoutPackage Include="MicrosoftNetStandardTargetingPackNupkg">
<PackageName>NETStandard.Library.Ref</PackageName>
<PackageVersion>$(NETStandardLibraryRefPackageVersion)</PackageVersion>
@@ -146,6 +153,13 @@
<PackageVersion>$(MicrosoftAspNetCoreAppRefPackageVersion)</PackageVersion>
<TargetFramework>$(TargetFramework)</TargetFramework>
<RelativeLayoutPath>packs/%(PackageName)/%(PackageVersion)</RelativeLayoutPath>
+ </BundledLayoutPackage>
+
+ <BundledLayoutPackage Include="MicrosoftAspNetCoreAppRuntimePackNupkg">
+ <PackageName>Microsoft.AspNetCore.App.Runtime.$(SharedFrameworkRid)</PackageName>
+ <PackageVersion>$(MicrosoftAspNetCoreAppRefPackageVersion)</PackageVersion>
+ <TargetFramework>$(TargetFramework)</TargetFramework>
+ <RelativeLayoutPath>packs/%(PackageName)/%(PackageVersion)</RelativeLayoutPath>
</BundledLayoutPackage>
<BundledLayoutPackage Include="MicrosoftNetCoreAppHostPackNupkg">
This likely still needs a bit of tweaking to use an appropriate condition when to do it. Also, it currently installs only the $(SharedFrameworkRid)
... maybe it also should install alternate RIDs as is done for the Apphost?
As a drawback, this approach significantly increases the size of the SDK installation on disk. This is annoying in particular as most of the files installed as part of the target runtime pack are in fact already present in the SDK on-disk installation, under the shared/
directory ... but it looks like that cannot be used as a drop-in replacement because the directory structure is different.
This likely still needs a bit of tweaking to use an appropriate condition when to do it. Also, it currently installs only the $(SharedFrameworkRid) ... maybe it also should install alternate RIDs as is done for the Apphost?
We can only build (rid-specific) packs for the target rid.
This likely still needs a bit of tweaking to use an appropriate condition when to do it. Also, it currently installs only the $(SharedFrameworkRid) ... maybe it also should install alternate RIDs as is done for the Apphost?
We can only build (rid-specific) packs for the target rid.
I'm not sure I understand what you mean here. This is not about building those packs (they have to have been built previously), this is just about the installer build packaging them up for distribution as tarball.
For example, in the current source-build, we build the runtime twice, and that generates two separate App.Host packages, and both of them get packaged up by the installer. Similarly, those two runtime builds already also generate two App.Runtime packages, so the installer could package both of those as well, if we think that would be useful.
I'm not sure I understand what you mean here.
If you're on Fedora 34, you can claim you're building the host or runtime for fedora.34-x64
but you can't claim to be building the portable linux-x64
.
The latter is meant to work across a range of distros. Microsoft builds it and publishes it for others to consume on nuget.org.
For example, in the current source-build, we build the runtime twice, and that generates two separate App.Host packages, and both of them get packaged up by the installer. Similarly, those two runtime builds already also generate two App.Runtime packages, so the installer could package both of those as well, if we think that would be useful.
What is the difference between the packages?
Is one using the target rid (like fedora.34-x64
) and the other using the portable rid (like linux-x64
)?
If that is the difference, the latter isn't really portable, because it will in some ways depend on Fedora 34.
It's a package that pretends to be the portable rid for the sake of source-build. It is not meant for the user.
Can we look at including the runtime package in all source-built SDKs for .NET 7?
Due to the size of the runtime packs, it might be a good idea to have them separate and optional.
Or maybe it would be a good idea to be able to create source built NuGet packages and have them be acquired via NuGet in the same way the non source-built ones are.
.NET 7 will add support for NativeAOT.
We'd like a source-build SDK to be able to build self-contained NativeAOT applications.
This issue was already tracking adding support for self-contained, and I've added NativeAOT in the title too. We can split it up when that makes sense.
I've summarized what I believe to be the current state of play and the agreed-upon solution for NativeAOT in 7.0 in https://github.com/dotnet/runtime/pull/76206. Please check this out and let me know if you have any concerns.
@MichaelSimons you were asking about this issue earlier today?
@tmds would you agree this is more important than supporting building and using workloads from source?
Yes. These features apply to any app, while the workloads are for specific use-cases.
@richlander I'm wondering, do you think it would be beneficial in the future if Microsoft sdk images include some of these features (so: without having to reach out to nuget.org) as the features are becoming more prevalent?
For self-contained deployment -
Self-contained deployment acquires runtime packs and AppHost packs (cross-target) from NuGet (MS Built). These are not packaged with SDK due to the size constraint. And so, as the packages are not offered with Linux distribution-built SDK as well and end-up acquired from the NuGet. To acquire packages which are source-built by Linux distributions, here are few options.
A few points to note –
@dsplaisted @omajid @tmds @MichaelSimons @baronfel
These are not packaged with SDK due to the size constraint.
I was wondering about just that size issue. The main problem appears to me that if we place a copy of the Microsoft.NETCore.App.Runtime.<rid>
package under the packs/
directory, then all those files are present twice: under packs
and under shared/Microsoft.NETCore.app
.
Is this duplication really necessary? Would it be possible to rework the dotnet loader so it takes all the runtime assemblies (and other files) from the packs/
directory if present, so we would then just not need the shared/
directory at all? Or am I missing something here?
I also think size is an important factor, esp if we'd like to include these features in a container image.
The duplication isn't needed, but it comes from staying close to how things work now with the nuget.org packages. And since that is working well, I imagine there isn't much interest upstream in exploring a design to minimize the size. And besides the effort, there's also the risk of breaking things by entangling them.
For self-contained deployment
What about R2R and NativeAOT?
The idea has come up before that we could use the shared framework from shared
instead of the runtime pack from packs
in order to save space or downloads. I think that even though they are mostly the same, there are still some differences. It's possible those differences wouldn't apply to source-built versions of the .NET SDK. I think @jkotas knows the details.
We do not want to have hard coupling between the runtime used by the build output with the runtime that that the SDK happens to run on. The output of the build can use different (servicing or major) version, different architecture, different runtime flavor (mono, aot, ...), etc.
In theory, we can special case the case where the shared runtime exactly matches the runtime that the self-contained deployment wants to use, and skip the download of the runtime pack in that case. We did not think that the self-contained publish is common enough for this special casing and all complications that come with it to be worth it.
I also think size is an important factor, esp if we'd like to include these features in a container image.
This size factor can be fixed by linking the files to the same physical file.
This size factor can be fixed by linking the files to the same physical file.
I was wondering about that.
Whether it could be symbolic links, or have to be hard links? And if the runtime repo could provide such an output, or whether source-build repo could/should do it?
Whether it could be symbolic links, or have to be hard links?
I do not know. Some tools may have problems with symbolic links.
And if the runtime repo could provide such an output, or whether source-build repo could/should do it?
Build of any more complicated app ends up restoring something from nuget. I do not think we want to include more stuff in the default SDK or otherwise complicate things to fine-tune the amount of stuff that gets downloaded. For example, we have long-standing issue in that we download all runtime packs (even ASP.NET Core and Windows Deskop packs) when publishing self-contained console apps that we did not care enough to fix.
If the distro builds want to include more packages in place (and potentially optimize the amount of diskspace used by linking), I think it would be a specific to just the distro builds.
Some tools may have problems with symbolic links.
Yes, symbolic links may not work for different reasons.
And, hard links come with a packaging challenge because the hard links link to files that are in a different package. For example, a pack package with a hard link to a file that is provided by a runtime package.
Though the archive format may support this, the default archive tooling doesn't*. So packages with hard links need to be crafted.
(*: it can't distinguish between what package provides, and what package hard links)
And, hard links come with a packaging challenge because the hard links link to files that are in a different package. For example, a pack package with a hard link to a file that is provided by a runtime package.
Couldn't the runtime package just automatically always provide both the shared runtime and the runtime pack? If this is hard links, there wouldn't be any disk space reason not to, right?
Couldn't the runtime package just automatically always provide both the shared runtime and the runtime pack? If this is hard links, there wouldn't be any disk space reason not to, right?
Yes, though the runtime pack is used by the sdk, we could ship it in the runtime package (with no additional size cost).
We also need to package the R2R and NativeAOT tooling.
@ashnaga @MichaelSimons can we for an upcoming preview of .NET 8 target to include the runtime pack as part of the vmr output?
Once we have that, we can then for the subsequent preview focus on reducing its size either by using symbolic or hard links.
And, can you propose how we could include R2R and NativeAOT?
can we for an upcoming preview of .NET 8 target to include the runtime pack as part of the vmr output?
I would expect that vmr is building the runtime pack and R2R .nupkg files. Is that not the case?
I would expect that vmr is building the runtime pack and R2R .nupkg files. Is that not the case?
Yes, some of these nuget packages get built.
The output of the vmr is the "DOTNET_ROOT" file layout which is then packaged up by the distro maintainers. The request is to include the files from the nuget package(s) at the appropriate location under this DOTNET_ROOT.
@ashnaga @MichaelSimons can we for an upcoming preview of .NET 8 target to include the runtime pack as part of the vmr output?
Can we do this?
@ashnaga @MichaelSimons what's the next step?
@ashnaga is driving this one. I will let her respond.
@tmds Thank you for your patience. I made an evaluation to ensure that packages are picked from dotnet/packs folder if available and not being pulled from NuGet. Currently, if there is a package with following naming convention - "dotnet/packs/Microsoft.NETCore.App.Runtime.linux-x64" then only it will be picked by SDK. The naming convention does not imply distribution/RID specific packages which we are using in case of AppHost packages - https://learn.microsoft.com/en-us/dotnet/core/distribution-packaging
In order to make it work for distribution/RID specific packages, the source-built version will need to change how the Bundled versions targets file is generated: https://github.com/dotnet/installer/blob/7eea2f60fdcf419d65872c6173f2bdcd8381905a/src/redist/targets/GenerateBundledVersions.targets#L196-L212
@tmds To make this work in preview timeframe, are you happen to make these changes?
cc @dsplaisted @MichaelSimons if you have specific questions on changes in the mentioned file.
To make this work in preview timeframe, are you happen to make these changes?
If no one else is available that is more familiar, I will look into it.
In the meanwhile we should try to determine what is needed to provide NativeAOT. Then we can start working on that too.
@ashnaga do you have a picture of what is needed for NativeAOT?
A heads up: I'll be on summer leave from mid July to mid August. It would be good if we can make some more progress before that.
do you have a picture of what is needed for NativeAOT?
Add dotnet/llvm-project to https://github.com/dotnet/dotnet, and make it build objwriter there. (It should be built before dotnet/runtime.)
This requires adding some things to installer src/SourceBuild/content/repo-projects
.
With the move to the vmr, I don't know how to go about building that.
@MichaelSimons can you provide some help?
@tmds this should be a starting point for adding new repositories into Source Build: https://github.com/dotnet/source-build/blob/main/Documentation/sourcebuild-in-repos/new-repo.md
this should be a starting point for adding new repositories into Source Build: https://github.com/dotnet/source-build/blob/main/Documentation/sourcebuild-in-repos/new-repo.md
I'm looking for the command that you can run on the installer
repo to make it build up the complete sources (which is what is put into dotnet/dotnet
). I don't see it mentioned in these docs.
I think the right way is to clone the VMR, manually put the llvm
in src/
. Register the repository (repo-project
...) and build it.
Once you get that working, we will open a PR against installer
to put the repo-project
file under src/VirtualMonoRepo
and also change the source-mappings.json
. Then, when you open a PR, the PR build will validate everything.
You can also start with a PR and work your way through that but it will possibly cost a lot of iterations and compute as the PR build builds much more than the VMR.
Alternatively, you can utilize a GitHub codespace in dotnet/dotnet
or in dotnet/installer
(there's a VMR devcontainer for that). But the first step probably should always be to manually add the new repo and registering it with Source Build and trying out manually.
Couple of notes to the above: To make everything work in the PR build of installer, we will also have to make sure the LLVM flows into the VMR which means it would have to be declared as a dependency in one of the repositories (in eng/Version.Details.xml
). It is still possible to have the following workflow:
installer
llvm-project
there (in eng/Version.Details.xml
)llvm-project
repo to src/VirtualMonoRepo/source-mappings.json
in the installer
dev branch.devcontainer
)The installer VMR-specific Codespace is configured to create and sync the VMR into /workspaces/dotnet
so it should bring in the llvm-project
there.
In this workflow, you can:
installer
dev branch and awlays start a fresh Codespaceinstaller
branch, you can already add the new repo-project
files and everything - this way your changes will persist between attempts. Do not open an installer PR in this workflow - it's not required. Also I am not sure if you will need to have billing set up for the Codespace to work for you.
The following PRs will likely be useful. If you weren't already aware, there was a point in with llvm was partially building in source-build. This was prior to the VMR but most of the changes should transfer over.
https://github.com/dotnet/installer/pull/14697 https://github.com/dotnet/llvm-project/pull/242
- Declare the dependency on llvm-project there (in eng/Version.Details.xml)
Something like the following, but what to use as the Name and Version?
<Dependency Name="XXX" Version="YYY">
<Uri>https://github.com/dotnet/llvm-project</Uri>
<Sha>c01ca3bc8a420b3796d816c43bdd06e337c72ea6</Sha>
</Dependency>
- Add the llvm-project repo to src/VirtualMonoRepo/source-mappings.json in the installer dev branch
That would look like:
{
"name": "llvm-project",
"defaultRemote": "https://github.com/dotnet/llvm-project"
},
The installer VMR-specific Codespace is configured to create and sync the VMR into /workspaces/dotnet so it should bring in the llvm-project there.
This seems to come down to calling the vmr-sync.sh
script. This script defaults to syncing the installer
repository. I don't think it will sync any other repository?
I see the repository can be specified through the --repository
argument. If the repository is not yet known, will the script add it?
I tried calling the script after committing the previous changes:
$ ./eng/vmr-sync.sh --vmr ../dotnet --tmp ../tmp
...
+ /home/tmds/workspace/installer/.dotnet/dotnet darc vmr update --vmr ../dotnet --tmp ../tmp --debug --recursive --readme-template /home/tmds/workspace/installer/src/VirtualMonoRepo/README.template.md --tpn-template /home/tmds/workspace/installer/src/VirtualMonoRepo/THIRD-PARTY-NOTICES.template.txt --additional-remotes installer:/home/tmds/workspace/installer installer:16e55b6d7a6df7db7a88c45ab812163996f7c1ff
[1.1.0-beta.23226.1+cdd8256f08d1e0c0495e29986b4c9b107697a9ce / Microsoft.DotNet.Darc.exe] darc command issued: vmr update --vmr ../dotnet --tmp ../tmp --debug --recursive --readme-template /home/tmds/workspace/installer/src/VirtualMonoRepo/README.template.md --tpn-template /home/tmds/workspace/installer/src/VirtualMonoRepo/THIRD-PARTY-NOTICES.template.txt --additional-remotes installer:/home/tmds/workspace/installer installer:16e55b6d7a6df7db7a88c45ab812163996f7c1ff
dbug: Cloning https://github.com/dotnet/installer to /home/tmds/workspace/tmp/installer
info: Cloning https://github.com/dotnet/installer to /home/tmds/workspace/tmp/installer
dbug: Cloning https://github.com/dotnet/installer to /home/tmds/workspace/tmp/installer
fail: Failed to synchronize repo installer
Something went wrong when cloning repo https://github.com/dotnet/installer at <default branch> into /home/tmds/workspace/tmp/installer.
dbug: System.Exception: Something went wrong when cloning repo https://github.com/dotnet/installer at <default branch> into /home/tmds/workspace/tmp/installer
---> LibGit2Sharp.LibGit2SharpException: could not load ssl libraries
at LibGit2Sharp.Core.Ensure.HandleError(Int32 result) in /_/LibGit2Sharp/Core/Ensure.cs:line 154
at LibGit2Sharp.Core.Ensure.ZeroResult(Int32 result) in /_/LibGit2Sharp/Core/Ensure.cs:line 172
at LibGit2Sharp.Core.Proxy.git_clone(String url, String workdir, GitCloneOptions& opts) in /_/LibGit2Sharp/Core/Proxy.cs:line 275
at LibGit2Sharp.Repository.Clone(String sourceUrl, String workdirPath, CloneOptions options) in /_/LibGit2Sharp/Repository.cs:line 793
at Microsoft.DotNet.DarcLib.GitRepoCloner.Clone(String repoUri, String commit, String targetDirectory, CheckoutType checkoutType, String gitDirectory)
--- End of inner exception stack trace ---
at Microsoft.DotNet.DarcLib.GitRepoCloner.Clone(String repoUri, String commit, String targetDirectory, CheckoutType checkoutType, String gitDirectory)
at Microsoft.DotNet.DarcLib.GitRepoCloner.Clone(String repoUri, String targetDirectory, String gitDirectory)
at Microsoft.DotNet.DarcLib.VirtualMonoRepo.RepositoryCloneManager.PrepareCloneInternal(String remoteUri, String dirName, CancellationToken cancellationToken)
at Microsoft.DotNet.DarcLib.VirtualMonoRepo.RepositoryCloneManager.PrepareClone(SourceMapping mapping, String[] remoteUris, String checkoutRef, CancellationToken cancellationToken)
at Microsoft.DotNet.DarcLib.VirtualMonoRepo.VmrUpdater.UpdateRepository(String mappingName, String targetRevision, String targetVersion, Boolean noSquash, Boolean updateDependencies, IReadOnlyCollection`1 additionalRemotes, String readmeTemplatePath, String tpnTemplatePath, CancellationToken cancellationToken)
at Microsoft.DotNet.Darc.Operations.VirtualMonoRepo.UpdateOperation.ExecuteInternalAsync(IVmrUpdater vmrManager, String repoName, String targetRevision, IReadOnlyCollection`1 additionalRemotes, CancellationToken cancellationToken)
at Microsoft.DotNet.Darc.Operations.VirtualMonoRepo.VmrOperationBase`1.ExecuteAsync(TVmrManager vmrManager, String repoName, String targetRevision, IReadOnlyCollection`1 additionalRemotes, CancellationToken cancellationToken)
...
Looks like LibGit2Sharp and Fedora 37 are not getting along :/
Trying the codespace instead:
- Push the dev branch into your fork and start a Codespace (you have to select the VMR .devcontainer)
I've pushed my commit to a branch: https://github.com/tmds/installer/tree/llvm_project, and created a code space specifying the vmr config.
It composes the dotnet
directory, but there is no llvm-project
directory.
In the creation.log
there is this:
...
2023-05-31 13:32:27.094Z: warn: The following mappings do not appear in current update's dependency tree:
- llvm-project
...
What should I change to get the repo cloned into the vmr directory?
Something like the following, but what to use as the Name and Version?
See https://github.com/dotnet/installer/pull/14697 specifically https://github.com/dotnet/installer/pull/14697/files#diff-fb62e94a1d6f29f863e3d0a22aa38269f6cd1d7f03b109dc06e2cbf2548b86d3L20. This shouldn't use the SourceBuildTarball element as we are trying to remove it. Instead it should use SourceBuild but that requires LLVM be onboarded to source-build so that it has a source-build leg and publishes a source-build intermediate.
@omajid changes for bundling the runtime packs should be in preview 5. There will be two additional directories in the vmr built output: packs/Microsoft.NETCore.App.Runtime.<rid>/<version>
and Microsoft.AspNetCore.App.Runtime.<rid>/<version>
. The idea is to bundle these in the packages with their corresponding shared framework folders (shared/Microsoft.NETCore.App/<version>
and shared/Microsoft.AspNetCore.App/<version>
).
Can you do this for our builds?
If you install rdfind
on Fedora during the build, the output should have hard links instead of duplicating the files.
It will be interesting to see if they survive the packaging and install process...
Eventually we will document this as part of https://learn.microsoft.com/en-us/dotnet/core/distribution-packaging.
I've opened two PRs for getting llvm-project in the vmr: https://github.com/dotnet/installer/pull/16757, https://github.com/dotnet/llvm-project/pull/439.
@ashnaga @MichaelSimons @premun
Based on where we are with source-buildable NativeAOT, it makes sense to defer this to .NET 9 so we can focus our efforts on the upcoming .NET 8 release.
@jkotas @MichalStrehovsky Does this merge of ObjWriter in C# impact the instruction on what is needed to enable NativeAOT with source-build instructions.
PR to build llvm-project is closed now, considering this as an alternative.
Does this merge of https://github.com/dotnet/runtime/issues/77178#issuecomment-1816106564 impact the instruction on what is needed to enable NativeAOT with source-build https://github.com/dotnet/source-build/issues/1215#issuecomment-1569490565.
Yes, we are almost there. A bug-fix level work remains to enable NativeAOT with source-build - tracked by https://github.com/dotnet/runtime/issues/66859#issuecomment-1960836451
We also need to some work so the ILCompiler pack becomes part of the source-built SDK that is built by the vmr.
The work should be similar to the runtime pack we've added in .NET 8 to enable self-contained publish.
I'll look into that after looking at the bug-fix.
Thanks to the reference packs, a source-build sdk can now build .NET Core applications that can be considered themselves source-buildable.
This doesn't apply to self-contained applications, and ReadyToRun applications. These require downloading a
Microsoft.NETCore.App.Runtime.<rid>
package from nuget.org. (This includes framework dependent ReadyToRun, probably because it uses the package to aquirecrossgen
.)This issue is for building and including the Runtime package with the sdk. This makes it possible for the maintainer to apply patches during source-build, and the applications built this way using the sdk can be considered source-buildable.
cc @dagood @nguerrera @omajid