search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
14.95k stars 4.65k forks source link

Size of source-build intermediate packages have increased significantly #92240

Closed NikolaMilosavljevic closed 2 months ago

NikolaMilosavljevic commented 12 months ago

It appears that there was some new feature work in 9.0 related to LLDB-focused debug info, that has caused a significant increase in source-build intermediate packages. Sample affected build: https://dev.azure.com/dnceng/internal/_build/results?buildId=2267121&view=artifacts&pathAsName=false&type=publishedArtifacts

Here is the comparison between 8.0 and 9.0:

Package 8.0 (MB) 9.0 Preview 1 (MB) 9.0 Preview 7 (MB)
runtime.Crossgen2Archive n/a 303 106
runtime.Crossgen2Pack 265 303 106
runtime.ILCompiler 66 115 131
runtime.linux (main package) 184 443 425
runtime.runtime.linux 131 301 326

Some open questions:

cc @agocke @dotnet/source-build-internal

mangod9 commented 11 months ago

@agocke, this looks related to the upload issue we had hit a few weeks ago? Has a fix been made for it to strip native symbols for NativeAOT?

mangod9 commented 2 months ago

@NikolaMilosavljevic, assume this is no longer an issue?

mangod9 commented 2 months ago

Closing for now since I believe this was an intermittent issue.

NikolaMilosavljevic commented 1 month ago

Preview 7 sizes are a bit better, but most of the concerns are still there.

NikolaMilosavljevic commented 1 month ago

@mangod9 the main concern is still present in Preview 7 builds

Main intermediate package (Microsoft.SourceBuild.Intermediate.runtime.linux-x64.9.0.0-rc.1.24402.1.nupkg) contains the following large-size packages:

Package Size (MB)
dotnet-apphost-pack-9.0.0-rc.1.24402.1-linux-x64.tar.gz 86
Microsoft.NETCore.App.Host.linux-x64.9.0.0-rc.1.24402.1.nupkg 86
Microsoft.NETCore.App.Runtime.linux-x64.9.0.0-rc.1.24402.1.nupkg 153
runtime.linux-x64.Microsoft.NETCore.DotNetAppHost.9.0.0-rc.1.24402.1.nupkg 88

singlefilehost is present in 3 of these 4 packages. Host size is close to 200 MB

mangod9 commented 1 month ago

@agocke, assume there isn't much that can be done here? The last time the issue was that there were symbols included as part of NativeAOT publish.

jkotas commented 1 month ago

The symbol stripping is disabled in source build for all binaries by default. Distro builds want the symbol stripping disabled in our build. They have their own process to strip and archive the symbols. This behavior is controlled by KeepNativeSymbols that is set to true by default here: https://github.com/dotnet/runtime/blob/a4fe68fcb3233b8b229444557d38c7565938de05/Directory.Build.props#L394 .

mangod9 commented 1 month ago

in that case @NikolaMilosavljevic guess we should figure out what thresholds these package sizes are breaching? Is it some upload limits?

NikolaMilosavljevic commented 1 month ago

in that case @NikolaMilosavljevic guess we should figure out what thresholds these package sizes are breaching? Is it some upload limits?

The threshold for AzDO upload is 500 MB per artifact. Intermediate package sizes are growing, and this should be monitored. Perhaps we already have artifacts size tests in AzDO push code.

Full, product, source build is not affected as we do not use intermediate packages. This can only, potentially, affect repo-level source-build.