dotnet / dotnet-docker

Docker images for .NET and the .NET Tools.
https://hub.docker.com/_/microsoft-dotnet
MIT License
4.48k stars 1.94k forks source link

NuGetFallbackFolder adds unnecessary bloat to build containers #237

Closed natemcmaster closed 5 years ago

natemcmaster commented 7 years ago

In .NET Core 2.0, the CLI started shipping a feature that adds an offline package feed called the NuGetFallbackFolder. This adds unnecessary bloat to SDK images in the following ways:

Suggestions

Or some combination of the ideas above.

Details How I measured the numbers Using: microsoft/dotnet-nightly:2.0.0-preview1-sdk

cc @emgarten @rrelyea @dotnet/dotnet-cli @glennc

emgarten commented 7 years ago

This will improve once the fallback folder is used as an actual fallback folder.

Long term to solve this I'd like to see either a dotnet or NuGet command that takes a set of project.assets.json files and prunes out all unneeded files from the package folders. This would include assemblies from unused frameworks, all dlls and files not listed in the assets file, and resource dlls that aren't used.

natemcmaster commented 7 years ago

This will improve once the fallback folder is used as an actual fallback folder.

@emgarten is this just a preview1 limitation that will be fixed before 2.0.0 RTM?

Also FYI the ~/.dotnet/NuGetFallbackFolder on Linux takes up about 1.0 GB.

emgarten commented 7 years ago

@emgarten is this just a preview1 limitation that will be fixed before 2.0.0 RTM?

That is the current goal

tmds commented 7 years ago

Does this help?

ENV NUGET_XMLDOC_MODE=skip
ENV DOTNET_SKIP_FIRST_TIME_EXPERIENCE=1
natemcmaster commented 7 years ago

The images already set NUGET_XMLDOC_MODE=skip, but it's not honored by the first-time experience. If we decided to disable the offline package cache, setting DOTNET_SKIP_FIRST_TIME_EXPERIENCE is the best way to do that.

tmds commented 7 years ago

Is the /opt/dotnet/store also used during dotnet restore to avoid duplicating those packages to ~/.nuget/packages?

natemcmaster commented 7 years ago

If you meant /usr/share/dotnet/store, that is the runtime store feature that's new to 2.0. These are a subset of files from the ~/.nuget/packages folder that have been crossgened for faster boot time, similar to the "optimization cache" feature in .NET Core 1.x. The subset doesn't include all assets required to build an app.

tmds commented 7 years ago

The subset doesn't include all assets required to build an app.

Ah. So this doesn't get used by dotnet restore and the sdk image won't contain a store since it would only bloat the image.

MichaelSimons commented 7 years ago

@emgarten, What is the latest status of the underlying issue? Can you link to the product issue?

emgarten commented 7 years ago

@MichaelSimons it is tracked here: https://github.com/dotnet/cli/issues/6507

natemcmaster commented 6 years ago

It looks like the fallback folder behavior is working now. However, I think there is still an opportunity here to trim some bloat. If I've done my bash/math right, here are the (uncompressed) size of files in /usr/share/dotnet

MichaelSimons commented 6 years ago

I logged a cli issue to add support for this scenario. There are numerous use cases which would benefit from this trimming capability. Our Dockerfiles should not have to define this trimming logic.

JeanMdK commented 6 years ago

The NuGetFallbackFolder is actually taking to much room in my disk : since it's only caching packages to avoid searching them online if they're not there, can we just remove all the files from the folder ? Is it safe ? Thank you for your time, John

natemcmaster commented 6 years ago

Hey @MichaelSimons, I'd like to take a stab at this for 2.2 (after #662 is in, of course). We started producing a version of the fallback folder which trims the .nupkg and .xml files which is the biggest part of the (uncompressed) bloat.

@JeanMdK yes, you can safely delete this folder. NuGet will download what you need. You are likely to find, however, that this will just shift content from the NuGetFallbackFolder to %USERPROFILE%/.nuget/packages.

MichaelSimons commented 6 years ago

@natemcmaster - Can you provide more details on this lighter fallback folder? Is this a published artifact I can take a look at? @richlander and I have discussed this and are in favor of this for 2.2.

natemcmaster commented 6 years ago

Yes, there are published artifacts for this starting in 2.1.3. I've written up more details about the various package archives we produce in https://github.com/aspnet/Universe/pull/1324. For Docker, you would want to use the nuGetPackagesArchive-ci-server-$(Version).zip. Example: http://dotnetcli.blob.core.windows.net/dotnet/aspnetcore/Runtime/2.1.3/nuGetPackagesArchive-ci-server-2.1.3.zip. This contains the same contents as the LZMA bundled in the SDK, but without the .nupkg and docxml files.

Under the hood, NuGet restore takes a list of fallback folders in the MSBuild property RestoreAdditionalProjectFallbackFolders. By default, the .NET Core SDK adds $(DotNetInstallRoot)/sdk/NuGetFallbackFolder/ to this list. (See Microsoft.NET.NuGetOfflineCache.targets).

There is more than one way to do this, but I would recommend the following for 2.2 previews. I'm like 80% sure this should work, but in full disclosure, I haven't tried it on all platforms, just in Azure services.

MichaelSimons commented 5 years ago

The SDK is removing the lzma/NuGetFallbackFolder in 3.0 and making use of targeting packs instead. The removal has already happened.

Closing this issue, there are no plans to port any of this work back to previous versions.