Source Index and pack .pdb files into regular .nupkg

[pranavkm]

Wherever we ship DLLs that are used in customer apps, we need to include the portable PDB along side it.

We need to include portable PDBs in:

• [ ] The shared framework
• [ ] The NUPKGs

And we need to do this in:

• [ ] 2.1.0-preview1
• [ ] dev

---

Original text:

From https://github.com/aspnet/KRuntime/issues/380

The plan would be to create a single nupkg rather than a .nupkg and .symbols.nupkg as part of our build and use SourceLink to support having pdbs sit in the package.

[ctaggart]

Good discussion. It sounds like for nuget packages that are not .NET Core Framework/Runtime itself, adding the portable symbols to the nuget packages is still the simple solution that works today. It is worth noting that several projects including all of the Google Cloud libraries are already doing this. However, a large number of projects are waiting for Microsoft to do this too. Newtonsoft.Json is one such library.

A demo of how easy it is to debug into a source link enabled pdb file if it was included in Newtonsoft.Json: https://youtu.be/gyRGhCQPkB4?t=1m

[ctaggart]

We have a high bar for putting files into our nuget packages, so we are making sure we do due diligence on this one. @muratg and @Eilon are currently assigned on this issue to make a decision.

@muratg & @Eilon, how much longer before you complete due diligence on this and make a decision? The .NET ecosystem will benefit greatly from source link enabled pdb files bundled in the nuget packages.

[ctaggart]

A draft of the spec for NuGet Package Debugging & Symbols Improvements was announced yesterday. I commented there, but I think it is worth repeating here.

A long term plan is forming to put source link enabled pdb files into .snupkg files (new extension). SourceLink 3 will probably ship in the same time frame, with much of the functionality moved into msbuild and nuget.

Until those tools are ready, can aspnet please ship the portable pdb files in the nupkg files? That works today with SourceLink 2 and also your custom implementation.

[Eilon]

Ok, yes, we are biting the bullet and we are doing this!

Please see the updated text in the top of this issue for what we plan to do.

We are doing this as an experiment for 2.1.0-preview1 to see how well this works. We want to see the actual net increase in installer size and package sizes, as well as get feedback on the actual experience of debugging things.

[ctaggart]

Yay! When using the new project system, target .NET Core to get a good experience. https://github.com/dotnet/sdk/issues/1458 needs to be fixed in order to have a good experience when targeting the .NET Framework. dotnet sourcelink test file.nupkg can be run to see if all the source code is available for each of the nupkg packages (or individual pdb files).

[Eilon]

O, change of plans, sorry for the confusion. We (ASP.NET team) discussed this further with the .NET team and decided to not include the PDBs in the NUPKGs or shared runtime, but to instead include them on a symbol server that will have everything needed for debugging. We're tracking that work item here: https://github.com/aspnet/Universe/issues/850

Ultimately, our goal is to have a great debugging experience for everyone, and I think this new approach will achieve that. This is a very fluid space with a lot of ever-changing variables, but I think we're getting closer to a great solution.

[ctaggart]

We are doing this as an experiment for 2.1.0-preview1 to see how well this works.

You didn't even do the experiment to see how well it works. You decided against it without giving reasons why. You chose a solution that does not work currently for the rest of the .NET community who can't publish to symbol servers.

[Eilon]

Sorry about that, let me elaborate. We are working on getting symbol server support for additional debugging scenarios in the .NET Core 2.1 timeframe. So, by publishing PDBs to those servers, debuggers will be able to pull down the PDBs and you get a great debugger experience. The PDBs already have source linking enabled, so you can step straight into GitHub sources (wherever that specific feature is supported).

Can you clarify the remark about "the rest of the .NET community"? This issue applies only to the PDBs for ASP.NET/EF Core, unless I've misunderstood some requirements.

[ctaggart]

So, by publishing PDBs to those servers, debuggers will be able to pull down the PDBs and you get a great debugger experience.

Symbol Servers 👎

My experience with Symbol Servers is usually not a great experience. What makes using Symbol Servers a better experience then including the pdb files in the nupkg? If you put the pdb files in the nupkg, you don't need Symbol Servers. It is one less system to deal with (host, configure, manage).

host

There are lots of options to host nuget feeds, including services like MyGet and AppVeyor in addition to NuGet Gallery. I am not aware of Symbol Server services.

configure

Using nuget clients like nuget.exe and paket you can specify a list of sources. I am not aware of any solution like that for symbol servers.

manage

To clean up disk space, it is pretty easy to delete nupkg files that are versioned. You can easily delete all of the prerelease versions before the last stable release. If you use Symbol Servers, how do you go about removing all of the corresponding pdb files?

[Eilon]

Because this is just a question for the .NET/ASP.NET/EF Core packages, there's just one symbol server, the Microsoft one. At least when using VS, the symbols are cached in a folder of your choice, so it's easy to clear them all out (but perhaps not as easy to clear out just a subset):

One of the downsides of including the PDBs in the NUPKGs is the increased size and that most of the time they aren't used. Keep in mind that most DLLs in the .NET Core release are themselves doubled (one MSIL DLL, and one crossgen'ed DLL), so there are also two PDBs (one for each DLL).

[vancem]

Note that we are not saying that people should not put PDBs into their main Nuget packages. Indeed we suggest that they do precisely because we do NOT have a story for 3rd parties to use a symbol server (we have plans, but they are not ready).

However this particular work item is of much narrower scope. It is whether ASP.NET (and certainly the .NET Framework) should package its PDBS into their nuget packages.

Now the ASP.NET and .NET Framework team are a bit different than most other in the .NET community because

1. There is already a well established symbol server (namely http://msdl.microsoft.com/download/symbols) that can be used to host PDBS created by Microsoft, and this symbol server is already trusted (that is it would be considered a very serious security bug if hackers could exploit this server to violate our customers).

2. We generate MUCH More of the code of a typical app. Between ASP.NET and the .NET Framework itself, typically 80% or more of the application comes from these Nuget packages. That makes any size impact greater. We know from internal experiments that the size WILL be a factor (we expect a 20% to 40$ increase in package size (probably toward the bigger end). This translates directly into download times, which are already known to be a pain point in some scenarios (again because apps contain so much of our code).

3. While ASP.NET does not have native code, the .NET Framework does. Native PDBs are much larger, and thus the problem is bigger. Within Microsoft at least, a solution that was not different between native and managed code would be useful.

Note that other package managers (e.g. for Linux etc), have come to the same conclusion (that having separate packages for symbols is good), for undoubtedly the same reason (only dev scenarios need the symbols, but the are large, so why make everyone pay for something that few need (pay for play).

[ctaggart]

Note that we are not saying that people should not put PDBs into their main Nuget packages. Indeed we suggest that they do precisely because we do NOT have a story for 3rd parties to use a symbol server (we have plans, but they are not ready).

Can we agree that the .NET community should put source linked PDBs into their main NuGet packages until there is a better solution? I think you are already saying this, but I just want to confirm. I think with these projects finally enabling source link, it will be a major win and other major .NET projects will finally adopt it too. These Microsoft projects lead by example. With your clear explanation of why Microsoft is different and is using a symbol server, that should be good enough to help spur adoption.

[clairernovotny]

@ctaggart I think that's the direction NuGet is going, either there or with the snupkg files. The snupkg files just being a delivery mechanism to the NuGet symbol server. I think they were looking at also indexing pdb's in the main package so either would work.

[vancem]

I agree with @ctaggart point that people besides Microsoft should be

1. Enabling their build to use SourceLink (assuming their source code is hosted on the web in some way (e.g. GitHub). Most Microsoft code is SourceLinked now and if it is not it is a bug.
2. They should include their (portable) PDBs in their nuget packages. Microsoft does not do this but only because of special Microsoft circumstances (given in my previous comment).

We strongly encourage people to take these steps. Ultimately we want to have a secure symbol server story that everyone can use, but that that is far enough away that people should NOT wait. They should be including PDBs now.