search

NuGet / Home

Repo for NuGet Client issues
Other
1.5k stars 252 forks source link

Proposal for better symbol handling #1696

Open clairernovotny opened 9 years ago

clairernovotny commented 9 years ago

The current mechanism for symbol handling in NuGet, pushing symbol packages to symbol source, is no longer sufficient for some usages. The .NET Native tools require symbols to be available alongside the dll at build time, not on-demand at debug time as loaded from a symbol server.

Including symbols, especially source-indexed symbols, alongside the dll in the main package is less than ideal for a few reasons: they're large and vastly increase the package size for everyone, and it hinders "Just My Code" debugging by treating all packages as "my code."

I propose one of two options:

  1. Enable publishing of .symbols. packages to NuGet.org and treat symbols packages like a satellite locale package -- upon install/restore, the contents are copied to the "parent" package in the right TFM directory. As a possible optimization to symbols package size, an additional naming convention can be used to split out on pivots -- say symbols.uap10.0-2.0.0.nupkg (the exact convention to be worked out...).
  2. Start with 1, but don't copy symbols to the parent package. Instead, make the build pipeline "NuGet aware" so that the symbols are picked up in the right places. The same should be done with the locale satellites too, especially given the shared nature of the local package store now.

/cc @terrajobst

terrajobst commented 9 years ago

Thanks for writing this up! Please join this discussion in the Roslyn rep.

ctaggart commented 9 years ago

I discussed the distribution of pdb files (symbol files) with @yishaigalatzer at the MVP Summit last week. The Portable PDB format makes the PDB files compact enough to include in the NuGet packages. I think this is the best approach, but there are 3 options.

  1. Include (portable) pdbs in the nupkgs.
  2. Create satellite nupkgs for pdbs like runtime.json is used for native dependencies. May be debug.json?
  3. Use symbol servers.

I don't like symbol servers and I'd rather see NuGet used to distribute pdb files.

Regarding "Just My Code", that is a separate discussion that should be brought up with the Visual Studio team. There are a few adjustments to the VS tooling that would improve the experience such as adding a Only modules in my solution option here as well as setting a default cache directory so that source files may be downloaded.

image

yishaigalatzer commented 8 years ago

This is something we are reviewing and discussing with the dotnet team.

clairernovotny commented 8 years ago

@yishaigalatzer I'm glad there's discussion happening, but can it please happen in public somewhere?

Thanks!

yishaigalatzer commented 8 years ago

@onovotny you jumped before i finished retyping :)

We are having a discussion with the debugger team, not everything happens in the open still. The main thing here discussed is schedule, and that is not super appropriate here.

Once we can lock down resources/schedule we can visit this again.

clairernovotny commented 8 years ago

Thanks - agree that schedule is harder, the more relevant part for the public is design/spec type of discussion so we can find the best solution :)

xgalaxy commented 8 years ago

Just wanted to throw my 2cents in on this subject.

I work in an office with a bunch of Java developers and I am constantly envious of the *-sources.jar tooling that is available to them. This approach is so much better than symbol servers. It is kind of absurd that a viable parallel doesn't exist in .NET land.

I recently came across a project called GitLink that is capable of rewriting PDB's to link to the versioned sources in a GIT repo. This is really cool, but obviously something like this isn't really viable for people that don't use GIT.

I really don't think symbol servers are a good solution. Convincing my company to setup private NuGet feeds was difficult enough. It would just be much easier if we had the ability to package symbols with NuGet.

ctaggart commented 8 years ago

@xgalaxy GitLink is a tool to do source indexing. The pdb parsing code was derived from my project, SourceLink. It is most often used with Git, but doesn't require it. There are Subversion and TFS examples too.

xgalaxy commented 8 years ago

@ctaggart Yes. I understand. My larger point is that it is a shame that getting libraries off of NuGet makes it impossible for me to source step debug into those libraries unless those library authors happened to submit packages to a publicly accessable symbol server or used not well known tools such as SourceLink or GitLink. Java has a much better solution to this with their *-sources.jar tooling.

clairernovotny commented 8 years ago

Ping? Any movement here on this issue?

terrajobst commented 8 years ago

For context, see this update in the roslyn repo.

yishaigalatzer commented 8 years ago

There is no progress on this issue as indicated by the milestone (vNext, as in not now). We think we have a completely different idea for supporting the .NET native tool chain (by making it pull from symbol servers).

Bloating the packages IMHO is not a good idea, and I like @onovotny 's idea, as it lines up with where we want to go. The key issue now is resourcing and prioritizing this work. As current all of us are heads down on

  1. .NET CLI
  2. All sources
  3. Protocol speed improvements

At the moment we have no people available to look into this.

ctaggart commented 7 years ago

I wasn't the first one to bring this up at the Microsoft MVP conference this last week. It feels like a good time to revisit this. I really love the new cleaner dotnet cli tooling. Amazing job with the much simpler msbuild project files!

Big Picture = Open Source to Everyone

Adding the portable pdb files to the nupkg will allow Visual Studio, Visual Studio Code, and other debuggers access to the source code on demand while debugging, and probably when editing too.

Portable PDB Files are Small Enough

I understand the argument that legacy pdb files are too big to package in the nupkg files. Although, I think the benefits outweigh the larger file size even for legacy pdb files. Portable pdb files are significantly smaller, cross platform, and are small enough to package in the nupkg files.

Example

I like using Roslyn nupkg as examples, since if Roslyn leads the way by packaging the portable pdb files, everyone will follow suit. Using Microsoft.CodeAnalysis.CSharp as en example, including the legacy increases the nupkg size by 180%. The portable pdb file increases a mere 40%. The pdb file size on disk is about the same size as the doc xml that is included.

debugtype pdbonly: image

debugtype portable: image

without pdb, 1,878 KB image

debugtype full, increase from 1,878 KB to 5,260 KB, 180% increase image

debugtype portable, increase from 1,878 KB to 2,639 KB, 41% increase image

Making the .NET Open Source More Open

In summer of 2013, I hoped the .NET community would adopt source linking and packaging the pdb files. Several projects did, but not enough. It is really amazing when they do and it works. I think with portable pdb files and the newest revision in dotnet msbuild tooling, we have the chance to make it happen.

Status

I would love to hear from @yishaigalatzer and the NuGet team and see if we can revisit the decision to include the portable pdb files. It really is the simplest solution for today. Regarding the tooling for portable pdb files, I noticed .NET Core 1.1 is going to have some enhancements. I didn't get a change to sync up with @tmat and I would love to get an update on that front.

yishaigalatzer commented 7 years ago

Including pdbs is going to be supported by default in the new pack target.

@tmat has plans with his team to up the debugging support (and can perhaps talk to that).

So from the nuget side we are on trajectory to resolve this, hopefully soon.

jairbubbles commented 7 years ago

@yishaigalatzer Is there a way to test this ? Will it be available into command line too ?

ctaggart commented 7 years ago

image

Including pdbs is going to be supported by default in the new pack target.

@yishaigalatzer, I'd like to confirm that including Portable PDB files in NuGet packages is now acceptable going forward. It is pretty critical for source link support to prosper. Now that NuGet and dotnet share downloaded packages, this makes even more sense now. FYI, I'm submitting my helper tools to the .NET Foundation, to help gain community support & adoption.

Having a road map to fix issues like this would help:

ctaggart commented 7 years ago

Several of these issues are related. I have a list at https://github.com/ctaggart/SourceLink/wiki/Contributing#issues

By adding a source link in the portable pdb file and then embedding the portable pdb file in the dll by doing <DebugType>embedded<DebugType>, that goes in the nupkg, and we have a wonderful solution. It looks like aspnet will hopefully be the first adopters. https://github.com/aspnet/Universe/issues/131#issuecomment-283131373

stakx commented 7 years ago

@onovotny wrote above:

Including symbols […] alongside the dll in the main package […] hinders "Just My Code" debugging by treating all packages as "my code."

@ctaggart: Do you know whether this "pain point" has been addressed in the meantime (and if so, is there a separate GitHub issue for it)? I've run a quick experiment and Visual Studio 2017 appears to honor "Just My Code" with a NuGet package containing source-linked PDBs, but I'd like to have this confirmed by someone more knowledgeable.

I'm currently preparing a SourceLink-ed version of Moq, and "Just My Code" not working is the only concern I have about including source-linked PDBs in the NuGet package. I feel it would be bad if distributing the PDBs that way meant that the end user would have no way of stopping their debugger from diving into Moq's source code (which might not be desired in every case).

(And last but not least, I'd like to say "thanks" for the great work you did with SourceLink!)

anangaur commented 4 years ago

@onovotny Is this issue relevant after the .snupkg support? https://devblogs.microsoft.com/nuget/improved-package-debugging-experience-with-the-nuget-org-symbol-server/

clairernovotny commented 4 years ago

This issue was more about how .NET Native needed the symbols alongside the dll. A symbol server wasn't enough. That said, .NET Native is probably not a high priority these days, I'm guessing.

anangaur commented 4 years ago

Yepp.. not high on our priority list, as of now. Should this issue be closed or left open to track further comments?

clairernovotny commented 4 years ago

I'd prefer keeping it open to track against .NET Native so it doesn't get lost