search

axodotdev / cargo-dist

📦 shippable application packaging
https://axodotdev.github.io/cargo-dist/
Apache License 2.0
1.45k stars 61 forks source link

Rework Symbol Support #136

Open Gankra opened 1 year ago

Gankra commented 1 year ago

I've disabled all symbols pending this redesign of the feature, due to several issues:

I propose two fixes:

Fix 1: We shouldn't consider it a hard error to fail to generate a pdb

This creates an awkward situation with manifest --artifacts=all, due to a lack of #45. I think rather than "solving" #45 we should just add an optional: true field to dist-manifest.json indicating the symbols artifacts are ok to fail. Then we should also change cargo dist build to silently remove optional: true entries from its output if they indeed failed (preventing CI from trying to upload nothing). Someone reading dist-manifest.json then just needs to accept that they might fail to fetch those (seems fine).

Fix 2: We should unconditionally wrap every kind of symbol file in a zip

That will allow us to preserve the name rustc/cargo told us was the Right one. This will handle the most important aspects of #116, which is getting the symbols to a server/user with the right name. The consumer of such a zip just needs to understand they have to properly spider the folder. Ideally each folder will contain one entry.

I'm not 100% sure what to name the zips and the folders they contain. This is most awkward for blah.dSYM because we want blah-v1.0.0-x86_64-apple-darwin.dSYM.tar.xz to contain a dir containing the .dSYM but if we just name the dir blah-v1.0.0-x86_64-apple-darwin.dSYM things might freak out and think it's an actual dSYM (because those are dirs!). Maybe just call the inner dir blah-v1.0.0-x86_64-apple-darwin-dSYM? Are tar and zip happy with that kind of rename? I think so?

MarinPostma commented 2 months ago

Hey! We're trying to ship debug syms alongside the bins. Are there any workarounds?

edit: Having the syms only for the linux build would be enough for us right now

Gankra commented 2 months ago

Do you know the symbol format you'd like for linux? There's 2 or 3 different ones.

Gankra commented 2 months ago

So to elucidate, there's basically two major linux formats: dwp and the "strip" format.

The strip format is traditionally what linux distros use, well-supported by debugging tools, and analogous to windows pdb and apple dSYM. It allows you to produce a fully stripped and optimized binary, while still holding onto all the debuginfo/symbols separately. Unfortunately, rustc doesn't yet support generating it for you. However we can potentially have cargo-dist polyfill it.

The dwp format (and its subformat, dwo) is a newer thing that was introduced to try to make incremental/local debuginfo cheaper to construct. Without getting into the details of it, it's basically unsuitable for shippable builds, as it requires a substantial amount of debuginfo to remain in the shipping binary (bloating things a lot), and the debuginfo is meaningless without the information in the binary. The one upside of this format is that rustc shipped builtin support for it, so turning it back on isn't a ton of work.

What you need probably depends on how exactly you want to use the debuginfo (are you uploading these to sentry, your own symbol servers, or...?).

MarinPostma commented 2 months ago

Wow, thanks, @Gankra, for the detailed write-up! Our need is very basic. We simply want to debug syms to be able to download them to the production machine and profile/debug them when necessary. I think the strip format is what we need.