Add support for Go source bundles

[cleptric]

The only way to enable source context for Go is to deploy your source code next to the binary, which is odd. We should add support for source bundles for Go as well, which gives our users more options.

[szokeasaurusrex]

@cleptric I spent some time investigating this issue yesterday and today.

It appears that the Go compiler includes debug symbol information in the binaries it generates. However, the debug symbol information does not include a Debug ID, which the CLI requires in order to be able to generate and upload a source bundle. I tried commenting out the CLI code that enforces this limitation and uploading the source bundle anyways – the CLI generated and uploaded the source bundle properly, but source context did not work when I sent an example event because the lack of a debug ID means that Sentry cannot find the source bundle corresponding to the error event.

I discussed this situation on a call with @loewenheim and @Swatinem today to see if there is some way we would be able to work around this limitation. Our consensus was that – even if we were to disable the limitation in the CLI requiring debug files to have a debug ID – the only way that Sentry would be able to correlate events back to the source bundle would be if there is a unique ID associated with each build and source bundle, which the Go SDK would send with each event, and which Sentry would be able to use to associate the event with the correct source bundle. We tried using several different tools to see whether the binary had a debug ID or some other unique build ID, but we unfortunately could not find anything.

@cleptric, are you aware of whether the Go compiler is supposed to inject a debug ID or whether there is a way to enable such functionality? Perhaps we missed something. But if not, @loewenheim, @Swatinem, and I agree that the only way it would be possible for us to support Go bundles would be with some change in the Go SDK that allows us to somehow associate error events back to the correct source bundle.

[jerbob92]

@szokeasaurusrex is it always needed to match on a debug ID? If there is one source bundle that has the same release/dist, it can just use that to match right?

[szokeasaurusrex]

is it always needed to match on a debug ID? If there is one source bundle that has the same release/dist, it can just use that to match right?

@jerbob92, this functionality is indeed supported for JavaScript sourcemaps, but I am unsure whether we support such functionality for languages that get compiled into binaries (i.e. where we use debug files instead of sourcemaps).

In any case, we have moved away from the release/dist matching in JavaScript and now only maintain it as a legacy sourcemap uploading method, and we instead encourage people to use Debug IDs, since it provides a much more straightforward and less error-prone experience.

[jerbob92]

@szokeasaurusrex I understand. Perhaps it's possible to inject something into the ELF header of the binary and read it out in Go using debug/elf?

[szokeasaurusrex]

@jerbob92 How exactly would you envision something like this working? Is there any precedent in other tool that inject something into the ELF header?

[jerbob92]

@szokeasaurusrex I don't know, I just tried coming up with creative ideas to still get a debug ID in the binary. Do other compilers inject a debug ID in the symbol information?

[jerbob92]

@szokeasaurusrex Doesn't the Go compiler already inject a build ID? You can extract it using go tool buildid {binary}

[szokeasaurusrex]

Doesn't the Go compiler already inject a build ID? You can extract it using go tool buildid {binary}

I was unaware of this build ID since we were trying to use the DWARF debug ID (as we do for other compiled languages), which the Go compiler does not generate.

The Go build ID you are mentioning seems promising, but we can only use it if there is a way to extract the build ID from the binary at runtime from within the Go program, since the Sentry Go SDK would need to send the build ID with each event so we can use the correct debug information

[jerbob92]

I was unaware of this build ID since we were trying to use the DWARF debug ID (as we do for other compiled languages), which the Go compiler does not generate.

Can you tell me which debug ID you are talking about? Afaik there is no "DWARF debug ID". Only thing I can find is the .note.gnu.build-id in the ELF header that is often used, it's not much different from the .note.go.buildid in the ELF header in Go.

The code to read the build ID is here: https://github.com/golang/go/blob/master/src/cmd/internal/buildid/buildid.go But it's in an internal package and it needs the binary so I'm not if you can get it during runtime, I'll do some investigation.

[szokeasaurusrex]

Afaik there is no "DWARF debug ID".

This is precisely the problem here: Go's debug information (which is embedded in the executable) does not include a debug ID, unlike debug information files generated by other languages/compilers.

[jerbob92]

@szokeasaurusrex what I mean is that a standardized DWARF debug ID does not exist, or if it does, can you point me to anything to documents it?

[szokeasaurusrex]

@jerbob92 I am referring to the UUID that is output from the dwarfdump -uuid [FILE] command.

For example, if you compile a basic C program with GCC on Mac, and then pass the executable to the above command, you will obtain output that looks like so:

% dwarfdump -uuid a.out                           
UUID: D620F0EB-5208-3BC2-98CE-F86F293C4D17 (arm64) a.out

Rust binaries also contain a UUID like the one above.

However, if you pass a compiled Go binary instead, the command exits without any output, indicating that the Go compiler does not store this same UUID in the binary.

This UUID is what we use in Sentry to identify debug files. You can see that if we pass the same C binary from above to sentry-cli debug-files check, we will see the same UUID listed as the Debug ID:

% sentry-cli debug-files check a.out
Debug Info File Check
  Type: dsym executable
  Contained debug identifiers:
    > Debug ID: d620f0eb-5208-3bc2-98ce-f86f293c4d17
      Code ID:  d620f0eb52083bc298cef86f293c4d17
      Arch:     arm64
  Contained debug information:
    > symtab, unwind
  Usable: yes

If we instead pass a Go binary, sentry-cli cannot find this UUID, so it instead outputs zeroes as the Debug ID and tells us that the binary is unusable:

% sentry-cli debug-files check my_go_binary 
Debug Info File Check
  Type: dsym executable
  Contained debug identifiers:
    > Debug ID: 00000000-0000-0000-0000-000000000000
      Arch:     arm64
  Contained debug information:
    > symtab
  Usable: no (missing debug identifier, likely stripped)

[jerbob92]

Ah, thank you for that, that's something different than the .note.gnu.build-id from ELF it seems. And how do you normally get that UUID in Sentry during runtime when an error occurs?

[jerbob92]

@szokeasaurusrex I just tried it and for me sentry-cli debug-files check {go-binary} does result in a debug ID:

Debug Info File Check
  Type: elf executable
  Contained debug identifiers:
    > Debug ID: 8fb10288-482a-3585-ed4f-bbb6d89f9627
      Arch:     x86_64
  Contained debug information:
    > symtab, debug, unwind
  Usable: yes

This is on a Linux machine.

It just doesn't have the Code ID, which Symbolic only reads if it comes from .note.gnu.build-id or if it is of type NT_GNU_BUILD_ID: https://github.com/getsentry/symbolic/blob/229104525fd35499045ad085333906f7f4efc30d/symbolic-debuginfo/src/elf.rs#L663

So I think support for the Go build-id could be added to Symbolic by looking for .note.go.buildid (if it's even needed).

I don't know why your Go binary does not have a Debug ID (maybe it has to do with DWARF issues on OSX? https://github.com/golang/go/issues/62577 ).

Edit: since Go 1.22 it's also possible to get the GNU build-id in there, it's derived from the Go build-id, before it was already an option to set your own GNU build-id with the -B flag: go build -ldflags=-B=gobuildid

With that command I have a Debug ID and a Code ID.

[szokeasaurusrex]

@jerbob92, in that case, I am guessing this might be some kind of platform-dependent difference between Linux and Mac.

I also tried using go build -ldflags=-B=gobuildid, but on my Mac, this does not produce a file with a usable Debug ID (checked with both dwarfdump and sentry-cli debug-files check.

[jerbob92]

go build -ldflags=-B=gobuildid does not add the Build ID, it adds the Code ID. The Debug ID was already there for me on Linux.

[jerbob92]

@szokeasaurusrex how do you normally get this Debug ID in exceptions or during normal runtime? Or is that not necessary to make it work?

[szokeasaurusrex]

how do you normally get this Debug ID in exceptions or during normal runtime? Or is that not necessary to make it work?

@jerbob92, I am unsure – that is most likely an implementation detail of the SDK sending the event