search

golang / go

The Go programming language
https://go.dev
BSD 3-Clause "New" or "Revised" License
122.83k stars 17.51k forks source link

x/build: collect core dumps from builders #49165

Open prattmic opened 2 years ago

prattmic commented 2 years ago

Sometimes tests dump core, perhaps with no extra crash information. Investigating these issues would be much improved if we could get the core file from the builder.

This came up recently in https://golang.org/cl/358675, where https://storage.googleapis.com/go-build-log/2fb3aaf5/linux-amd64_26aedb56.log failed with little more than:

signal: segmentation fault (core dumped)
FAIL    runtime 5.189s
FAIL

35248 is another case where post-submit failures could likely have been fixed without a reproducer if the core dump was available.

I believe there is an issue regarding collecting multiple files from builders generally which is likely a prerequisite for this, though I can't find it...

cc @mknyszek @golang/release

mknyszek commented 2 years ago

Looks like this needs:

  1. The ability to collect multiple files from a builder,
  2. setting a unique core pattern in run.bash,
  3. enabling core dumps in run.bash (currently disabled),
  4. running builders with GOTRACEBACK=crash instead of GOTRACEBACK=system.

Then the coordinator could, say, blindly search the work tree and /tmp for anything resembling the core dump pattern and then upload it to some GCS bucket with an expiry (so it doesn't grow infinitely large).

For most use-cases, it's probably enough to stop there. A stretch goal would be surfacing it in the UI somehow.

thanm commented 1 year ago

Another similar use case would be copying back profiles (coverage profiles, for example, or CPU profiles for PGO purposes).