cmd/compile: switch to a register-based calling convention for Go functions

[aclements]

I propose that we switch the Go internal ABI (used between Go functions) from stack-based to register-based argument and result passing for Go ~1.16~ 1.17.

I lay out the details of our proposal and the work required in this document.

The ABI specification can be found here.

This was previously proposed in #18597, where @dr2chase did some excellent prototyping work that our new proposal builds on. With multiple ABI support, we’re now in a much better position to execute on this without breaking compatibility with the existing body of Go assembly code. I’ve opened this new issue to focus on discussion of our new proposal.

/cc @dr2chase @danscales @thanm @cherrymui @mknyszek @prattmic @randall77

An incomplete and evolving list of tasks:

• [X] Define ABI specification (CL, @aclements)
• [X] Add GOEXPERIMENT=regabi and asm macro (CL)
• [x] cmd/compile: late call lowering (CLs)
• [x] cmd/compile: implement register arguments (@dr2chase)
  • [x] spill around morestack call (CL)
  • [x] add temporary ABI pragma for testing (CL)
  • [x] add abstract argument registers (WIP CL)
  • [x] implement ABI rules (CL)
  • [x] use abstract argument registers
• [x] cmd/compile: implement register returns
  • [x] late return lowering (CL, @dr2chase)
  • [x] implement ABI rules (CL, @thanm)
• [x] cmd/compile: ABI wrappers (CL, old CL)
  • [x] cmd/link: mangle wrapper names for ELF (CL)
  • [x] cmd/link: mangle wrapper names for PE (CL)
  • [x] implement zeroing of X15 in ABI0->ABIInternal wrappers (CL)
• [X] cmd/asm: add syntax for runtime packages to reference ABIInternal symbols (CL)
  • [x] Mark runtime.{goexit,asyncPreempt}, reflect.{makeFuncStub,methodValueCall} definitions ABIInternal (CL)
  • [x] Mark reference to runtime.callbackWrap in sys_windows_amd64.s as ABIInternal (CL)
• [x] runtime: fix assembly closure calls (mcall) (@aclements, CL)
• [x] reflect: support calling ABIInternal (CL)
• [x] reflect: support MakeFunc and method value calls (CL)
• [x] go and defer statements
  • [x] cmd/compile: desugar go/defer to zero-argument (@thanm, CL)
  • [x] cmd/compile: fix defer recover() (@cherrymui, CL)
  • [x] cmd/compile: desugar go/defer with results (@cherrymui, CL)
• [X] cmd/cgo: decouple cgo callbacks from ABI (CL)
• [x] runtime: finalizer support (CL)
• [x] runtime: Windows callback support (CL)
• [x] cmd/internal/obj: introduce and use RegEntryTmp registers in prologue (@mknyszek, CL)
• [x] cmd/compile: generate correct ABI0 argument maps for body-less functions (see WriteFuncMap) (@dr2chase, CL)
• [x] cmd/compile: make cgo_unsafe_args generate an ABI0 function (or an ABIInternal-with-no-registers function) (@cherrymui, CL)
• [x] runtime: update debug call protocol (CL)
• [x] Fix ABI0/ABIInternal confusion in cgo (CL)
• [x] Run signature fuzzer and fix bugs (everyone)
• [x] Get all.bash to pass on Linux, Windows, and Darwin (everyone)

High-priority non-critical path

• [x] Export ABI information across package boundaries to eliminate more wrappers (@aclements, CL)
• [x] Make funcPC always return the "native" PC of a function (maybe also introduce ABIOther) (@cherrymui, CL)
• [x] Strip unnecessary ABIInternal annotations once funcPC is in
• [x] Tracebacks
  • [x] Define extended argument metadata format (in progress, @cherrymui)
  • [x] cmd/compile: produce extended argument metadata (@cherrymui, CL)
  • [x] runtime: consume extended argument metadata (@cherrymui, CL)
  • [x] cmd/compile,runtime: traceback metadata for spill slot liveness
• [x] DWARF
  • [x] cmd/compile: ensure arguments have appropriate DWARF locations
  • [x] Ensure GDB and Delve can find register arguments
  • ~[ ] cmd/compile: add a DWARF vendor attribute with function argument frame size (context)~
• [x] Wrapper mangling for non-critical arches
  • [x] cmd/link: mangle wrapper names for Mach-O (CL)
  • [x] cmd/link: mangle wrapper names for xcoff
  • [x] cmd/link: mangle wrapper names for Plan 9
• [x] runtime: simplify go/defer call paths
  • [x] assert that go/defer frames are zero sized (CL)
  • [x] replace defer reflectcall with a direct closure call (CL)
• [x] cmd/internal/obj: reject splittable ABIInternal functions without morestack spill info (e.g., asm functions) because we can't generate a correct morestack path (CL)
• [x] cmd/asm: don't reference args_stackmap in ABIInternal functions (because it's for ABI0) (CL)
• [x] Port performance-critical assembly to ABIInternal (see CL 296372 for valuable functions) (CL 308931, CL 310184)
• [x] math: fix overhead of assembly->Go calls CL 310331
• [x] Release notes
  • Mention undocumented behavior we've observed applications depending on that may have changed.
  • Using reflect.ValueOf(fn).Pointer() to get the PC of an assembly function will now return the PC of the ABI wrapper
  • Performance surprise: calling from assembly to Go is slightly more expensive.
  • Performance surprise: calling an assembly function via a closure is slightly more expensive.
  • Traceback format is much improved

Enabling steps

• [x] Enable regabiwrappers by default on amd64 (Linux, Windows, and Darwin)
• [x] Enable regabidefer
• [x] Enable regabireflect
• [x] Enable regabig
• [x] Enable regabiargs

Testing

• [ ] Function signature fuzzer (ongoing)
  • [ ] Static call, closure call, method call, interface call, reflect.{ValueOf(target),MakeFunc(x, target),Method(x)}.{Call,Interface}, called from defer, called from go, called as a finalizer
  • [x] {Big,Small} {argument,result} in {memory,registers} and {not addressed,addressed and not leaked to heap,addressed and leaked to heap}
  • [x] Use defer in a test function (check arguments, modify results)
  • [x] Pass pointers-to-stack and pointers-to-heap
  • [x] Cause result to move to heap
  • [x] runtime: add MoveStackOnNextCall, assertions for pointer-to-stack/pointer-to-heap, assertions for live/dead (@aclements, CL)
  • [x] Integrate runtime testing hook
  • [ ] Clean up fuzzer, and get into tools repo.
• [x] cmd/compile: revive clobberdead (CL)
• [x] cmd/compile: add clobberdeadreg (CL)

Post-MVP

• [x] Move g from TLS to register (CL)
  • [x] Use g register on non-Linux OSes (see CL)
• [x] Reserve and use X15 as zero register (CL)
  • ~[ ] Should this be X0 for more compact instruction encoding?~
• [ ] Port to other platforms
  • [x] amd64
  • [x] arm64
  • [ ] mips
  • [x] ppc
  • [ ] s390x
  • [x] risc-v
  • ~[ ] porting guide~
• [x] Eliminate legacy ABI paths for all platforms, even if they're using ABIInternal with 0 registers
• [ ] Eliminate spill slots?
• [ ] Pass pointer-shaped method receiver in context register (#44827)
• [x] runtime: simplify go/defer call paths
  • [x] simplify go/defer paths to assume zero args/results
  • [x] simplify special _defer allocator and remove special _defer cases from mallocgc (because _defer records will be fixed size).
• [ ] cmd/vet: add syntax and checks for register arguments and results in assembly like x+0(FP)
• [ ] cmd/vet: check argument size for ABIInternal functions and fix runtime asm declarations

Cleanup (can be done later)

• [ ] runtime: port all assembly -> Go calls to ABIInternal?
• [x] Eliminate ABIAlias support from cmd/compile, cmd/link, and object format
• [x] remove now-redundant fields from ssa.AuxCall
• [x] rationalize use of LocalsOffset/FrameOffset (for registers that use a link register, e.g. arm64, powerppc).
• [ ] cmd/compile: revisit abiutils.go and clean up the API (also reduce ABIConfig copying)
• [ ] cmd/internal/obj: generate stack maps in obj so we can better compact the morestack/body maps and reduce subtlety?
• [x] cmd/compile,runtime: re-enable passing arguments to go in runtime (context)
• [ ] cmd/compile: always attach ir.Func to function ir.Names (context)
• [x] cmd/compile: support ABIInternal for tail calls (context)
• [x] cmd/compile: once wrapping of defers is on for all arch/OS combinations, it should be possible to simplify or remove some of the openDefer processing in ssagen (notablly we could get rid of openDeferSave()
• ~[ ] runtime: think about an ABI-insensitive debugCall~ Delve uses DWARF information
• [ ] Fix up named slots & value association
• [ ] fix firstpos in ssa.go – sometimes it's late.

[gopherbot]

Change https://golang.org/cl/308929 mentions this issue: cmd/compile/abi-internal: declare X15 scratch in function bodies

[gopherbot]

Change https://golang.org/cl/309009 mentions this issue: cmd/compile/abi-internal: declare R14 completely fixed

[gopherbot]

Change https://golang.org/cl/309034 mentions this issue: runtime: move zero-sized frame check from newproc to newproc1

[gopherbot]

Change https://golang.org/cl/309110 mentions this issue: cmd/compile: don't modify underlying type when creating bitmap for bodyless function

[gopherbot]

Change https://golang.org/cl/309169 mentions this issue: runtime: update debug call protocol for register ABI

[gopherbot]

Change https://golang.org/cl/309340 mentions this issue: cmd/link: build dynexp symbol list directly

[gopherbot]

Change https://golang.org/cl/309339 mentions this issue: cmd/link: move cgo export map from loadcgo to setCgoAttr

[gopherbot]

Change https://golang.org/cl/309341 mentions this issue: cmd/compile,cmd/link: resolve cgo symbols to the correct Go ABI

[gopherbot]

Change https://golang.org/cl/309338 mentions this issue: cmd/link: refactor setCgoAttr

[gopherbot]

Change https://golang.org/cl/309649 mentions this issue: cmd/compile: rescue stmt boundaries from OpArgXXXReg and OpSelectN.

[gopherbot]

Change https://golang.org/cl/309634 mentions this issue: runtime: eliminate externalthreadhandler

[gopherbot]

Change https://golang.org/cl/309789 mentions this issue: cmd/asm: require NOSPLIT for ABIInternal asm functions

[gopherbot]

Change https://golang.org/cl/309790 mentions this issue: cmd/internal/obj: don't emit args_stackmap for ABIInternal asm funcs

[gopherbot]

Change https://golang.org/cl/310171 mentions this issue: cmd/internal/objabi,test: use correct GOEXPERIMENT build tags in test/run.go

[gopherbot]

Change https://golang.org/cl/310172 mentions this issue: cmd/internal/objabi: enable regabiwrappers by default

[gopherbot]

Change https://golang.org/cl/310174 mentions this issue: cmd/internal/objabi: enable regabireflect by default

[gopherbot]

Change https://golang.org/cl/310175 mentions this issue: cmd/internal/objabi: enable regabidefer by default

[gopherbot]

Change https://golang.org/cl/310173 mentions this issue: cmd/internal/objabi: enable regabig by default

[gopherbot]

Change https://golang.org/cl/310176 mentions this issue: cmd/internal/objabi: enable regabiargs by default

[gopherbot]

Change https://golang.org/cl/310077 mentions this issue: runtime: fix data race in abi finalizer test

[gopherbot]

Change https://golang.org/cl/310184 mentions this issue: internal/bytealg: port more performance-critical functions to ABIInternal

[gopherbot]

Change https://golang.org/cl/310331 mentions this issue: math: avoid assembly stubs

[gopherbot]

Change https://golang.org/cl/310609 mentions this issue: cmd/internal/objabi: make regabi an alias for regabi sub-experiments

[gopherbot]

Change https://golang.org/cl/310649 mentions this issue: test/abi: reenable test on windows

[gopherbot]

Change https://golang.org/cl/310630 mentions this issue: runtime: fix race in TestFinalizerRegisterABI

[gopherbot]

Change https://golang.org/cl/310729 mentions this issue: cmd/link: fix file-local checks in xcoff

[gopherbot]

Change https://golang.org/cl/310733 mentions this issue: runtime: mark stdcallN functions cgo_unsafe_args

[gopherbot]

Change https://golang.org/cl/310690 mentions this issue: cmd/compile: spill all the parameters around morestack

[gopherbot]

Change https://golang.org/cl/310849 mentions this issue: internal/buildcfg: make regabi enable regabiargs

[aclements]

all.bash is now passing with all regabi experiments enabled on linux/amd64, darwin/amd64, and windows/amd64. We're testing on large code bases now. The results of that will ultimately determine whether we launch this in 1.17, but things appear to be in pretty good shape now. There's still some work to be done. In particular, we know we've regressed debug quality and need to address that. However, we believe it's solid enough to enable by default on these three platforms, and plan to trickle out the CLs to enable each piece over the next couple days, so that if it does cause issues on the dashboard we can better isolate which piece was responsible.

The results are looking phenomenal. On a set of application benchmarks we've been using, geomean improvement is 6%:

                                        base                     regabi
                                      sec/op         sec/op             vs base
BiogoIgor                             15.66 ± 1%     15.09 ± 1%   -3.65% (p=0.000 n=20)
BiogoKrishna                          17.95 ± 1%     17.52 ± 2%   -2.42% (p=0.001 n=20)
BleveIndexBatch100                    5.911 ± 1%     5.511 ± 2%   -6.76% (p=0.000 n=20)
BleveQuery                            6.481 ± 2%     5.546 ± 1%  -14.43% (p=0.000 n=20)
FoglemanFauxGLRenderRotateBoat        8.673 ± 0%     8.411 ± 0%   -3.02% (p=0.000 n=20)
FoglemanPathTraceRenderGopherIter1    20.05 ± 1%     20.28 ± 2%        ~ (p=0.052 n=20)
GopherLuaKNucleotide                  30.25 ± 1%     25.51 ± 1%  -15.67% (p=0.000 n=20)
MarkdownRenderXHTML                  246.8m ± 2%    240.6m ± 2%   -2.53% (p=0.001 n=20)
Tile38WithinCircle100kmRequest       804.8µ ± 2%    754.0µ ± 2%   -6.32% (p=0.000 n=20)
Tile38IntersectsCircle100kmRequest  1010.2µ ± 2%    908.7µ ± 2%  -10.04% (p=0.000 n=20)
Tile38KNearestLimit100Request        1.019m ± 2%    1.015m ± 1%        ~ (p=0.314 n=20)
geomean                              666.8m         627.0m        -5.98%

And the Bent benchmarks are 6.5% faster geomean, with many seeing much larger improvements. 3 of the 94 benchmarks see non-trivial slowdowns, which we're looking into, though we don't consider this blocking.

I'm going to be away for the next few months, so I'm super excited we were able to get to this point. Everyone who's been working on this has done a phenomenal job. These last few weeks in particular have been an amazing push for the finish line and I hope everyone gets some well-deserved time to relax soon.

[andig]

@aclements I've been looking for go version benchmarks- could you add a link to where/how you've found these? Are there any plans for adding the register ABI to darwin/arm64?

[aclements]

@andig, @mknyszek put together the benchmark suite. He's been working on open-sourcing it, but I'm not sure what the current status is.

Are there any plans for adding the register ABI to darwin/arm64?

Yes. ARM64 is the next architecture we plan to work on, and Darwin will definitely be included in that.

[gopherbot]

Change https://golang.org/cl/311689 mentions this issue: cmd/compile: preserve/translate names for parameters

[gopherbot]

Change https://golang.org/cl/312429 mentions this issue: runtime: fix runtimeQPC ABIInternal call

[gopherbot]

Change https://golang.org/cl/312650 mentions this issue: runtime: make nanotime1 ABIInternal on Windows

[gopherbot]

Change https://golang.org/cl/312869 mentions this issue: cmd/compile: fix bug in defer wrapping

[mvdan]

Do we have any numbers or guesses as to how this will end up affecting binary sizes? I originally imagined it wouldn't be a significant change either way, but I just noticed https://golang.org/cl/304470 talking about a 1.5% size increase for the sake of stack traces, and it got me thinking.

[gopherbot]

Change https://golang.org/cl/312989 mentions this issue: cmd/compile: spos handling fixes to improve prolog debuggability

[cherrymui]

Binary sizes are decreasing, even with that CL. @dr2chase may have better numbers.

[gopherbot]

Change https://golang.org/cl/313071 mentions this issue: dashboard: convert regabi builders to noregabi

[gopherbot]

Change https://golang.org/cl/313212 mentions this issue: cmd/compile: make the stack allocator more careful about register args.

[gopherbot]

Change https://golang.org/cl/314431 mentions this issue: cmd/compile: regabi support for DWARF location expressions

[gopherbot]

Change https://golang.org/cl/315071 mentions this issue: cmd/compile: handle field padding for register-passed structs

[gopherbot]

Change https://golang.org/cl/315390 mentions this issue: internal/buildcfg: enable regabi for Android

[gopherbot]

Change https://golang.org/cl/315610 mentions this issue: cmd/compile: fix abbrev selection for output params

[andig]

@andig, @mknyszek put together the benchmark suite. He's been working on open-sourcing it, but I'm not sure what the current status is.

@mknyszek I would really be interested in how Go measured performance evolved over time and what to expect of the next major version. It would be great if there was a benchmark corpus and maybe a repo/gist with collected data.

[mknyszek]

@andig, yeah, that's the plan going forward, but I haven't had the time yet. I'm hoping to do more around this in the second half of 2021.

FWIW, there is some work being done here with microbenchmarks, e.g. golang.org/cl/290431, but it will take time to get something complete up and running.

[mknyszek]

This work will continue into Go 1.18, but at this point I don't think any major new things will be happening here for the Go 1.17 release. Kicking it to the next milestone.

[DmitriyMV]

@mknyszek so does it mean that register-based calling convention is delayed until Go 1.18?

[thanm]

@mknyszek so does it mean that register-based calling convention is delayed until Go 1.18?

Go 1.17 will enable the register abi for GOARCH=amd64 systems. The 1.18 activity to which @mknyszek is referring is the work to enable the register abi for other architectures (for example, arm64). The new 1.18 milestone for this issue is for that work.