Open lidavidm opened 1 month ago
I eventually bisected this to a change in the Go compiler: https://github.com/golang/go/commit/3560cf0afb
@zeroshade believes it may be mallocator, but a pure-Go reproducer does not exhibit any problems:
lidavidm@debian ~/C/a/g/adbc ((adfb70a2)) [1]> go test -v ./driver/flightsql/... -bench=. -run=nonexistent -benchtime 5s (adbc-go)
goos: linux
goarch: amd64
pkg: github.com/apache/arrow-adbc/go/adbc/driver/flightsql
cpu: Intel(R) Core(TM) i9-10885H CPU @ 2.40GHz
BenchmarkDefaultAllocator
BenchmarkDefaultAllocator-16 4518 1195807 ns/op
BenchmarkMallocator
BenchmarkMallocator-16 4952 1237689 ns/op
PASS
ok github.com/apache/arrow-adbc/go/adbc/driver/flightsql 16.428s
? github.com/apache/arrow-adbc/go/adbc/driver/flightsql/cmd/testserver [no test files]
So running things locally myself, I was using the GOTOOLCHAIN
env var in order to build using different versions of go. Using your example and just swapping out the binary shared object which is being loaded, I'm not seeing anywhere near the level of difference that you are seeing comparing go1.22.5 and go1.21.12:
go1.22.5:
Completed 100 iterations
Completed 200 iterations
Completed 300 iterations
Completed 400 iterations
Completed 500 iterations
Mean: 0.012383669883012772
Median: 0.011782394954934716
Sum: 6.191834941506386
go1.21.12:
Completed 100 iterations
Completed 200 iterations
Completed 300 iterations
Completed 400 iterations
Completed 500 iterations
Mean: 0.01193752330634743
Median: 0.011252027237787843
Sum: 5.968761653173715
And if I run the script multiple times, I end up with roughly the same variance between both cases. So i'm not quite sure what's going on and how you're reproducing it.
@lidavidm even more interesting is that the change your referring to was only introduced in go1.22.5 and was backported to go1.21.12. Which means both versions you pointed out in this issue have that change in them, so something doesn't add up here.
Again, look at the report...It's go1.21.7 for the "reference". I specifically was not comparing 1.21.12.
Strap in friends, I've narrowed down the cause and it's a doozy!
In case you don't want to follow along, here's the TL;DR:
https://github.com/golang/go/commit/3560cf0afb introduced a change into go1.22.5 (and was backported to go1.21.12) which forces Go always update the stack boundaries when making a cgo call. This has the effect of causing a significant performance degradation in the scenario of repeated calls to the ADBC driver (in the example @lidavidm provided above we go from ~2s to >~5.5s). For the time being this can be mitigated by building our distributable binaries with go1.22.4 or go1.21.11 which do not have this change. But going forward this is a problem. At the moment, the only likely solution I can think of is to file an issue with the golang/go repo and see if someone can fix this.
Now, if you want the gory details of this let's have some fun!
https://github.com/golang/go/commit/3560cf0afb was introduced to address the issue identified by https://github.com/golang/go/issues/62440 in which the issue was ultimately that in between calls to a Go callback from C, C was changing out the stack leading to a crash when the known bounds of a goroutine didn't have enough space on the stack left to operate. The solution ended up to always have the Goroutine update its stack boundaries before going on to perform a call from C.
So naturally, we assumed that the performance issue was due to the calls into Go from C being more expensive, but we didn't know why this was the case as getting some stack boundaries shouldn't be an expensive operation, right?
After getting the example working with pprof
, I did some digging to see if the issue could be in the upstream Arrow library (the cdata
package) or in how we package the drivers here. What stuck out to me was this:
Notice that it just says runtime._ExternalCode
which is calling libc.so.6
and taking up over 50% of the runtime. This was confusing because according to this, the slowdown is in libc
, not in the Go runtime? How could that be?
For comparison, if I compile with go1.22.4 instead, I get the following flamegraph:
Note how small the runtime._ExternalCode
entry there is.
Important note: I had to disable all our manual calls to runtime.GC()
in the driver.go
file otherwise all of the GC calls, mark and sweep, and all of that ends up dominating the profiles making it much more difficult to pin down the causes.
So I dd a bunch of research and found out that runtime._ExternalCode
is just what pprof
puts when it has no insight into the profiling. Which means it's native code (C or assembly) which pprof
can't report the profiling for. So now what? Well, there is the handy dandy perf
tool on linux!
After reading a bunch of tutorials and figuring out how it all works, I gave it a shot and got the following by using perf record -g -- python repro.py
...
When compiling with go1.22.4
I got the following results from perf
:
This breaks down to roughtly:
blas_thread_server
(i.e. numpy doing stuff! not sure why, but weird!)runtime.goexit.abi0
(grpc cleanup callbacks)When I compile with go1.22.5
I get this instead:
Which breaks down to:
blas_thread_server
(i.e. numpy. seriously, wtf??)runtime.goexit.abi0
(grpc cleanup)__isoc99_sscanf
This is where I had a huge WTF moment: how the hell did sscanf
get involved here? And why the hell is it taking up so much time?? After a few hours, I was finally able to figure it out.
I played around with the options in perf and googled a LOT and ultimately found that I could try changing the way it attempts to determine the call-graph and frames. I tried using --call-graph dwarf
but that failed horribly, as I expected, because we're not using DWARF executables. But when I changed it to lbr
I was able to get the following output:
In that screenshot you can see the full chain:
runtime.callbackUpdateSystemStack
pthread_getattr_np
__isoc99_sscanf
(which is just what GCC calls whenever you use sscanf
in C code).And the calls to sscanf
end up completely and utterly dominating the entire cost of the release callbacks, which is ludicrous and hilarious.
Ultimately, I think I'm gonna have to file an issue on golang/go repo to address this because I can't see any other way around it.
From a quick glance at glibc source it's because pthread_getattr_np is implemented by parsing /proc/self/maps
What happened?
There appears to be an incredible amount of FFI overhead when using a driver built by recent versions of Go.
Stack Trace
N/A
How can we reproduce the bug?
Environment/Setup
Linux x86_64
Conda package versions 1.0.0 (built with go1.21.7, IIRC) vs 1.1.0 (go 1.22.5)