DeedleFake
commented
3 years ago From what I can tell from the documentation for the new cgo.Handle, it's intended only for a situation where a pointer needs to be passed from Go to C and then back to Go without the C code doing anything with what it points to. As it passes a handle ID, not a real pointer, the C code can't actually get access to the actual data. Maybe a function could be provided on the C side that takes a handle ID and returns the original pointer, thus allowing the C code to access the data? Would that solve this issue?
Edit: Wait, that doesn't make sense. Could you just use Handle to make sure that it's held onto? Could the definition of Handle be extended to mean that the pointer itself is valid for the duration of the Handle's existence? In other words, this would be defined to be valid:
// void doSomethingWithAPointer(int *a);
import "C"
func main() {
v := C.int(3)
h := cgo.NewHandle(&v)
doSomethingWithAPointer(&v) // Safe because the handle exists for that pointer.
h.Delete()
}Alternatively, if that's not feasible, what about a method on Handle that returns a valid pointer for the given value?
// Pointer returns a C pointer that points to the underlying value of the handle
// and is valid for the life of the handle.
func (h Handle) Pointer() C.uintptr_tDisclaimer: I'm not familiar enough with the internals of either the Go garbage collector or Cgo to know if either of these even make sense.
ansiwen
ianlancetaylor
bcmills
phlogistonjohn
Merovius
beoran
hnes
rsc
egonelbre
prattmic
randall77
balasanjay
aclements
eliasnaur
cherrymui
dot-asm
Update, 2021-10-20: the latest proposal is the API in https://github.com/golang/go/issues/46787#issuecomment-942547949.
Problem Statement
The pointer passing rules state:
and
There are C APIs, most notably the
iovecbased ones for vectored I/O which expect an array of structs that describe buffers to read to or write from. The naive approach would be to allocate both the array and the buffers withC.malloc()and then either work on the C buffers directly or copy the content to Go buffers. In the case of Go bindings for a C API, which is assumably the most common use case for Cgo, the users of the bindings shouldn't have to deal with C types, which means that all data has to be copied into Go allocated buffers. This of course impairs the performance, especially for larger buffers. Therefore it would be desirable to have a safe possibility to let the C API write directly into the Go buffers. This, however, is not possible becauseObviously, what is missing is a safe way to pin an arbitrary number of Go pointers in order to store them in C memory or in passed-to-C Go memory for the duration of a C call.
Workarounds
Break the rules and store the Go pointer in C memory
(click)
with something like ```go IovecCPtr.iov_base = unsafe.Pointer(myGoPtr) ``` but `GODEBUG=cgocheck=2` would catch that. However, you can circumvent cgocheck=2 with this casting trick: ```go *(*uintptr)(unsafe.Pointer(&IovecCPtr.iov_base)) = uintptr(myGoPtr) ``` This might work, as long as the GC is not moving the pointers, which might be a fact as of now, but is not guaranteed.Break the rules and hide the Go pointer in Go memory
(click)
with something like ```go type iovecT struct { iov_base uintptr iov_len C.size_t } iovec := make([]iovecT, numberOfBuffers) for i := range iovec { bufferPtr := unsafe.Pointer(&bufferArray[i][0]) iovec[i].iov_base = uintptr(bufferPtr) iovec[i].iov_len = C.size_t(len(bufferArray[i])) } n := C.my_iovec_read((*C.struct_iovec)(unsafe.Pointer(&iovec[0])), C.int(numberOfBuffers)) ``` Again: This might work, as long as the GC is not moving the pointers. `GODEBUG=cgocheck=2` wouldn't complain about this.Break the rules and temporarily disable
cgocheck(click)
If hiding the Go pointer as a uintptr like in the last workaround is not possible, passing Go memory that contains Go pointers usually bails out because of the default `cgocheck=1` setting. It is possible to disable temporarily `cgocheck` during a C call, which can especially useful, when the pointer have been "pinned" with one of the later workarounds. For example the `_cgoCheckPtr()` function, that is used in the generated Cgo code, can be shadowed in the local scope, which disables the check for the following C calls in the scope: ```go func ... { _cgoCheckPointer := func(interface{}, interface{}) {} C.my_c_function(x, y) } ``` Maybe slightly more robust, is to export the runtime.dbgvars list: ```go type dbgVar struct { name string value *int32 } //go:linkname dbgvars runtime.dbgvars var dbgvars []dbgVar var cgocheck = func() *int32 { for i := range dbgvars { if dbgvars[i].name == "cgocheck" { return dbgvars[i].value } } panic("Couln't find cgocheck debug variable") }() func ... { before := *cgocheck *cgocheck = 0 C.my_c_function(x, y) *cgocheck = before } ```Use a C function to store the Go pointer in C memory
(click)
The rules allow that a C function stores a Go pointer in C memory for the duration of the call. So, for each Go pointer a C function can be called in a Go routine, that stores the Go pointer in C memory and then calls a Go function callback that waits for a release signal. After the release signal is received, the Go callback returns to the C function, the C function clears the C memory from the Go pointer, and returns as well, finishing the Go routine. This approach fully complies with the rules, but is quite expensive, because each Go routine that calls a C function creates a new thread, that means one thread per stored Go pointer.Use the
//go:uintptrescapescompiler directive(click)
`//go:uintptrescapes` is a compiler directive that > specifies that the function's uintptr arguments may be pointer values that have been converted to uintptr and must be treated as such by the garbage collector. So, similar to the workaround before, a Go function with this directive can be called in a Go routine, which simply waits for a release signal. When the signal is received, the function returns and sets the pointer free. This seems already almost like a proper solution, so that I implemented a package with this approach, that allows to `Pin()` a Go pointer and `Poke()` it into C memory: [PtrGuard](https://github.com/ansiwen/ptrguard) But there are still caveats. The compiler and the runtime (cgocheck=2) don't seem to know about which pointers are protected by the directive, because they still don't allow to pass Go memory containing these Go pointers to a C function, or to store the pointers in C memory. Therefore the two first workarounds are additionally necessary. Also there is the small overhead for the Go routine and the release signalling.Proposal
It would make Cgo a lot more usable for C APIs with more complex pointer handling like
iovec, if there would be a programmatic way to provide what//go:uintptrescapesprovides already through the backdoor. There should be a possibility to pin an arbitrary amount of Go pointers in the current scope, so that they are allowed to be stored in C memory or be contained in Go memory that is passed to a C function within this scope, for example with aruntime.PtrEscapes()function. It's cumbersome, that it's required to abuse Go routines, channels and casting tricks in order provide bindings to such C APIs. As long as the Go GC is not moving pointers, it could be a trivial implementation, but it would encapsulate this knowledge and would give users a guarantee.I know from the other issues and discussions around this topic that it's seen as dangerous if it is possible to pin an arbitrary amount of pointers. But
//go:uintptrescapesfunctions, therefore it is also possible to pin arbitrary amount of Go pointers already.Related issues: #32115, #40431
/cc @ianlancetaylor @rsc @seebs
edit: the first workaround had an incorrect statement. edit 2: add workarounds for disabling cgocheck