DFHack / dfhack

Memory hacking library for Dwarf Fortress and a set of tools that use it
Other
1.84k stars 463 forks source link

`WRAP`ped C++ functions that return "nonprimitive" types lead to dangling pointers on the Lua stack #4696

Closed ab9rf closed 3 weeks ago

ab9rf commented 3 weeks ago

If a C++ function that returns something other than a "primitive" type is wrapped using one of the WRAP macros or because it's been specified by structures as a callable vmethod or cmethod and then called via a function_identity type identity (that is, from Lua), a pointer to the return value is wrapped in a Lua metatable and pushed onto the Lua stack. The return value, which is local to the function_identity::call_and_push_impl function, is then destroyed when that function exits, which dangles the pointer in the Lua stack and results in undefined behavior should any attempt be made to use that object on the Lua stack.

I can think of two solutions:

The type_identity virtual method is_primitive (as presently implemented) appears to me to correctly capture which types can be safely pushed onto the Lua stack in this situation.

I'm not sure that the first option can done without adding more information to the identity_traits type trait for types as I don't think we can use identity_traits<T>::get()->is_primitive() in a template constraint; in order to use it as a constraint it would have to be obtaining via a type-level expression (e.g. something like identity_traits<T>::is_primitive). The second option doesn't have this problem, but it results in the return value being discarded, which is obviously a less than satisfactory solution.

related discord discussion

ab9rf commented 3 weeks ago

oh also need to make sure that functions returning void still work even though obviously void can't be pushed onto a lua stack :)