If a C++ function that returns something other than a "primitive" type is wrapped using one of the WRAP macros or because it's been specified by structures as a callable vmethod or cmethod and then called via a function_identity type identity (that is, from Lua), a pointer to the return value is wrapped in a Lua metatable and pushed onto the Lua stack. The return value, which is local to the function_identity::call_and_push_impl function, is then destroyed when that function exits, which dangles the pointer in the Lua stack and results in undefined behavior should any attempt be made to use that object on the Lua stack.
I can think of two solutions:
change the templates for function_identity to constrain the RT type parameter to types that can be pushed directly onto the Lua stack; this would make wrapping a C++ function that returns a nonprimitive type a compile-time error.
add a test in call_and_push_impl to test if the return type is "primitive" and only push the value if it is; if it's not, push nil instead.
The type_identity virtual method is_primitive (as presently implemented) appears to me to correctly capture which types can be safely pushed onto the Lua stack in this situation.
I'm not sure that the first option can done without adding more information to the identity_traits type trait for types as I don't think we can use identity_traits<T>::get()->is_primitive() in a template constraint; in order to use it as a constraint it would have to be obtaining via a type-level expression (e.g. something like identity_traits<T>::is_primitive). The second option doesn't have this problem, but it results in the return value being discarded, which is obviously a less than satisfactory solution.
If a C++ function that returns something other than a "primitive" type is wrapped using one of the
WRAP
macros or because it's been specified by structures as a callable vmethod or cmethod and then called via afunction_identity
type identity (that is, from Lua), a pointer to the return value is wrapped in a Lua metatable and pushed onto the Lua stack. The return value, which is local to thefunction_identity::call_and_push_impl
function, is then destroyed when that function exits, which dangles the pointer in the Lua stack and results in undefined behavior should any attempt be made to use that object on the Lua stack.I can think of two solutions:
function_identity
to constrain theRT
type parameter to types that can be pushed directly onto the Lua stack; this would make wrapping a C++ function that returns a nonprimitive type a compile-time error.call_and_push_impl
to test if the return type is "primitive" and only push the value if it is; if it's not, push nil instead.The
type_identity
virtual methodis_primitive
(as presently implemented) appears to me to correctly capture which types can be safely pushed onto the Lua stack in this situation.I'm not sure that the first option can done without adding more information to the
identity_traits
type trait for types as I don't think we can useidentity_traits<T>::get()->is_primitive()
in a template constraint; in order to use it as a constraint it would have to be obtaining via a type-level expression (e.g. something likeidentity_traits<T>::is_primitive
). The second option doesn't have this problem, but it results in the return value being discarded, which is obviously a less than satisfactory solution.related discord discussion