Checking user data type in modules?

[UltraEngine]

I'm attempting to integrate modules with sol. My first test case is a module that modifies a window's style. The module works fine if I treated the HWND as a number (Win64 still uses 32-bit window handles) but I am having trouble storing it as userdata, which will be better for cross-platform compatibility, since Linux and Mac use pointers for window handles, which cannot fit in a Lua number.

In the main program:

struct LuaWindowHandle {};

L->new_usertype<LuaWindowHandle>("HWND");

L->new_usertype<Window>(
    "WindowClass",
    "GetHandle", [](Window& w) { return (LuaWindowHandle*)w.GetHandle(); },// Uses a pointer, which will work for Linux and Mac
    //"GetHandle", [](Window& w) {return uint32_t(w.GetHandle()); },// This works for win32 only
}

Window::GetHandle returns a win32 HWND, which I cast to a LuaWindowHandle pointer so that sol recognizes the type.

In my Lua module, I have this code:

static int SetWindowZenMode(lua_State* L)
{
    HWND hwnd = (HWND)luaL_checkudata(L, 1, "HWND");
    //HWND hwnd = (HWND)luaL_checkinteger(L, 1);// This works for win32 only
    int state = luaL_checkint(L, 2);

luaL_checkudata is producing a type error. If I call luaL_typename(L, 1) it returns a big integer. I don't know what that number is, but it isn't the HWND.

Can you tell what I am doing wrong?

[UltraEngine]

This is what the metatable looks like. I tried "UltraEngine::Window::BindClass::LuaWindowHandle" and "UltraEngine::Window::BindClass::LuaWindowHandle*" for the name, to no avail. https://imgur.com/a/qelNT1w

[Rochet2]

Let me try to confirm my understanding. In the example code you try to cast the return value of GetHandle to an arbitrary struct pointer to make the code work on all platforms. However, the code does not work.

The problem seems to be at least the way you cast from userdata into HWND. Also the metatable name may or may not be correct. The pointer returned by checkudata function is not the pointer you return from GetHandle. It is a pointer to the userdata that contains the pointer. So you should be using something on the lines of

LuaWindowHandle** udataptr = (LuaWindowHandle**)luaL_checkudata(L, 1, "sol.LuaWindowHandle*");
LuaWindowHandle* udata = *udataptr;
HWND hwnd = (HWND)udata;

Here is a small code I whipped up to try show how it could probably work. https://godbolt.org/z/3YhvPr3rn

```cpp #define SOL_ALL_SAFETIES_ON 1 #include #include // std::uintptr_t, uint32_t, uint64_t #include // std::numeric_limits #include // std::unordered_map #include // std::cout, std::endl // Try changing this // typedef uint32_t HWND; typedef uint64_t HWND; // Mock window struct Window { HWND GetHandle() const { return value; } HWND value = std::numeric_limits::max(); int state = 0; }; std::unordered_map windows; // The helper struct to try ensure type safety in some way I guess struct LuaWindowHandle { LuaWindowHandle() = delete; ~LuaWindowHandle() = delete; }; // Your module static int SetWindowZenMode(lua_State* L) { // Get with metatable name (without sol) // Note that we need to dereference the void pointer before we cast it, since luaL_checkudata returns a pointer to the userdata // and the userdata contains the actual pointer that we returned to lua. // You may also want to take a look at this https://sol2.readthedocs.io/en/latest/api/usertype_memory.html HWND hwnd1 = static_cast(reinterpret_cast(*(LuaWindowHandle**)luaL_checkudata(L, 1, "sol.LuaWindowHandle*"))); // Get using state view (sol) if you have access to the LuaWindowHandle // Not sure if I should use sol::stack::get or sol::stack::get_usertype, but both probably work // The former does not work in modules I guess. // https://sol2.readthedocs.io/en/latest/api/stack.html?highlight=stack#stack-get-usertype HWND hwnd2 = static_cast(reinterpret_cast(sol::stack::get_usertype(L, 1))); // Some test prints std::cout << hwnd1 << std::endl; std::cout << hwnd2 << std::endl; int state = luaL_checkinteger(L, 2); // Just a mock to see that it works windows[hwnd1]->state = state; return 0; } // Your program int main() { // Set up lua sol::state lua; lua.open_libraries(); // Set up functions and methods lua["SetWindowZenMode"] = SetWindowZenMode; lua.new_usertype("HWND"); auto ut = lua.new_usertype("WindowClass"); ut["GetHandle"] = [](Window& w) { return (LuaWindowHandle*)w.GetHandle(); }; // Test the code Window w; windows[w.value] = &w; std::cout << w.value << std::endl; lua["w"] = &w; lua.script("print('metatables:', w, w:GetHandle())"); // You can easily see the metatable names by printing the values std::cout << "state before: " << w.state << std::endl; lua.script("SetWindowZenMode(w:GetHandle(), 42)"); std::cout << "state after: " << w.state << std::endl; return 0; } ```

[Rochet2]

You may also be interested in lightuserdata https://sol2.readthedocs.io/en/latest/api/user.html?highlight=light https://www.lua.org/pil/28.5.html

They are just void* pointers that you can pass around through lua. But they provide no typesafety or metatables and thus no methods either.

Seems void pointer is automatically handled as lightuserdata by sol. So you can just cast to it and it should work. Here is an example code with the lightuserdata https://godbolt.org/z/s7fzKPMxM

```cpp #define SOL_ALL_SAFETIES_ON 1 #include #include // std::uintptr_t, uint32_t, uint64_t #include // std::numeric_limits #include // std::unordered_map #include // std::cout, std::endl // Try changing this // typedef uint32_t HWND; typedef uint64_t HWND; // Mock window struct Window { HWND GetHandle() const { return value; } HWND value = std::numeric_limits::max(); int state = 0; }; std::unordered_map windows; // Your module static int SetWindowZenMode(lua_State* L) { if (!lua_islightuserdata (L, 1)) { return luaL_argerror(L, 1, "Expected HWND"); } HWND hwnd1 = static_cast(reinterpret_cast(lua_touserdata(L, 1))); // Some test prints std::cout << hwnd1 << std::endl; int state = luaL_checkinteger(L, 2); // Just a mock to see that it works windows[hwnd1]->state = state; return 0; } // Your program int main() { // Set up lua sol::state lua; lua.open_libraries(); // Set up functions and methods lua["SetWindowZenMode"] = SetWindowZenMode; auto ut = lua.new_usertype("WindowClass"); ut["GetHandle"] = [](Window& w) { return (void*)w.GetHandle(); }; // Test the code Window w; windows[w.value] = &w; std::cout << w.value << std::endl; lua["w"] = &w; lua.script("print('metatables:', w, w:GetHandle())"); // You can easily see the metatable names by printing the values std::cout << "state before: " << w.state << std::endl; lua.script("SetWindowZenMode(w:GetHandle(), 42)"); std::cout << "state after: " << w.state << std::endl; return 0; } ```

[Rochet2]

Since you are making a module, which I guess is a DLL or SO or similar, take a look at this: integrating into existing code https://sol2.readthedocs.io/en/latest/tutorial/existing.html

[UltraEngine]

It did not occur to me that sol might actually be able to handle void*. I am using this now and it works great.

I don't expect I will actually need to expose any C++ classes to a Lua module. I can't think of any reason it would be necessary.

Thanks for your help.