ADL Customization breaks with `std::vector` and usertype fields

[GameBoyNikolai]

I'm encountering an issue in Sol v3.3.0 relating to the new (to me) ADL Customization Endpoints used in combination with collections and usertypes.

Below is an example with a simple struct vec2 exposed to Lua with the Customization Endpoints. Creating a usertype which exposes two functions, one that accepts a std:vector<> and one that returns a std::vector<> of the custom type. After finding #1531 I was able to make this half of the issue work by defining two additional endpoints for sol::nested<vec2>, but this seems like a bug rather than the intended usage.

The stranger issue is when attempting to expose the custom type via a property declared in the usertype. Attempting to access this field throws the error sol: runtime error: [string \"...\"]:15: attempt to index a sol.glm::vec<2,float,0>* value (field 'field').

I'm assuming all of these issues would disappear if I promoted vec2 to a usertype, but this would be fairly disruptive to the rest of my code. Its especially strange because each of these cases worked in the previous version of Sol I had been using (v2.20.6) with the equivalent template-specialization Customization Endpoints.

Environment details:

• Sol v3.3.0
• Lua 5.3.6
• MSVC 2022 compiling C++17

// Struct and ADL Customization setup
struct vec2 {
    float x, y;
};

// Block A
namespace sol {
template <>
struct lua_type_of<vec2>
    : std::integral_constant<sol::type, sol::type::table> {};
}
//

template <typename Handler>
bool sol_lua_check(sol::types<vec2>, lua_State* state, int index, Handler&& handler, sol::stack::record& tracking) {
    int absolute_index = lua_absindex(state, index);
    if (sol::stack::check<sol::table>(state, absolute_index, handler)) {
        tracking.use(1);
        return true;
    }

    return false;
}

vec2 sol_lua_get(sol::types<vec2>, lua_State* state, int index, sol::stack::record& tracking) {
    int absolute_index = lua_absindex(state, index);
    sol::table point = sol::stack::get<sol::table>(state, absolute_index);

    vec2 ret{ 0, 0, };

    ret.x = point[1];
    ret.y = point[2];

    tracking.use(1);

    return ret;
}

int sol_lua_push(lua_State* state, const vec2& val) {
    return sol::stack::push(state, sol::table::create_with(state,
                                                    1, val.x,
                                                    2, val.y));
}

// Block B
template <typename Handler>
bool sol_lua_check(sol::types<sol::nested<vec2>>,
                   lua_State* L,
                   int index,
                   Handler&& handler,
                   sol::stack::record& tracking) {
    return sol_lua_check<Handler>(sol::types<vec2>{}, L, index, handler, tracking);
}

vec2 sol_lua_get(sol::types<sol::nested<vec2>>,
                 lua_State* L,
                 int index,
                 sol::stack::record& tracking) {
    return sol_lua_get(sol::types<vec2>{}, L, index, tracking);
}
//

// Register usertype and use both objects
int main(int argc, char* argv[])
{
    sol::state state;
    state.open_libraries(sol::lib::base,
                         sol::lib::io,
                         sol::lib::string,
                         sol::lib::table,
                         sol::lib::debug
                         );

struct DoThings {
        DoThings() : field(34.345, 983.234) {
            std::cout << "ctor called" << std::endl;
        }

        void accept_single(vec2 p) {
            std::cout << p.x << ", " << p.y << std::endl;
        }

        void accept(std::vector<vec2> points) {
            for (auto& p : points) {
                std::cout << p.x << ", " << p.y << std::endl;
            }
        }

        vec2 generate_single() {
            return { 1.f, 2.f };
        }

        std::vector<vec2> generate() {
            std::vector<vec2> points;
            points.push_back({ 1.f, 2.f });
            points.push_back({ 2.f, 3.f });
            points.push_back({ 3.f, 4.f });

            return points;
        }

        glm::vec2 field;
    };

    state.new_usertype<DoThings>("DoThings", 
        "field", &DoThings::field, 
        "accept_single", &DoThings::accept_single, 
        "accept", &DoThings::accept, 
        "generate", &DoThings::generate, 
        "generate_single", &DoThings::generate_single);

    state.script(R"lua(
        a = DoThings.new()

        -- works when following the ADL Endpoints example in the docs
        a:accept_single({3, 4.2})
        p = a:generate_single()
        print(p[1], p[2])

        -- works with Block B implemented
        a:accept({ {3, 4.2}, {3434.0, 23498} })
        points = a:generate()
        for _, p in ipairs(points) do
            print(p)
            -- oddly this only works with Block A implemented and crashes otherwise; not mentioned in the docs AFAIK
            print(p[1], p[2])
        end

        -- currently crashes :(
        print(a.field[1], a.field[2])
    )lua");
}