New type solver refuses to accept string literal despite being one of the requested types

[Daw588]

type Form = "do-not-register" | (() -> ())

local function observer(register: () -> Form) end

observer(function()
    if math.random() > 0.5 then
        return "do-not-register" -- TypeError: Type 'string' could not be converted into '"do-not-register" | (() -> ())' Luau(1000)
    end
    return function() end
end)

Workaround:

type Form = "do-not-register" | (() -> ())

local function observer(register: () -> Form) end

observer(function(): Form
    if math.random() > 0.5 then
        return "do-not-register" -- OK
    end
    return function() end
end)

This does not happen to code like this:

local function min(): "str" | (() -> ())
    return "str"
end

[biseson]

TL;DR The newer type system expects the more precise literal type, but without an explicit type annotation, it defaults to string.

It's like that because, in Luau's new type solver, type inference for anonymous functions may not always correctly infer a specific string literal type when it appears in a union like "do-not-register" | (() -> ()), the type system infers "do-not-register" as a broader string type; not matching the expected literal type.

This likely occurs because the return type of the anonymous functions is initially inferred as a string, even though "do-not-register" is a string literal, that matches the expected "do-not-register".