Issue with tabs and the Luau Lexer.

[rscwn]

It appears there was a change to the Lexer which made it so '\t' ( tabs ) are no longer handled in favor of 4 spaces, this creates issues with luau scripts that use tabs, leading to compiler errors like the following

:5: Expected identifier when parsing expression, got ' '

Or in my case crashes within my engine as I wasn't properly handling the compiler errors.

After some debugging I found the culprit for this is the function called isSpace located in Lexer.h:263

Here's the function from an older luau version for reference

inline bool isSpace(char ch)
{
    return ch == ' ' || ch == '\t' || ch == '\r' || ch == '\n' || ch == '\v' || ch == '\f';
}

And here it is in the current one

inline bool isSpace(char ch)
{
    return ch == ' ' || ch == '    ' || ch == '\r' || ch == '\n' || ch == '\v' || ch == '\f';
}

Fixing this is very simple, just revert to the older function.

Id also like to add, the change is useless, as in my research on the lexer, the 4 spaces are always handled by the first case ( ch == ' ' ), just separately char by char, and thus the second case ( ch == ' ' <- 4 spaces, just github wont properly show it. ) is never reached and cannot be.

[IAmTheTable]

Having this issue too!

[alexmccord]

This function has been untouched for two and a half years. Even then, the code existed before that in different places and implementation is byte for byte identical since day 1.

https://github.com/luau-lang/luau/blame/0d2688844ab285af1ef52f15878b57911c3cf056/Ast/include/Luau/Lexer.h#L275-L278.

I'm not sure I understand how ch == '\t' is being replaced with ch == ' ' but it looks like some buggy software on your end is making this replacement. ch == ' ' is not even valid C or C++ anyhow.