Match statement is not recognized

[Thirumalai-Shaktivel]

Example:

def _get_slots(cls):
    match cls.__dict__.get('__slots__'):
        case None:
            return
        case str(slot):
            yield slot
        # Slots may be any iterable, but we cannot handle an iterator
        # because it will already be (partially) consumed.
        case iterable if not hasattr(iterable, '__next__'):
            yield from iterable
        case _:
            raise TypeError(f"Slots of '{cls.__name__}' cannot be determined")

Error:

syntax error: Token 'cls' (of type 'identifier') is unexpected here
    --> ./Lib/dataclasses.py:1130:11
     |
1130 |     match cls.__dict__.get('__slots__'):
     |           ^^^

[Thirumalai-Shaktivel]

Here, match and case are soft keywords: https://docs.python.org/3/reference/compound_stmts.html#the-match-statement

The match and case keywords are proposed to be soft keywords, so that they are recognized as keywords at the beginning of a match statement or case block respectively, but are allowed to be used in other places as variable or argument names.

https://peps.python.org/pep-0622/#the-match-statement @certik how do we handle this?

[certik]

So match is a keyword that is not reserved. See https://docs.python.org/3/reference/lexical_analysis.html#soft-keywords, they call it a soft keyword. In our tokenizer.re, we have the following sets of reserved keywords:

"as"      
"assert"  
"async"   
"await"   
"break"   
"class"   
"continue"
"def"     
"del"     
"elif"    
"else"    
"except"  
"finally" 
"for"     
"from"    
"global"  
"if"      
"import"  
"in"      
"is"      
"lambda"  
"None"    
"nonlocal"
"pass"    
"raise"   
"return"  
"try"     
"while"   
"with"    
"yield"   
"yield from"

These cannot be used as variable names such as in with = 5.

match, case are different (perhaps also _ as the docs say, not sure about how we handle that --- looks like _ can also be a variable name). We have to allow them as variable names.

[certik]

We have several options how to tackle the match and case non-reserved keywords (each has pros/cons):

• we can write a "strong tokenizer" like the fixed-form LFortran tokenizer, which is a recursive descent parser that recognizes the context that we are in, and based on the context it will know if match and case are keywords, or just a name (such as a variable).
• we can enable Bison GLR, just like we use in the free-form, so the tokenizer always returns match as a keyword, and in Bison we add an id rule that is a "TK_NAME | KW_MATCH | KW_CASE", this creates conflicts, but Bison GLR can take care of it
• we have some special handling of match/case in the tokenizer by looking ahead, perhaps till the end of the line (but much simpler than the first option; similar to how we handle do loops and format in free-form parser in LFortran) and return KW_MATCH, otherwise return TK_NAME("match").

[certik]

We should probably fix this one first, now when the new parser is merged.

[Thirumalai-Shaktivel]

Remaining issues:

match():
   # Comment
    case():
        x = 0

match ():     # Comment
    case ():
        pass
    case ['remove',
              *files]:
        print('Removing files: {}'.format(files))

match = {2:
    3}
match[2:
    8]