42-Ikole-Systems / TMK-SH

An awesome POSIX compliant shell.
MIT License
0 stars 0 forks source link

Add reserved word tokens #24

Closed Tishj closed 1 year ago

Tishj commented 1 year ago

https://pubs.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html#tag_02_10_02

Listed in the grammar symbols list:

/ The following are the reserved words. /

%token If Then Else Elif Fi Do Done / 'if' 'then' 'else' 'elif' 'fi' 'do' 'done' /

%token Case Esac While Until For / 'case' 'esac' 'while' 'until' 'for' /

/ These are reserved words, not operator tokens, and are recognized when reserved words are recognized. /

%token Lbrace Rbrace Bang / '{' '}' '!' /

%token In / 'in' /

mraasvel commented 1 year ago

I'm not sure if these should be lexer tokens. Since they appear here: https://pubs.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html#tag_02_10_02 The way I interpret is that they are simply parsed as "Word" tokens, and recognised as reserved words by the parser when the appropriate rules are applied

These are reserved words, not operator tokens, and are recognized when reserved words are recognized

Tishj commented 1 year ago

I'm not sure if these should be lexer tokens. Since they appear here: https://pubs.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html#tag_02_10_02 The way I interpret is that they are simply parsed as "Word" tokens, and recognised as reserved words by the parser when the appropriate rules are applied

These are reserved words, not operator tokens, and are recognized when reserved words are recognized

If we care about these distinctions then we should not use Word in the Token::Type and instead use Token. Also the lexer produced tokens should likely be a subset of the Token class, so we dont end up with two Token classes that are almost identical copies of one-another

Tishj commented 1 year ago

I'm trying to think of a situation where it would be wrong to promote a token that directly matches a reserved word into a tokentype of that reserved word, but I can't think of any

mraasvel commented 1 year ago

Alright, then we will add the reserved words and also the Token/Word/Name/Assignment types, I will update my PR to parse into Token type instead of the Word type in that case.

mraasvel commented 1 year ago

I'm trying to think of a situation where it would be wrong to promote a token that directly matches a reserved word into a tokentype of that reserved word, but I can't think of any

echo if? In this case it's just an argument/word

Tishj commented 1 year ago

I'm trying to think of a situation where it would be wrong to promote a token that directly matches a reserved word into a tokentype of that reserved word, but I can't think of any

echo if? In this case it's just an argument/word

Ah that makes sense yea, hadnt thought of that