Tokenizer (Variables Expansion)

[edu-bm7]

Variable expansion now works both inside and outside double quotes ". Here_doc doesn't expand variables to mimic bash behaviour. Moving to make the wildcard * have its proper token and be left to be handle by the parser logic.

[edu-bm7]

Variable expansion is working flawlessly with support for ambiguous redirect, expansions inside and outside quotes with proper memory management. Still need to work into the wildcard * and add it to the available tokens.

[edu-bm7]

Variables expansion wasn't working as expected, I was doing the expansion in the middle of tokenization when in fact we need to first expand all variables, then cocatenate the result, to then tokenize everything. Expect some major breaking changes commits

[edu-bm7]

Variable expansion now works without any error, we should focus on tokenizing the wildcard * and decide if it'll be expanded in the tokenization phase or in the parser phase

[edu-bm7]

Wildcards * should work for the current working directory

This is the statment on the subject of our project, so the wildcard needs to know the state of the program, we should expand it on the parser phase then, so when we do cd /test && ls a* the ls is perfomed inside /test. I'll proceed to add this token and close this issue once is done

[edu-bm7]

This is the wildcard behaviour, which will be done in the parser phase, if it founds a match it prints it, if doesnt it prints the string per se

[edu-bm7]

So in order for the Wildcard * work properly and know which token is a string literal and which one isn't, we must go back and add back the quotes from the input into our tokens. E.g. echo "foo", will be of TOKEN_STRING("foo") instead of TOKEN_STRING(foo). this will prevent us for missreading at the parser phase inputs like: echo foo*"baz*"*bar*.o, where baz* needs to be taken literal in our wildcard search and not as only baz.

We'll also, remove the quotes from literal tokens at the parser phase, after wildcards expansions, if any.

[edu-bm7]

After further thinking, this approach might not be the best one, because we can have quotes as our output, it wont be easy to differentiate which one is from string literals and which one isn't.

We must think of a way to tell our tokens that some parts are string literals and not wildcard patterns. E.g. echo a*"b*"*m b* must be seen as a single token and not a b with wildcard.

We could start use the TOKEN_SPACE( ) to denote all spaces in the output to be ignored or handled by the parser, and the concatenation of string literals be done by splitting into different tokens.

[edu-bm7]

Wildcard is now working as expected, further testing is necessary. The way is working is that the widlcard * has it own token and any token surrounding it (with the exception of space, redirections and command) will be concatenated and used as pattern.

One problem though, the TOKEN_EXIT_CODE isn't been properly tokenized, we should rework its functionality. Gonna focus on the exit code tomorrow morning, if it's fixed by noon I'll close this issue .

[edu-bm7]

Variable Expansions now work flawlessly in the tokenizer part 7153906