Parsing `#include <...>` is not right

[n0tknowing]

Case 1

#include <foo//bar.h>

It's error with message expected '>', but the expected is either the file is opened or error because the file doesn't exist (or whatever the value inside errno as long as it's related to opening a file).

Case 2

#include < stdio.h >

It's success where the expected is failed because both leading and trailing whitespaces is removed. Even if it's success, it should open a file named " stdio.h " not "stdio.h".

Case 3

#include <file with two  spaces.h>

It's expected to open a file named "file with two spaces.h" not "file with two spaces.h"

Why it happened

• Tokenizer tokenizes the entire source file before the preprocessor, thus it lose some information.
• In join_tokens(), multiple occurence of whitespaces is replaced by a single space (ASCII: 0x20).

The fix

• For case 2 and case 3, it can be fixed by modify function join_tokens().
• But for case 1, rewriting the tokenizer is the solution. Instead of run the tokenizer as separate pass, it should run along with the preprocessor, tokenize lazily, and return linked list of token that's later used by the parser.

[n0tknowing]

If the tokenizer is rewritten, the project will be renamed.

[n0tknowing]

The tokenizer has been rewritten a week ago, and currently working on integrating it with the preprocessor rest of code