Create Source Buffer and Lexer

[miguelcsx]

Description

This issue is focused on laying the foundation for the compiler by implementing a source buffer and a lexer. The source buffer will handle the input source code, and the lexer will tokenize the input into meaningful symbols for further processing by the parser.

Tasks

1. Source Buffer:

   • [x] Design the interface for the SourceBuffer class.
   • [x] Implement methods to load source code from a file or string.
   • [ ] Implement methods to manage and access the current position within the buffer.
   • [ ] Ensure efficient handling of character streams and lookahead.

2. Lexer:

   • [ ] Design the interface for the Lexer class.
   • [ ] Define token types and structure (tokens.hpp or equivalent).
   • [ ] Implement the tokenization logic:
     • [ ] Handle different types of tokens (keywords, identifiers, literals, operators, etc.).
     • [ ] Manage whitespace, comments, and special symbols.
   • [ ] Implement error handling and reporting for invalid tokens.
   • [ ] Create unit tests for the lexer to ensure accurate tokenization.

3. Documentation:

   • [ ] Write comprehensive comments for all public methods and classes.
   • [ ] Create a README or documentation file that explains the usage of the SourceBuffer and Lexer.

4. Code Quality:

   • [ ] Ensure that the code follows clean code principles (e.g., clear naming conventions, modularity).
   • [ ] Implement linting and code formatting tools in the CI/CD pipeline.
   • [ ] Perform code reviews before merging into the main branch.

Acceptance Criteria

• The SourceBuffer class is fully implemented and tested, capable of handling different input sources.
• The Lexer class can accurately tokenize input source code into defined tokens.
• All code is documented, follows clean code standards, and passes the CI/CD checks.
• Unit tests for the lexer are written and all tests pass successfully.

Related Issues

• This issue is a part of the compiler project roadmap.
• Future issues will build on the source buffer and lexer, such as the parser and AST generation.

Additional Notes

• Consider future scalability when designing the SourceBuffer and Lexer, as they will be foundational components for the entire compiler.

[miguelcsx]

For the first task Source Buffer, I used section 3.2 of the Dragon book, check it for more info, we could add a pair buffer, and make dynamic sentinels.