Add Error Handling to Lexer Component

Problem: Currently, the lexer component lacks proper error handling, which is crucial for robust compilation and helpful error reporting. Without adequate error handling, the compiler might produce incorrect results or fail silently when encountering invalid input.

Solution: To address this issue, we need to implement error handling in the lexer component. Error handling should cover scenarios such as encountering unrecognized characters, invalid tokens, and unexpected situations during tokenization.

Proposed Changes:

Error Tokens: Introduce a new token type, e.g., TokenType::ERROR, to represent errors encountered during tokenization.
Error Reporting: When the lexer encounters an error, it should generate an error token with an appropriate error message or code. This token can then be used by higher-level components for error reporting.
Error Recovery: Implement strategies for error recovery to gracefully handle errors without causing cascading issues. This might include skipping invalid input to continue tokenization or providing informative error messages.
Unit Tests: Create unit tests to ensure that error handling functions correctly and that error tokens are generated as expected in error scenarios.
Expected Outcome: Once error handling is implemented, the lexer component will be more robust and capable of providing meaningful feedback in case of errors. This improvement will contribute to a more reliable and user-friendly compiler.
Additional Information:

This issue is related to the lexer component and impacts the overall correctness and usability of the compiler. Error handling is a critical aspect of compiler development, and addressing this issue will be a significant step toward enhancing the project's quality.

diyorsattarov / 2

Add Error Handling to Lexer Component #3