Adding some additional lexer documentation

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (03c1044) 90.15% compared to head (ee34b30) 90.15%.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #5 +/- ## ======================================= Coverage 90.15% 90.15% ======================================= Files 155 155 Lines 27300 27300 ======================================= + Hits 24612 24613 +1 + Misses 2688 2687 -1 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[fosdickio]

This is odd. I ran cargo fmt -- and now it thinks code coverage has decreased. 😒

[nhatcher]

This is odd. I ran cargo fmt -- and now it thinks code coverage has decreased. 😒

You added more lines. what I don't understand is why the linting test was passing before