Basic reader

[ilammy]

Implement basic reader that is able to scan a string and break it into tokens, producing a list of tokens and their locations as well as diagnostics.

Tracking productions specified by R7RS:

• [x] token
  • [x] identifier
    • [x] initial subsequent*
    • [x] | symbol-element* |
    • [x] peculiar-identifier
      • [x] explicit-sign
      • [x] explicit-sign sign-subsequent subsequent*
      • [x] explicit-sign . dot-subsequent subsequent*
      • [x] . dot-subsequent subsequent*
  • [x] boolean
  • [x] number
    • [x] prefix-R complex-R
      • [x] radix-R
      • [x] exactness
    • [x] real-R
      • [x] sign ureal-R
        • [x] uinteger-R
        • [x] uinteger-R / uinteger-R
        • [x] decimal-10
          • [x] uinteger-10 suffix
          • [x] digit-10+ . digit-10* suffix
          • [x] . digit-10+ suffix
      • [x] infnan
    • [x] real-R @ real-R
    • [x] real-R ( + | - ) real-R? i
    • [x] real-R infnan i
    • [x] ( + | - ) ureal-R? i
    • [x] infnan i
  • [x] character
    • [x] #\ any-character
    • [x] #\ character-name
    • [x] #\x Unicode-scalar
  • [x] string
    • [x] " string-element* "
    • [x] mnemonic-escape
    • [x] inline-hex-escape
    • [x] whitespace-escape
  • [x] ( | ) | #( | #u8( | ' | ` | , | ,@ | .
• [x] atmosphere
  • [x] comment
    • [x] ; line
    • [x] #| #| nested |# |#
    • [x] #; (s expr)
  • [x] directive
  • [x] whitespace
• [x] label

Yes, there are a lot of them. And this does not include recovery code paths.