Tokenizer improvements

[jakobnissen]

Bugfixes and doc improvements to Tokenizer

Issue 125 revealed several issues with the existing tokenizer implementation, several of which are fixed here:

• tokenize now correctly uses the version argument
• Documentation is now clearer, and several tokenizer functions are doctested
• Several tokenizer tests were forgotten - these are now included and updated to work.

[codecov[bot]]

Codecov Report

Patch coverage: 92.30% and project coverage change: -0.06% :warning:

Comparison is base (733b8cf) 95.98% compared to head (989800b) 95.92%.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #126 +/- ## ========================================== - Coverage 95.98% 95.92% -0.06% ========================================== Files 16 16 Lines 1793 1793 ========================================== - Hits 1721 1720 -1 - Misses 72 73 +1 ``` | [Flag](https://app.codecov.io/gh/BioJulia/Automa.jl/pull/126/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BioJulia) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/BioJulia/Automa.jl/pull/126/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BioJulia) | `95.92% <92.30%> (-0.06%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BioJulia#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files Changed](https://app.codecov.io/gh/BioJulia/Automa.jl/pull/126?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BioJulia) | Coverage Δ | | |---|---|---| | [src/tokenizer.jl](https://app.codecov.io/gh/BioJulia/Automa.jl/pull/126?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BioJulia#diff-c3JjL3Rva2VuaXplci5qbA==) | `95.38% <92.30%> (-1.54%)` | :arrow_down: | ... and [1 file with indirect coverage changes](https://app.codecov.io/gh/BioJulia/Automa.jl/pull/126/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BioJulia)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[tlienart]

nice thanks!! could this be made into a patch release? 🙏

[jakobnissen]

Done