Small tweaks to Tokenizer internals

[jakobnissen]

• Remove unused internal argument to function
• Do not force specialization on input tokens in code compilation

[codecov[bot]]

Codecov Report

Merging #90 (c800797) into master (46d6d33) will increase coverage by 0.24%. The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master      #90      +/-   ##
==========================================
+ Coverage   92.87%   93.12%   +0.24%     
==========================================
  Files          17       17              
  Lines        1614     1817     +203     
==========================================
+ Hits         1499     1692     +193     
- Misses        115      125      +10     

Flag	Coverage Δ
unittests	93.12% <100.00%> (+0.24%)	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
src/tokenizer.jl	91.76% <100.00%> (+0.62%)	:arrow_up:
src/re.jl	92.51% <0.00%> (-0.89%)	:arrow_down:
src/precond.jl	84.09% <0.00%> (-0.29%)	:arrow_down:
src/edge.jl	100.00% <0.00%> (ø)
src/sset.jl	100.00% <0.00%> (ø)
src/memory.jl	100.00% <0.00%> (ø)
src/traverser.jl	100.00% <0.00%> (ø)
src/nfa.jl	99.29% <0.00%> (+0.07%)	:arrow_up:
src/machine.jl	98.33% <0.00%> (+0.15%)	:arrow_up:
src/Stream.jl	93.93% <0.00%> (+0.18%)	:arrow_up:
... and 7 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 46d6d33...c800797. Read the comment docs.