C++ Tokenizer implementation

[OlivierBinette]

Check if release PR fulfills these requirements

• [ ] Changelog has been updated and breaking changes have been marked (if any).
• [ ] Tests have been added if necessary.

Type of change

• [ ] Breaking changes.
• [ ] Backward compatible feature additions.
• [ ] Backward compatible bug fixes.

Additional comments and description of changes

[codecov-commenter]

Codecov Report

Merging #26 (0167206) into dev (7b9c58b) will decrease coverage by 4.26%. The diff coverage is 50.00%.

@@            Coverage Diff             @@
##              dev      #26      +/-   ##
==========================================
- Coverage   81.31%   77.04%   -4.27%     
==========================================
  Files          15       15              
  Lines         305      305              
==========================================
- Hits          248      235      -13     
- Misses         57       70      +13     

Flag	Coverage Δ
unittests	77.04% <50.00%> (-4.27%)	:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
stringcompare/distance/__init__.py	100.00% <ø> (ø)
stringcompare/preprocessing/tokenizer.py	0.00% <0.00%> (-61.91%)	:arrow_down:
stringcompare/preprocessing/__init__.py	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 7b9c58b...0167206. Read the comment docs.