Closed bclarkson-code closed 2 months ago
The tokeniser should be extended to allow special characters (e.g pad tokens) and regex parsing (to e.g avoid merging across words)
The tokeniser should be extended to allow special characters (e.g pad tokens) and regex parsing (to e.g avoid merging across words)