Closed JianLoong closed 1 year ago
Currently, the way the string input is tokenized is just using a simple split method.
Perhaps using a better tokenize library would give better results. For example here
Can I work on this?
Currently, the way the string input is tokenized is just using a simple split method.
Perhaps using a better tokenize library would give better results. For example here