feat (tokenization): included regular expression tokenization

coral-media / php-ml

Fresh approach to Machine Learning in PHP. Algorithms, Cross Validation, Neural Network, Preprocessing, Feature Extraction and much more in one library.

MIT License

1 stars 1 forks source link

feat (tokenization): included regular expression tokenization #15

Closed rernesto closed 3 months ago

rernesto commented 3 months ago

Added regular expression based tokenization. By default, supports UTF-8 and split in every non-alphabetic character. Class constructor supports a regular expression array with in place replacement, check default value.
Added corresponding test.

Refs: #14