search

rth / vtext

Simple NLP in Rust with Python bindings
Apache License 2.0
147 stars 11 forks source link

PY Implement get_params methods for tokenizers #60

Closed rth closed 4 years ago

rth commented 4 years ago

This implements the get_params methods for all tokenizers, and also makes them all inherit from the base class vtext.tokenize.BaseTokenizer.

The rust crate now optionally depends on pyo3 and dict_derive.