search

ankane / tokenizers-ruby

Fast state-of-the-art tokenizers for Ruby
Apache License 2.0
132 stars 6 forks source link

Add models #21

Closed petergoldstein closed 1 year ago

petergoldstein commented 1 year ago

Fills in the missing models. Also adds additional configuration option support to BPE.