search

ankane / tokenizers-ruby

Fast state-of-the-art tokenizers for Ruby
Apache License 2.0
132 stars 6 forks source link

Add processors #22

Closed petergoldstein closed 1 year ago

petergoldstein commented 1 year ago

Adds the processors except for the Sequence processor.

ankane commented 1 year ago

You're on fire today 🔥 🔥 🔥