ankane / tokenizers-ruby

Fast state-of-the-art tokenizers for Ruby
Apache License 2.0
132 stars 6 forks source link

Fix the default replacement character for the Metaspace encoder #18

Closed petergoldstein closed 1 year ago

petergoldstein commented 1 year ago

I used a "_" as opposed to the correct Unicode character when I set this originally.