issues
search
ankane
/
tokenizers-ruby
Fast state-of-the-art tokenizers for Ruby
Apache License 2.0
132
stars
6
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
Using gem on Alpine and ARM-based CPUs
#36
robinroestenburg
closed
2 months ago
2
Leverage new `rb-sys` features for simplified CI and cross-platform builds
#35
ianks
closed
6 months ago
1
Leverage `rb-sys` features for simplified CI and cross-platform builds
#34
ianks
closed
6 months ago
1
Add optional punctuation cleanup during decoding - clean_up_tokenization equivalent
#33
w-zygmuntowicz
closed
7 months ago
5
Clarify how to load tokenizers from files
#32
datasciencedavid
closed
10 months ago
1
Error when loading tokenizers gem
#31
crohr
closed
1 year ago
5
Issues when deploying to Ubuntu 20.04
#30
TheBrockEllis
closed
1 year ago
6
Add ByteFallback, Fuse, Replace, and Strip decoders. Added Prepend normalizer. Also added byte_fallback config option to BPE tokenizer.
#29
petergoldstein
closed
1 year ago
4
Cannot install tokenizer 0.3.2
#28
pribadi1st
closed
1 year ago
5
error when using in docker alpine
#27
haanhduclinh
closed
1 year ago
5
Ruby 2.7 didn't have JSON.load_file method
#26
elct9620
closed
1 year ago
1
Issue with CharBPETokenizer and pile_tokenizer.json
#25
max-fry-apps
closed
1 year ago
3
Fill out the set of getters and setters
#24
petergoldstein
closed
1 year ago
4
Add trainers
#23
petergoldstein
closed
1 year ago
0
Add processors
#22
petergoldstein
closed
1 year ago
1
Add models
#21
petergoldstein
closed
1 year ago
0
Add missing parameters to BpeTrainer
#20
petergoldstein
closed
1 year ago
0
Add Decoders
#19
petergoldstein
closed
1 year ago
0
Fix the default replacement character for the Metaspace encoder
#18
petergoldstein
closed
1 year ago
0
Add normalizers
#17
petergoldstein
closed
1 year ago
0
Add PreTokenizers
#16
petergoldstein
closed
1 year ago
5
Adds the num_special_tokens_to_add method to tokenizer
#15
petergoldstein
closed
1 year ago
0
Adds and completes a number of methods on the Tokenizer
#14
petergoldstein
closed
1 year ago
3
Add support for pretokenized arguments to encode/encode_batch
#13
petergoldstein
closed
1 year ago
1
Using tiktoken
#12
ScotterC
closed
1 year ago
1
Adds support for a pair argument. Also addresses multibyte offset issue.
#11
petergoldstein
closed
1 year ago
2
Unable to run Rails when gem is installed
#10
marckohlbrugge
closed
1 year ago
6
Adds a number of Python library equivalent functions to the Ruby interface
#9
petergoldstein
closed
1 year ago
1
More flexible handling of special tokens
#8
petergoldstein
closed
1 year ago
6
Adds word_ids to the Encoding interface so Ruby callers can access this value
#7
petergoldstein
closed
1 year ago
2
Version 0.2 relying on an old version of libssl
#6
kwi
closed
1 year ago
4
Support to Ruby 3.2.0 (release 0.2.1)
#5
vickymadrid03
closed
1 year ago
3
Fix (#3)
#4
kojix2
closed
2 years ago
0
I get an error when installing
#3
kojix2
closed
2 years ago
5
Error compiling
#2
ur5us
closed
2 years ago
1
Use `rake-compiler` and `rb-sys` for build
#1
ianks
closed
1 year ago
8