issues
search
google
/
sentencepiece
Unsupervised text tokenizer for Neural Network-based text generation.
Apache License 2.0
10.03k
stars
1.16k
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
Bump the build-time-deps group in /.github/workflows/requirements with 2 updates
#1049
dependabot[bot]
opened
2 days ago
0
Bump the github-actions group with 2 updates
#1048
dependabot[bot]
opened
2 days ago
0
With unigram algorithm, constant piece at end of each sentences does not become a token
#1047
jogardi
opened
5 days ago
0
Error Attribute Error: type object 'SentencePieceTrainer' has no attribute 'train'. Did you mean: 'Train'?
#1046
bop578530
opened
1 week ago
0
builds for android devices
#1045
RaoufiTech
closed
1 week ago
0
decode token one by one
#1044
nigelzzz
closed
1 week ago
1
decode one by one can't show space
#1043
nigelzzz
closed
2 weeks ago
2
Why is the Hugging Face encoding 1 greater compared to the Google SentencePiece encoding when using the XLM-RoBERTa SentencePiece tokenizer?
#1042
RaoufiTech
closed
2 weeks ago
2
Bump the build-time-deps group across 1 directory with 3 updates
#1041
dependabot[bot]
closed
2 weeks ago
0
Bump the github-actions group with 3 updates
#1040
dependabot[bot]
closed
2 weeks ago
0
multi-thread batch encode seems slower than list comprehension
#1039
Mr-Grin
closed
2 weeks ago
1
Update setup.py
#1038
raushanksec
closed
1 month ago
1
Add support for windows arm64
#1037
Nagico2
closed
2 weeks ago
1
Bump the build-time-deps group across 1 directory with 4 updates
#1036
dependabot[bot]
closed
1 month ago
1
Bump the pip group in /.github/workflows/requirements with 2 updates
#1035
dependabot[bot]
closed
1 month ago
0
Bump certifi from 2023.11.17 to 2024.7.4 in /.github/workflows/requirements in the pip group
#1034
dependabot[bot]
closed
1 month ago
0
Bump the github-actions group across 1 directory with 6 updates
#1033
dependabot[bot]
closed
1 month ago
0
Bump the build-time-deps group in /.github/workflows/requirements with 4 updates
#1032
dependabot[bot]
closed
1 month ago
0
Zero Width Joiner issue for Sinhala Language
#1031
Nadil-K
opened
2 months ago
0
No typings in Python package
#1030
marcospgp
opened
2 months ago
0
When I set SPM_PROTOBUF_PROVIDER to "package" in CMakeLists.txt, the compilation fails.
#1029
hhxdestiny
opened
2 months ago
0
trainer_interface.cc: Integer value -1 is outside the valid range of values [0, 255] for the enumeration type 'ScriptType'
#1028
kcoul
opened
2 months ago
1
Bump urllib3 from 2.1.0 to 2.2.2 in /.github/workflows/requirements in the pip group
#1027
dependabot[bot]
closed
2 months ago
0
Error
#1026
silentghost1412
closed
2 months ago
1
install command line tools without sudo
#1025
zjesko
closed
2 months ago
1
Wrong calculation of max_score in unigram_model.cc
#1024
fairydreaming
opened
2 months ago
0
How to deal with id
#1023
980202006
opened
2 months ago
3
Parameterize lattice node allocator size to optimize chunk allocation performance
#1022
PriyankaRanganath
closed
2 months ago
3
How long does it take to train 31.2GB text data?
#1021
Mintchocolater
closed
3 months ago
1
Bump the build-time-deps group in /.github/workflows/requirements with 3 updates
#1020
dependabot[bot]
closed
3 months ago
0
Bump the github-actions group across 1 directory with 6 updates
#1019
dependabot[bot]
closed
2 months ago
1
resume/restart training of tokenizer
#1018
ganeshkrishnan1
closed
3 months ago
3
I want to obtain a model file using my vocab!
#1017
scj0709
closed
3 months ago
1
Convert SentencePiece .vocab format to OpenNMT-py .onmt_vocab format
#1016
HURIMOZ
closed
3 months ago
1
fixing minor typos in the API.md
#1015
Cassini-chris
closed
3 months ago
0
debloat the cmakelists.txt and add a bunch of customization for building
#1014
alexlnkp
opened
3 months ago
0
bump CMake minimum required version to avoid warnings
#1013
alexlnkp
closed
3 months ago
2
adding vocab_size consistency
#1012
Cassini-chris
closed
3 months ago
0
Bump requests from 2.31.0 to 2.32.0 in /.github/workflows/requirements in the pip group
#1011
dependabot[bot]
closed
3 months ago
0
Runtime error on iOS
#1010
l3utterfly
closed
2 weeks ago
11
Tokenization for phonetic languages
#1009
divyeshrajpura4114
closed
3 months ago
3
make it more friendly for mingw enviroments
#1008
Kreijstal
closed
3 months ago
1
Fix typo
#1007
xu-song
closed
3 months ago
0
Build sentencepiece with mingw
#1006
Kreijstal
closed
3 months ago
1
Fixing issues with the normalizer.cc (typo, type safety, cast fucn)
#1005
Cassini-chris
closed
4 months ago
0
Bump setuptools from 69.2.0 to 69.5.1 in /.github/workflows/requirements in the build-time-deps group
#1004
dependabot[bot]
closed
4 months ago
0
Bump the github-actions group with 6 updates
#1003
dependabot[bot]
closed
3 months ago
1
Add missing output formats to spm_encode flag documentation
#1002
mcognetta
closed
4 months ago
0
Tokenize at the word level without spacers nor joiners
#1001
HURIMOZ
closed
4 months ago
2
No make file found while build and install the Python wrapper
#1000
NickStrain
closed
4 months ago
2
Next