issues
search
hplt-project
/
sacremoses
Python port of Moses tokenizer, truecaser and normalizer
MIT License
486
stars
59
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
Regular expression error of penn_tokenize
#151
speedcell4
opened
1 month ago
2
More robust testing for chaining sacremoses CLI
#150
alvations
opened
2 months ago
0
2x faster MosesTokenizer
#149
de9uch1
opened
3 months ago
0
MosesDetokenizer performance
#148
jelmervdl
opened
10 months ago
0
Can we update the pypi package? Last update was 2022
#147
heffernankevin
closed
10 months ago
2
Parity with perl normalize.
#146
NIXBLACK11
closed
10 months ago
6
add citation
#145
faroit
closed
11 months ago
5
Add tokenization for Tetun Dili (tdt)
#144
BLKSerene
opened
11 months ago
4
Compile regexp in detokenizer
#143
jelmervdl
closed
11 months ago
2
Github action to cache big.txt
#142
jelmervdl
closed
12 months ago
0
Replace (defunct) Travis with Github Actions
#141
jelmervdl
closed
12 months ago
1
Optimise regular expressions by precompiling them
#140
jelmervdl
closed
12 months ago
4
Optimise `is_cjk(character)`
#139
jelmervdl
closed
12 months ago
3
Web and basic protected patterns by default
#138
samirsalman
opened
1 year ago
0
Quiet flag has no effect on the detokenizer
#137
XapaJIaMnu
closed
10 months ago
1
[Question] Why is <unk> token tokenized into 3 items?
#136
MarkusSpanring
closed
11 months ago
1
Is this package multi-threaded?
#135
ejkitchen
closed
10 months ago
1
Clean up Python 2 compatibility-related codes
#134
BLKSerene
closed
1 year ago
1
compile regex objects ahead of time for improved perf.
#133
erip
closed
12 months ago
1
Add replacement for space before special symbols
#132
dimitrityan
closed
2 years ago
0
Restrict click to be <8.1
#131
alecbrick
closed
11 months ago
4
Error with CLI `tokenize` using `click==8.1.3`
#130
gsarti
closed
2 years ago
10
New release?
#129
erip
closed
2 years ago
4
Code cleanup
#128
BLKSerene
closed
2 years ago
1
use setuptools instead of distutils
#127
alvations
closed
2 years ago
0
distutils is deprecated in Python 3.10
#126
tirkarthi
closed
2 years ago
1
Which of the 100 languages used in mBERT are not supported by this tokenizer?
#125
fake-warrior8
closed
2 years ago
1
Cache downloaded tokenizer files
#124
masus04
closed
2 years ago
1
Is there a way to use sacremoses in java?
#123
SefaZeng
closed
2 years ago
1
Trying to get in touch regarding a security issue
#122
JamieSlome
closed
2 years ago
2
Fixed protected patterns, truecase logic
#121
pluiez
closed
10 months ago
0
Chinese full stop “。” can't be split.
#120
BrightXiaoHan
opened
3 years ago
1
Add missing language for nonbreaking prefixes
#119
BLKSerene
closed
3 years ago
0
deep detokenizer
#118
chessgecko
opened
3 years ago
0
can't tokenise the period properly
#117
gdxie1
opened
3 years ago
0
No detokenize_penn?
#116
mfelice
opened
3 years ago
0
update release on github
#115
PeterLijunfeng
opened
3 years ago
0
Add tokenization for Tetun (tdt)
#114
raphaelmerx
opened
3 years ago
0
Truecaser seems do not process sentences beginning with quotation marks
#113
SunshineBot
opened
3 years ago
0
bug report
#112
Brucewuzhang
opened
3 years ago
0
fix "int' + 'str' error, revert to what mosesdecoder does
#111
Brucewuzhang
closed
10 months ago
2
Is this normal to tokenize "His number is No.123." to ['His', 'number', 'is', 'No.123.']. Should it be ['His', 'number', 'is', 'No.123', '.']?
#110
BrightXiaoHan
closed
3 years ago
0
Bug: normalize prints extra newline
#109
mjpost
closed
3 years ago
1
Fix detruecaser when the first token is all-caps
#108
yuyang-huang
closed
3 years ago
2
error: loading model for detruecaser
#107
colingair
opened
4 years ago
0
detrucase Error: unsupported operand type(s)
#106
colingair
opened
4 years ago
0
truecase training killed: 9
#105
colingair
opened
4 years ago
1
License Clarification
#104
TRQ3
closed
3 years ago
1
Improvements: virama and nukthas of Indic languages, easy way to specify basic protected patterns
#103
thammegowda
closed
3 years ago
5
Possible to retrain/keep training an existing model?
#102
petulla
opened
4 years ago
5
Next