hplt-project sacremoses issues

hplt-project / sacremoses

Python port of Moses tokenizer, truecaser and normalizer

MIT License

486 stars 59 forks source link

issues

Newest

Newest Most commented Recently updated Oldest Least commented Least recently updated

Regular expression error of penn_tokenize

#151 speedcell4 opened 1 month ago
2
More robust testing for chaining sacremoses CLI

#150 alvations opened 2 months ago
0
2x faster MosesTokenizer

#149 de9uch1 opened 3 months ago
0
MosesDetokenizer performance

#148 jelmervdl opened 10 months ago
0
Can we update the pypi package? Last update was 2022

#147 heffernankevin closed 10 months ago
2
Parity with perl normalize.

#146 NIXBLACK11 closed 10 months ago
6
add citation

#145 faroit closed 11 months ago
5
Add tokenization for Tetun Dili (tdt)

#144 BLKSerene opened 11 months ago
4
Compile regexp in detokenizer

#143 jelmervdl closed 11 months ago
2
Github action to cache big.txt

#142 jelmervdl closed 12 months ago
0
Replace (defunct) Travis with Github Actions

#141 jelmervdl closed 12 months ago
1
Optimise regular expressions by precompiling them

#140 jelmervdl closed 12 months ago
4
Optimise `is_cjk(character)`

#139 jelmervdl closed 12 months ago
3
Web and basic protected patterns by default

#138 samirsalman opened 1 year ago
0
Quiet flag has no effect on the detokenizer

#137 XapaJIaMnu closed 10 months ago
1
[Question] Why is <unk> token tokenized into 3 items?

#136 MarkusSpanring closed 11 months ago
1
Is this package multi-threaded?

#135 ejkitchen closed 10 months ago
1
Clean up Python 2 compatibility-related codes

#134 BLKSerene closed 1 year ago
1
compile regex objects ahead of time for improved perf.

#133 erip closed 12 months ago
1
Add replacement for space before special symbols

#132 dimitrityan closed 2 years ago
0
Restrict click to be <8.1

#131 alecbrick closed 11 months ago
4
Error with CLI `tokenize` using `click==8.1.3`

#130 gsarti closed 2 years ago
10
New release?

#129 erip closed 2 years ago
4
Code cleanup

#128 BLKSerene closed 2 years ago
1
use setuptools instead of distutils

#127 alvations closed 2 years ago
0
distutils is deprecated in Python 3.10

#126 tirkarthi closed 2 years ago
1
Which of the 100 languages used in mBERT are not supported by this tokenizer?

#125 fake-warrior8 closed 2 years ago
1
Cache downloaded tokenizer files

#124 masus04 closed 2 years ago
1
Is there a way to use sacremoses in java?

#123 SefaZeng closed 2 years ago
1
Trying to get in touch regarding a security issue

#122 JamieSlome closed 2 years ago
2
Fixed protected patterns, truecase logic

#121 pluiez closed 10 months ago
0
Chinese full stop “。” can't be split.

#120 BrightXiaoHan opened 3 years ago
1
Add missing language for nonbreaking prefixes

#119 BLKSerene closed 3 years ago
0
deep detokenizer

#118 chessgecko opened 3 years ago
0
can't tokenise the period properly

#117 gdxie1 opened 3 years ago
0
No detokenize_penn?

#116 mfelice opened 3 years ago
0
update release on github

#115 PeterLijunfeng opened 3 years ago
0
Add tokenization for Tetun (tdt)

#114 raphaelmerx opened 3 years ago
0
Truecaser seems do not process sentences beginning with quotation marks

#113 SunshineBot opened 3 years ago
0
bug report

#112 Brucewuzhang opened 3 years ago
0
fix "int' + 'str' error, revert to what mosesdecoder does

#111 Brucewuzhang closed 10 months ago
2
Is this normal to tokenize "His number is No.123." to ['His', 'number', 'is', 'No.123.']. Should it be ['His', 'number', 'is', 'No.123', '.']？

#110 BrightXiaoHan closed 3 years ago
0
Bug: normalize prints extra newline

#109 mjpost closed 3 years ago
1
Fix detruecaser when the first token is all-caps

#108 yuyang-huang closed 3 years ago
2
error: loading model for detruecaser

#107 colingair opened 4 years ago
0
detrucase Error: unsupported operand type(s)

#106 colingair opened 4 years ago
0
truecase training killed: 9

#105 colingair opened 4 years ago
1
License Clarification

#104 TRQ3 closed 3 years ago
1
Improvements: virama and nukthas of Indic languages, easy way to specify basic protected patterns

#103 thammegowda closed 3 years ago
5
Possible to retrain/keep training an existing model?

#102 petulla opened 4 years ago
5