-
Try
```python
>>> from wordcut import Wordcut
>>> wordcut = Wordcut.bigthai()
>>> print(wordcut.tokenize("จุ๋มสบายดีไหม"))
['จ', 'ุ๋ม', 'สบาย', 'ดี', 'ไหม']
```
-
So that http://standards.labs.theodi.org/search?q=comedy returns schema.org's mention of `ComedyEvent`: http://standards.labs.theodi.org/search?q=comedyevent
-
Hi urchade,
Thanks a lot for this great work. 😃
I'm trying to reproduce your results on CONLL04. But I found that the code reported an error:
Arguments: (AttributeError("'TransformerWordEmbedding…
-
### Steps to reproduce
1.3 occurrences in 3 agents at cycle 0: Java error: nil value detected
in Simulation 0
in list dates_prod_ut
-
The regex `"\"\"\".*\"\"\"` cannot match "\"\"\"abc\"\"\"" in the following code snippet, though Rust's innate regex does match:
```
use logos::Logos;
use regex::Regex;
#[derive(Logos, Debug, Pa…
-
Hello,
is it possible to change the tokenize dataSource url on a trigger?
-
is there is a way to tokenize a sentence taking into consideration phrasal-verbs.
example:
"The flight take off at three o'clock"
output should be:
[the, flight, take off, at, three, o'clock]…
-
a function specifically used to tokenize a given string. will be used for the getline tokenization and the path tokenization
-
### Self Checks
- [X] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general).
- [X] I have s…
-
### Description
When tokenizing a text and decoding these tokens, one can see that tokenization now (as of version 0.14.0) adds one additional starting space to `text` for every call of `Context.Toke…