-
### Self Checks
- [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones.
- [X] I confirm that I am using English to…
-
Chinese texts need a special kind of tokenization. Their texts cannot be simply split by whitespace or characters. It would be nice to add a separate module for segmenting Chinese texts.
Option 1: …
-
### Self Checks
- [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones.
- [X] I confirm that I am using English to su…
-
Chinese and Japanese, unlike English which relies on spaces for separation, use distinct punctuation marks such as full stops (。), exclamation marks (!), and question marks (?) to denote the end of se…
-
### Self Checks
- [X] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general).
- [X] I have s…
-
ICU is not a good choice in China. In addition, it is very important for Chinese word segmentation to customize the dictionary, because the application of words in different industries is completely d…
-
### Self Checks
- [X] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general).
- [X] I have s…
-
1. MNN版本:2.9.3
2. 编译与运行参考
https://mnn-docs.readthedocs.io/en/latest/transformers/diffusion.html
3. 编译MNN和MNN转换工具
cmake .. -DMNN_BUILD_DIFFUSION=ON -DMNN_BUILD_OPENCV=ON -DMNN_IMGCODECS=ON -DMNN_OP…
-
**Describe the bug**
The analyze module does not perform correct segmentation for Chinese texts. As Chinese does not have any white-space word segmentation, CATMA treats only punctuation symbols as w…
-
**Reported by vgjh2005 on 2014-04-15 10:16**
Hi
Please add Chinese words segmentation support. Separating each word in English is very simple. But Chinese is a very very complex language. It is very d…