-
I encountered a UnicodeDecodeError while using a Korean tokenizer integrated into our data processing pipeline. This issue seems to occur specifically when processing certain types of input data with …
-
**Elasticsearch version**: `7.13.3` (tested on `7.15.1` too)
**Plugins installed**: [`repository-s3`, `analysis-nori`]
**JVM version** (`java -version`): `Eclipse Adoptium/OpenJDK 64-Bit Server…
-
```
2024-09-01 15:00:25.122 | INFO | datatrove.executor.local:run:120 - Skipping 4095 already completed tasks
2024-09-01 15:00:25.772 | INFO | datatrove.utils.logging:add_task_logger:58 - La…
-
Hi @afshinrahimi, @yuan-li, do you still keep the raw data (un-tokenized one) ? Also, which tokenizer did you use for this dataset ?
I need to work with the raw form of the text, and de-tokenizing …
-
Out of the box, Tantivy only support latin languages. We could add some extra tokenizers:
Chinese ([tantivy-jieba](https://crates.io/crates/tantivy-jieba) and [cang-jie](https://crates.io/crates/ca…
-
I want to change the tokenizer so that it can be applied to Korean
I would appreciate it if you could change LLM_PATH and additionally let me know which part of the code should be modified.
-
Hello,
Based on your code, I added Korean tokens (using a Korean emotional dataset) to the tokenizer and fine-tuned the model with the LibriTTS R dataset. The Korean dataset is slightly less than 3…
-
Copolot suggested this repository while adding additional tokens (James) to my tokenizer.
Here's my two cents:
I'm afraid to say that this is basically character-level encoding or the same as on…
-
Currently, the tokenizer is hard-coded to default, it would be better to include some configurable tokenizer for Chinese (tantivy-jieba and cang-jie), Japanese (lindera and tantivy-tokenizer-tiny-segm…
ghost updated
7 months ago
-
Korean language has specific characteristic. When developing search service with lucene & solr in korean, there are some problems in searching and indexing. The korean analyer solved the problems with…