-
What if we set out a series of text texts in Chinese, Japanese, and Korean for testing by various packages. The challenge could be, as applicable:
* tokenise words and sentences
* tag parts of sp…
-
Hello, for llama when decoding Chinese or Japanese characters, since one character mgith need 2 or more tokens to decode, so when streaming, the chunk returned one token decode result is wrong,
![…
-
Anyone interested in creating data for chatbot-retrieval?
You can use the data in https://github.com/hunkimForks/DeepQA/tree/master/data/fbdata. We need to change the format of the data to feed the…
-
**Is your feature request related to a problem? Please describe.**
https://github.com/tantivy-search/tantivy#features
One of the features tantivy provides is to support custom tokenizers. For exampl…
dzcpy updated
3 years ago
-
First of all, thank you for creating and maintaining this incredible framework. It has been a valuable tool for our work.
I am currently attempting to fine-tune the Fast Conformer CTC model on a Ko…
-
```
model = AutoGPTQForCausalLM.from_quantized(
model_name,
#use_triton=True,
#warmup_triton=False,
trainable=True,
inject_fused_attention=False,
…
-
ReadClause and TranslateClause should be reorganised into:
- [ ] a tokenizer that splits the input into units, and identifies them as word, number, full stop, etc. as a token stream
- [ ] an SSML/HT…
-
**Is your feature request related to a problem? Please describe.**
[There](https://github.com/lazyloong/obsidian-fuzzy-chinese) is another plugin for obsidian which introduces fuzzy search of pin…
-
This issue is related to #9860.
Currently Nori and Kuromoji Analyzer use the same dictionary structure. (MeCab)
If we make combine DictionaryBuilder, we can reduce the code size.
But this task may …
-
CharFilters and TokenFilters may alter token lengths, meaning that subsequent filters cannot perform simple arithmetic to calculate the original ("correct") offset of a character in the interior of th…