-
### Question
I got loss to be 0 when training on Qwen2 backend,
{'loss': 0.0, 'learning_rate': 0.00015267175572519084, 'epoch': 0.0} …
-
-
data_process.py中,导入函数库时在下面这行:
**from chatglm_tokenizer.tokenization_chatglm import ChatGLMTokenizer**
出现以下错误:
======================
----> 5 from chatglm_tokenizer.tokenization_chatglm import Ch…
-
-
I am writing to ask for your help with a problem I am having with the tokenizer. I have been trying to solve it for a while now, but I have been unsuccessful.
However, I am having trouble with : Trac…
-
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1463, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.8/…
-
https://arxiv.org/pdf/1503.01655.pdf
In dictionary bulding section, I didn't find a description about how you deal with the multi-word expressions. How the tokenization and preprocessing of the …
-
Revisiting an old issue here: should `12 div-3` parse?
Under the new 4.0 tokenization rules, it certainly doesn't.
But under Michael Dyck's interpretation of the 3.1 rules, it does parse; and ac…
-
Add `.nlp.tokenize()` expression
-
Since the beginning, Sphinx and Manticore have not offered per-field tokenization settings (except for `morphology_skip_fields` and `infix/prefix_fields`), and it seems that there hasn't been much con…