-
Mistral and Llama2 models are working nicely, but I cannot get Yi models to work with tokenization.
I tried many repos with original Yi and also with finetunes.
Error I get:
```
raise TokenizerN…
-
### Describe the issue
Due to slow network, LLMLingua failed to download the model from huggingface during running.
```
requests.exceptions.ReadTimeout: (ReadTimeoutError("HTTPSConnectionPool(host…
-
**例行检查**
[//]: # (方框内删除已有的空格,填 x 号)
+ [x] 我已确认目前没有类似 issue
+ [x] 我已确认我已升级到最新版本
+ [x] 我已完整查看过项目 README,尤其是常见问题部分
+ [x] 我理解并愿意跟进此 issue,协助测试和提供反馈
+ [x] 我理解并认可上述内容,并理解项目维护者精力有限,**不遵循规则的 issue 可能…
-
Hi,
I tried to use this tool to convert a gltf, however, it throws this error:
`TypeError [ERR_ENCODING_INVALID_ENCODED_DATA]: The encoded data was not valid for encoding utf-8`
Any idea why this …
-
Nice project. I saw a note here:
https://github.com/smol-ai/developer/blob/03dc5d6d28d38fabf2276ab9a002b273529998b8/main.py#L10
You can get the total tokens for a request, then subtract it from …
-
I want to use torchchat to load the trained model directly from the local. How to change the torchchat/config/data/models.json? Need to change download _ and _ convert in download.py?And, what other d…
-
**例行检查**
[//]: # (方框内删除已有的空格,填 x 号)
+ [ x ] 我已确认目前没有类似 issue
+ [ x ] 我已确认我已升级到最新版本
+ [ x] 我已完整查看过项目 README,尤其是常见问题部分
+ [ x] 我理解并愿意跟进此 issue,协助测试和提供反馈
+ [ x] 我理解并认可上述内容,并理解项目维护者精力有限,**不遵循规则的 …
-
### Describe the bug
```
def count_tokens(text="", model="gpt-4"):
"""
Count the number of tokens in a string
"""
# Fix bug where models starting with openai/ for example can't…
-
## What would you like to be added:
It would be great to generate/load encoder from tokenizer.json file like
https://huggingface.co/CohereForAI/aya-101/resolve/main/tokenizer.json
or
https://…
-
### Question Validation
- [X] I have searched both the documentation and discord for an answer.
### Question
I recently upgraded llama_index to version 0.11.2.
I also changed from using Sentence…