-
I changed the `llm_model_path` to 'yentinglin/Llama-3-Taiwan-8B-Instruct'. Then the bug happened. It seems that the Llama-3-Taiwan-8B-Instruct tokenizer.json does not contain "". GFD is based on "byte…
-
var chatMessages= new ChatMessage[] {
ChatMessage.CreateUserMessage(cont)
}
**int tokenCount=_tokenizer (chatMessages)_; //**
ChatCompletion completion = await _sdk.GetChatClie…
-
### System Info / 系統信息
docker
### Who can help? / 谁可以帮助到您?
_No response_
### Information / 问题信息
- [ ] The official example scripts / 官方的示例脚本
- [ ] My own modified scripts / 我自己修改的脚本和任务
### Rep…
-
> Please provide us with the following information:
> ---------------------------------------------------------------
### This issue is for a: (mark with an `x`)
```
- [x] bug report -> please…
-
目前的tokenizer都与之前的不一样了(vocab里缺少了id 3-13, 新增了许多added_tokens),是有什么特别理由吗?
例如:
https://huggingface.co/01-ai/Yi-1.5-34B-Chat/blob/main/tokenizer.json
https://huggingface.co/01-ai/Yi-1.5-34B-32K/blob/ma…
-
Hi, I want to ask, what are the values of self.v_token_id = 15167, self.q_token_id = 16492, self.a_token_id = 22550, self.nl_id = 13 in tokenizer set based on? Or why is the value of v_token_id set …
-
Hi there, nice work on the internVL! We're really impressed by the new internvl-v1.5.
One thing we noticed is that the backing language model internlm/internlm2-chat-20b has a fast tokenizer (https…
-
### Feature [request]
Recently, multimodal large models based on the Transformer architecture have emerged one after another. Can text-generate-inference provide some support? For example, a feasib…
-
Hi, I'm confused about where to find the tokenizer:
--tokenizer_path checkpoints/lit-llama/tokenizer.model
Referring here to the readme:
![image](https://github.com/Lightning-AI/lit-llama/ass…
-
### Describe the bug
I'm trying to port an AllenNLP model to a framework that's still maintained so am considering `flair`. My original model is a character LSTM based tagger. It's character based …