Open jeanromainroy opened 5 months ago
However, testing with progressively longer prompts reveals it begins producing nonsensical output (e.g., "
...") after 8192 tokens, aligning with the "max_position_embeddings" value in the config.json file
🤔 not sure what would cause that. Do you have a prompt that should work in the MLX version that doesn't? Also if you are able to provide some expected output that would also be helpful.
The config also lists a "rope_theta" value, suggesting its role in achieving the large context window. Is "rope" supported in MLX?
MLX has RoPE and it should be used correctly already.
I'm getting random Cyrillic in my responses when using tokenizer.apply_tool_use_template. Anyone else? Seems to only be when using that tool template from the tokenizer.
Example output:
Write 'Action:' followed by a json-formatted list of actions that you want to perform in order to produce a good response to the user's last input. You can use any of the supplied tools any number of times, but you should aim to execute the minimum number of necessary actions for the input. You should use the directly-answer
tool if calling the other tools is unnecessary. The list of actions you want to call should be formatted as a list of json objects, for example:
[
{
"tool_name": title of the tool in the specification,
"parameters": a dict of parameters to input into the tool as they are defined in the specs, or {} if it takes no parameters
}
]```<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>
Action: ```json
[
{
"tool некоторыми": {},
"tool_name": "internet_search"
} forniscono]
```<EOS_TOKEN>
ignore, I was calling the tokenizer twice. fixed it in my code here for anyone who wants to test tool use (apologies in advance if there are bugs still lurking): https://github.com/fblissjr/mlx-funbox
looks like there's still random switching to multilingual and random Cyrillic (using a simple generate + apply tool template). has anyone tested on CUDA to see if similar?
Looks like it's the tokenizer.json that is not converting correctly. See the tokenizer.json from the Cohere HF model repo:
Compared to a freshly converted mlx_lm.convert -q (no other params) I just did from that same repo 20 minutes ago, which also matches the tokenizer.json from the mlx-community quant uploaded earlier (mlx-community/c4ai-command-r-plus-4bit):
Copying the original cohere tokenizer.json (https://huggingface.co/CohereForAI/c4ai-command-r-plus/blob/main/tokenizer.json) fixes this issue completely from my testing (output generation is slow, but so far so good!)
My guess is something is happening in the mlx_lm.convert process due to the large size of the vocab + the multilingual nature of the tokenizer + the strange tokenizer.json formatting.
edit: generation speed is also slightly faster now due to the correct tokenizer being used.
That is very odd. The tokenizer copying is very simple in MLX LM. We basically load with Hugging Face and then save it with Hugging Face. There is no MLX code involved. https://github.com/ml-explore/mlx-examples/blob/main/llms/mlx_lm/utils.py#L619
I wonder if we are somehow using the API incorrectly or maybe there is a bug in the way it's saved with Transformers.
@fblissjr you can reproduce the behavior with:
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("CohereForAI/c4ai-command-r-plus")
tokenizer.save_pretrained(".")
I feel that should not break the tokenizer.. so it might be worth filing an issue with the Cohere HF repo or the Transformer repos? Wdyt?
@awni my guess is the latter. looks more like it's saved incorrectly (and oddly just by looking at it) in the hf repo. Haven't seen a tokenizer.json like this before. here's a quick sample of 1 page on more tokenizer.json
from https://huggingface.co/CohereForAI/c4ai-command-r-plus/blob/main/tokenizer.json
{"version": "1.0", "truncation": null, "padding": null, "added_tokens": [{"id": 0, "content": "
@fblissjr you can reproduce the behavior with:
from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("CohereForAI/c4ai-command-r-plus") tokenizer.save_pretrained(".")
I feel that should not break the tokenizer.. so it might be worth filing an issue with the Cohere HF repo or the Transformer repos? Wdyt?
agreed. i made a community post on hf here: https://huggingface.co/CohereForAI/c4ai-command-r-plus/discussions/15
and here: https://github.com/huggingface/transformers/pull/30027
so this is interesting - the tokenizer.json on the bitsandbytes repo linked from the main cohere repo is a different size, and looks nothing like the original. https://huggingface.co/CohereForAI/c4ai-command-r-plus-4bit/blob/main/tokenizer.json
another interesting difference between the 4 bit bnb tokenizer and the original - in the original one, token id token id 255001 <|END_OF_TURN_TOKEN|>, special is set to False. In the 4bit bnb one, it's True.
Per comments on the hugging face repo, the differences between the two tokenizers.json files are unicode differences. I'll assume I've got something bugging on my end unless anyone else sees the same.
# Libraries
from transformers import AutoTokenizer
import mlx.core as mx
import mlx_lm
from mlx_lm.utils import load_model, get_model_path
# Language Model
PATH_MODEL = "/Users/admin/Models/CohereForAI/c4ai-command-r-plus-4bit/"
# Load the model & tokenizer
tokenizer = AutoTokenizer.from_pretrained(PATH_MODEL)
model = load_model(get_model_path(PATH_MODEL))
# Incrementally longer texts
...
text_7500_tokens = "Lorem ipsum dolor sit..." # Works
text_8500_tokens = "Lorem ipsum dolor sit..." # Stops working
...
# Format as list of messages
messages = [
{"role": "user", "content": f"{text_8500_tokens}\n\nSummarize the text above in one short paragraph."} # <-- set a text
]
# Apply chat template
prompt_decorated = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True
)
# Generate
response = mlx_lm.generate(
model,
tokenizer,
prompt_decorated,
temp=0.0,
max_tokens=64
)
This is what I have been using – I removed the texts which are just some random wikipedia page. The output is good until I try the 8500 tokens text which just outputs \<PAD>\<PAD>\<PAD>\<PAD>\<PAD>...
# Libraries from transformers import AutoTokenizer import mlx.core as mx import mlx_lm from mlx_lm.utils import load_model, get_model_path # Language Model PATH_MODEL = "/Users/admin/Models/CohereForAI/c4ai-command-r-plus-4bit/" # Load the model & tokenizer tokenizer = AutoTokenizer.from_pretrained(PATH_MODEL) model = load_model(get_model_path(PATH_MODEL)) # Incrementally longer texts ... text_7500_tokens = "Lorem ipsum dolor sit..." # Works text_8500_tokens = "Lorem ipsum dolor sit..." # Stops working ... # Format as list of messages messages = [ {"role": "user", "content": f"{text_8500_tokens}\n\nSummarize the text above in one short paragraph."} # <-- set a text ] # Apply chat template prompt_decorated = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True ) # Generate response = mlx_lm.generate( model, tokenizer, prompt_decorated, temp=0.0, max_tokens=64 )
This is what I have been using – I removed the texts which are just some random wikipedia page. The output is good until I try the 8500 tokens text which just outputs
...
Have you tried with apply_tool_template by chance? Curious if you see any of the oddities I see when using it.
Hey guys @awni, @fblissjr and @jeanromainroy,
The cohere team limited the context to 8k for all Command-R variants on purpose. If you check the config file for both r-v01 and r+, the max_position_embeddings is set to 8192.
It's a limit to avoid users experiencing OOM.
You can read more here: https://huggingface.co/CohereForAI/c4ai-command-r-v01/discussions/12
Hey @Blaizzy, I have run the exact same test with the new llama.cpp implementation of Command-R+ and it works way above 8k tokens.
Copying the original cohere tokenizer.json (https://huggingface.co/CohereForAI/c4ai-command-r-plus/blob/main/tokenizer.json) fixes this issue completely from my testing (output generation is slow, but so far so good!)
My guess is something is happening in the mlx_lm.convert process due to the large size of the vocab + the multilingual nature of the tokenizer + the strange tokenizer.json formatting.
edit: generation speed is also slightly faster now due to the correct tokenizer being used.
@fblissjr Indeed the tokenizer created from the conversion is slightly smaller ~2MB than the original.
I updated as you suggested. Can you check it?
Hey @Blaizzy, I have run the exact same test with the new llama.cpp implementation of Command-R+ and it works way above 8k tokens.
@jeanromainroy can you try again with the change in this branch, if it works I will make a PR.
pip install -U git+https://github.com/Blaizzy/mlx-examples.git@pc/commandR#subdirectory=llms --use-pep517
Link: https://github.com/Blaizzy/mlx-examples/tree/pc/commandR
You can also try to increase the default max_position_embeddings
and let me know if it works.
Copying the original cohere tokenizer.json (https://huggingface.co/CohereForAI/c4ai-command-r-plus/blob/main/tokenizer.json) fixes this issue completely from my testing (output generation is slow, but so far so good!) My guess is something is happening in the mlx_lm.convert process due to the large size of the vocab + the multilingual nature of the tokenizer + the strange tokenizer.json formatting. edit: generation speed is also slightly faster now due to the correct tokenizer being used.
@fblissjr Indeed the tokenizer created from the conversion is slightly smaller ~2MB than the original.
I updated as you suggested. Can you check it?
Actually did this myself yesterday with my own quant, and output was better and faster - no idea why. And now unsure if I just had a bug somewhere on my end or if it actually made a difference.
I'm planning to test out a larger CUDA machine later today or tomorrow to see how it works natively.
Let me know how it goes, but for now according to your report the issue should be fixed.
Hey @Blaizzy , I tried your fork and the model is still outputting \<PAD>\<PAD>\<PAD>... when I provide a long prompt.
Hey @Blaizzy , I tried your fork and the model is still outputting
... when I provide a long prompt.
I have made a new change, can you try it again please :)
Wait, I think I got it!
Give me 30 min :)
@jeanromainroy can you try this branch, the previous one had a git issue:
Still outputting \<PAD>\<PAD>\<PAD>... :(
Only PAD ? Can you share the whole output?
It's outputting \<PAD> for as long as I let it. In other words, max_tokens=256, results in 256 x \<PAD>
Got it!
@awni the cohere team added model_max_length
set to 128K on both command-r models.
Is there a way of setting using this number with the nn.Rope? Are there any deep changes needed? If so, please point them, I can work on it.
I'm not sure I fiollow your question. The nn.RoPE
layer does not make any assumptions about the maximum sequence length.
He's talking about the Command-R+'s config:
@jeanromainroy regarding:
I have run the exact same test with the new llama.cpp implementation of Command-R+ and it works way above 8k tokens.
My understanding is llama.cpp uses a fixed size context (the -c
flag) so while it works for long context it does not really "work" in the sense that it does not use any context before a certain limit. It simply truncates everything before the -c
tokens. Default value for -c
is 512:
-c N, --ctx-size N size of the prompt context (default: 512, 0 = loaded from model)
Maybe we could provide something similar.. but I think the default behavior is a little misleading.
@fblissjr could you share a command using mlx_lm.generate
that produces strange responses? I would like to debug that issue if you are still experiencing it.
@fblissjr could you share a command using
mlx_lm.generate
that produces strange responses? I would like to debug that issue if you are still experiencing it.
I can't with mlx_lm.generate because it only happens when I run it with apply_tool_template in the tokenizer. Not at home right now and haven't tested since the day it happened. I think you can mock up something like this:
tools = (a json object similar to the cohere example on HF) if tools, tokenizer.apply_tool_use_template (applied to prompt)
Basically you want to get the apply_tool_use_template to show up, which is a big page-ish long output that looks like this (copying and pasting from HF repo under tool use output example):
@awni you can use the example in the MLX model card to replicate @fblissjr example:
https://huggingface.co/mlx-community/c4ai-command-r-plus-4bit
The nn.RoPE layer does not make any assumptions about the maximum sequence length. My understanding is llama.cpp uses a fixed size context (the
-c
flag) so while it works for long context it does not really "work" in the sense that it does not use any context before a certain limit. It simply truncates everything before the-c
tokens. Default value for-c
is 512:-c N, --ctx-size N size of the prompt context (default: 512, 0 = loaded from model)
Maybe we could provide something similar.. but I think the default behavior is a little misleading.
I see now.
I thought because the PyTorch implementation takes context window size into account we were missing something.
Something like this:
class RotaryPositionalEmbeddings(nn.Module):
def __init__(self, dim, max_position_embeddings=2048, base=10000, device=None, scaling_factor=1.0):
super().__init__()
self.dim = dim
self.max_position_embeddings = max_position_embeddings
self.device=device
self.scaling_factor = scaling_factor
self.base = base
inv_freq = 1.0 / (self.base ** (torch.arange(0, self.dim, 2, dtype=torch.int64).float().to(device) / self.dim))
self.register_buffer("inv_freq", inv_freq, persistent=False)
# Build here to make `torch.jit.trace` work.
self._set_cos_sin_cache(
seq_len=max_position_embeddings, device=self.inv_freq.device, dtype=torch.get_default_dtype()
)
def _set_cos_sin_cache(self, seq_len, device, dtype):
self.max_seq_len_cached = seq_len
t = torch.arange(self.max_seq_len_cached, device=device, dtype=torch.int64).type_as(self.inv_freq)
t = t / self.scaling_factor
freqs = torch.outer(t, self.inv_freq)
# Different from paper, but it uses a different permutation in order to obtain the same calculation
emb = torch.cat((freqs, freqs), dim=-1)
self.register_buffer("cos_cached", emb.cos().to(dtype), persistent=False)
self.register_buffer("sin_cached", emb.sin().to(dtype), persistent=False)
@torch.no_grad()
def forward(self, x, seq_len=None):
# x: [bs, num_attention_heads, seq_len, head_size]
if seq_len > self.max_seq_len_cached:
self._set_cos_sin_cache(seq_len=seq_len, device=x.device, dtype=x.dtype)
return (
self.cos_cached[:seq_len].to(dtype=x.dtype),
self.sin_cached[:seq_len].to(dtype=x.dtype),
)
The nn.RoPE layer does not make any assumptions about the maximum sequence length.
Btw does that mean that we can use context size of any length with nn.RoPE? If not, what are the limitations?
Hi folks, still no consensus on what settings to set, what json files to use?
Using the mlx-community 4bit version, I have random japanese characters. What surprised me was that without even mentionning it in my prompt, at some point the model acknowledged and apoligized for their randomness and said it would try to avoid them.
@M-I could you elaborate on what you mean?
starting with
from mlx_lm import load, generate
model, tokenizer = load("mlx-community/c4ai-command-r-plus-4bit")
at some point, generate(model, tokenizer, prompt=conversation, tokenize=False, add_generation_prompt=True), verbose=True, max_tokens=1000) generated a bit where there was: "Let's start with defining the specific requirements and breaking down the project into manageable tasks. Feel free to share your thoughts and questions along the way! I'm excited to see your project come to life! 😊👍
P.S. Apologies for the random Japanese words (e.g., "グリーニング") that appeared in my response. It seems there might be an issue with my language model. I'll try to avoid this in future responses. 😅👍. I'm always ready to assist you with your project! 😊".
And it will go on and on, as is the end_token is never generated or acknowledged. I just assumed it was the price to pay for 4bit quantization, so I never mentioned the fact that there was Japanese, or even anything weird or out of place in it's response, but it just self reflected on its own. I'll try to find a prompt that I can share that creates similar responses. But others 12 in the discussion tab of the 4bit mlx model on hugginface noticed some weirdness.
I see, thank you for explaining :)
I think this should be a new issue.
As it's not related with this thread.
Cohere's new Command-R-Plus model reportedly features a 128k context window. However, testing with progressively longer prompts reveals it begins producing nonsensical output (e.g., "\<PAD>\<PAD>...") after 8192 tokens, aligning with the "max_position_embeddings" value in the config.json file. The config also lists a "rope_theta" value, suggesting its role in achieving the large context window. Is "rope" supported in MLX?