mntss
commented
1 month ago I think the problem you're seeing is caused by the prompt formatting and not the implementation differences.
I compared the TL to HF model and while there are some small differences in activations the logit differences seem negligible (max 0.0009 2.62e-06 diff in softmax outputs).
This model uses chatml template for the inputs, for your example, the format should be something like this:
'<|im_start|>system\n<|im_end|>\n<|im_start|>user\nhello my name is <|im_end|>\n<|im_start|>assistant\n'
While the example above will use:
<|endoftext|>hello my name is
I noticed that TL will fallback to prepend to EOS token if BOS is absent, which seems incorrect in this case.
Code:
import torch
from torch.nn import functional as F
from transformers import AutoTokenizer, AutoModelForCausalLM
from transformer_lens import HookedTransformer
tokenizer = AutoTokenizer.from_pretrained(
"Qwen/Qwen-1_8B-Chat",
trust_remote_code=True,
add_bos_token=True,
)
model_hf = AutoModelForCausalLM.from_pretrained(
"Qwen/Qwen-1_8B-Chat",
trust_remote_code=True,
device_map="cuda",
fp32=True,
).eval()
model = HookedTransformer.from_pretrained_no_processing(
"Qwen/Qwen-1_8B-Chat",
dtype=torch.float32,
device="cuda",
)
encoding = tokenizer("hello my name is ", return_tensors="pt")
response_hf = model_hf(encoding.input_ids.cuda())
logits_tl = model(encoding.input_ids.cuda())
diff = logits_tl - response_hf.logits
prob_diff = F.softmax(logits_tl, dim=-1) - F.softmax(response_hf.logits, dim=-1)
prob_diff.std().item(), prob_diff.mean().item(), prob_diff.max().item()
bryce13950
The whole Qwen model family seems to be pretty inaccurate. I have not done complete benchmarks to determine where the issue is yet. That still needs to be done to fine the specific area causing the error. This is probably due to einsum usage, and a slightly inaccuracy from the Transformers implementation. To solve this, we need to remove any potentially troublesome einsums that are in the model, which verify that any components used have similar implementations to transformers, which may result in the creation of more components in TransformerLens.
Describe the bug The output is currently switching languages in what seems to be all models. I tested three different models, and I found that when putting in English, the output will sometimes be a bit of nonsense, and often with some Chinese mixed in. I then decided to generate a bit in Chinese, which resulted in kanji Japanese being generated. This is particularly interesting, since the characters I was using are in both Chinese and Japanese, but if the model mistook my input as Japanese, it should have still generated the same writing style.
Code example
System Info This was found in colab using various versions of TransformerLens 2.x and 1.x.
Additional context
Checklist