Closed ohxh closed 3 weeks ago
weird! i thought the tokenizers were left-padding by default... ah, mistral does...
>>> llama3_tokenizer(["x", "x x"], padding=True)
{'input_ids': [[128000, 87, 128001], [128000, 87, 865]], 'attention_mask': [[1, 1, 0], [1, 1, 1]]}
>>> mistral_tokenizer(["x", "x x"], padding=True)
{'input_ids': [[2, 1, 1318], [1, 1318, 1318]], 'attention_mask': [[0, 1, 1], [1, 1, 1]]}
Oh huh… maybe an easier fix would be to force the tokenizer to always left pad
can you check "allow edits by maintainers" so i can make changes to this PR?
Oh huh… maybe an easier fix would be to force the tokenizer to always left pad
yeah i was thinking that, but i think your approach is better because the user might want right-padding for whatever reason--better to not mess with their tokenizer instance if we can avoid it.
I think it is checked already…
Glad I checked the PRs too, was just about to cut the 0.3 release so you just squeaked in!
First of all, this is a really neat repo!
I noticed that
batched_get_hiddens
always takes hidden states from the last token in each sequence in a batch. Since the sequences are padded to the same length, this means that batching affects the hidden states for all but the longest sequence in each batch.After this change, there's still some difference between the batched and non-batched hidden states, but I think that might be due to the model itself since batching changes the order of operations: https://github.com/huggingface/transformers/issues/23017#issuecomment-1649630232
I've only tried this on llama-3-8b, I'm not sure if it will need changes to work on other models.