OverflowError: out of range integral type conversion attempted while running python qlora.py

[amdnsr]

python qlora.py --model_name_or_path decapoda-research/llama-13b-hf

(I have updated the tokenizer_config.json and config.json as per the various discussions here tokenizer_class: LlamaTokenizer and architectures: LlamaForCausalLM)

==================================================================================

adding LoRA modules... trainable params: 125173760.0 || all params: 6922327040 || trainable: 1.8082612866554193 loaded model

Using pad_token, but it is not set yet. Traceback (most recent call last): File "qlora.py", line 758, in train() File "qlora.py", line 620, in train "unk_token": tokenizer.convert_ids_to_tokens(model.config.pad_token_id), File "/home/envs/qlora_env/lib/python3.8/site-packages/transformers/tokenization_utils_fast.py", line 307, in convert_ids_to_tokens return self._tokenizer.id_to_token(ids) OverflowError: out of range integral type conversion attempted

[amdnsr]

Running on Tesla V100 32GB GPU

[MaticsL]

same issue

update

Change model.config.pad_token_id to 0 should fix this problem but may harm to training.

        tokenizer.add_special_tokens(
            {
                "eos_token": tokenizer.convert_ids_to_tokens(model.config.eos_token_id),
                "bos_token": tokenizer.convert_ids_to_tokens(model.config.bos_token_id),
                "unk_token": tokenizer.convert_ids_to_tokens(0),
            }
        )

[LIO-H-ZEN]

same issue

[Qubitium]

Please check if this pr fixes your issue. The pr was designed to fix something else but will also bypass token conversion if tokenizer already contains the special tokens.

https://github.com/artidoro/qlora/pull/20

[ricksun2023]

same issue

[atillabasaran]

Please check if this pr fixes your issue. The pr was designed to fix something else but will also bypass token conversion if tokenizer already contains the special tokens.

20

this is solve my issue but I am getting maximum recursion depth error now.

File "/home/atilla/miniconda3/envs/qlora/lib/python3.9/site-packages/transformers/tokenization_utils_fast.py", line 257, in _convert_token_to_id_with_added_voc     return self.unk_token_id   File "/home/atilla/miniconda3/envs/qlora/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 1142, in unk_token_id     return self.convert_tokens_to_ids(self.unk_token)   File "/home/atilla/miniconda3/envs/qlora/lib/python3.9/site-packages/transformers/tokenization_utils_fast.py", line 250, in convert_tokens_to_ids     return self._convert_token_to_id_with_added_voc(tokens)   File "/home/atilla/miniconda3/envs/qlora/lib/python3.9/site-packages/transformers/tokenization_utils_fast.py", line 257, in _convert_token_to_id_with_added_voc     return self.unk_token_id   File "/home/atilla/miniconda3/envs/qlora/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 1142, in unk_token_id     return self.convert_tokens_to_ids(self.unk_token)   File "/home/atilla/miniconda3/envs/qlora/lib/python3.9/site-packages/transformers/tokenization_utils_fast.py", line 250, in convert_tokens_to_ids     return self._convert_token_to_id_with_added_voc(tokens)   File "/home/atilla/miniconda3/envs/qlora/lib/python3.9/site-packages/transformers/tokenization_utils_fast.py", line 257, in _convert_token_to_id_with_added_voc     return self.unk_token_id   File "/home/atilla/miniconda3/envs/qlora/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 1142, in unk_token_id     return self.convert_tokens_to_ids(self.unk_token)   File "/home/atilla/miniconda3/envs/qlora/lib/python3.9/site-packages/transformers/tokenization_utils_fast.py", line 250, in convert_tokens_to_ids     return self._convert_token_to_id_with_added_voc(tokens)   File "/home/atilla/miniconda3/envs/qlora/lib/python3.9/site-packages/transformers/tokenization_utils_fast.py", line 257, in _convert_token_to_id_with_added_voc     return self.unk_token_id   File "/home/atilla/miniconda3/envs/qlora/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 1142, in unk_token_id     return self.convert_tokens_to_ids(self.unk_token) RecursionError: maximum recursion depth exceeded

[LIO-H-ZEN]

Please check if this pr fixes your issue. The pr was designed to fix something else but will also bypass token conversion if tokenizer already contains the special tokens.

20

this is solve my issue but I am getting maximum recursion depth error now.

File "/home/atilla/miniconda3/envs/qlora/lib/python3.9/site-packages/transformers/tokenization_utils_fast.py", line 257, in _convert_token_to_id_with_added_voc     return self.unk_token_id   File "/home/atilla/miniconda3/envs/qlora/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 1142, in unk_token_id     return self.convert_tokens_to_ids(self.unk_token)   File "/home/atilla/miniconda3/envs/qlora/lib/python3.9/site-packages/transformers/tokenization_utils_fast.py", line 250, in convert_tokens_to_ids     return self._convert_token_to_id_with_added_voc(tokens)   File "/home/atilla/miniconda3/envs/qlora/lib/python3.9/site-packages/transformers/tokenization_utils_fast.py", line 257, in _convert_token_to_id_with_added_voc     return self.unk_token_id   File "/home/atilla/miniconda3/envs/qlora/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 1142, in unk_token_id     return self.convert_tokens_to_ids(self.unk_token)   File "/home/atilla/miniconda3/envs/qlora/lib/python3.9/site-packages/transformers/tokenization_utils_fast.py", line 250, in convert_tokens_to_ids     return self._convert_token_to_id_with_added_voc(tokens)   File "/home/atilla/miniconda3/envs/qlora/lib/python3.9/site-packages/transformers/tokenization_utils_fast.py", line 257, in _convert_token_to_id_with_added_voc     return self.unk_token_id   File "/home/atilla/miniconda3/envs/qlora/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 1142, in unk_token_id     return self.convert_tokens_to_ids(self.unk_token)   File "/home/atilla/miniconda3/envs/qlora/lib/python3.9/site-packages/transformers/tokenization_utils_fast.py", line 250, in convert_tokens_to_ids     return self._convert_token_to_id_with_added_voc(tokens)   File "/home/atilla/miniconda3/envs/qlora/lib/python3.9/site-packages/transformers/tokenization_utils_fast.py", line 257, in _convert_token_to_id_with_added_voc     return self.unk_token_id   File "/home/atilla/miniconda3/envs/qlora/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 1142, in unk_token_id     return self.convert_tokens_to_ids(self.unk_token) RecursionError: maximum recursion depth exceeded

same...

[ghtaro]

Hi, I changed to huggyllama/llama-7b and applied the chanige #20. I avoided the above errors and now below*

Traceback (most recent call last):
  File "/Workspace/Repos/toshiya.imoto@japan-d2.com/qlora/qlora.py", line 853, in <module>
    train()
  File "/Workspace/Repos/toshiya.imoto@japan-d2.com/qlora/qlora.py", line 824, in train
    metrics = trainer.evaluate(metric_key_prefix="eval")
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-7966afd1-6600-45a5-a135-50b716d0345e/lib/python3.10/site-packages/transformers/trainer_seq2seq.py", line 159, in evaluate
    return super().evaluate(eval_dataset, ignore_keys=ignore_keys, metric_key_prefix=metric_key_prefix)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-7966afd1-6600-45a5-a135-50b716d0345e/lib/python3.10/site-packages/transformers/trainer.py", line 3108, in evaluate
    self.control = self.callback_handler.on_evaluate(self.args, self.state, self.control, output.metrics)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-7966afd1-6600-45a5-a135-50b716d0345e/lib/python3.10/site-packages/transformers/trainer_callback.py", line 379, in on_evaluate
    return self.call_event("on_evaluate", args, state, control, metrics=metrics)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-7966afd1-6600-45a5-a135-50b716d0345e/lib/python3.10/site-packages/transformers/trainer_callback.py", line 397, in call_event
    result = getattr(callback, event)(
  File "/Workspace/Repos/toshiya.imoto@japan-d2.com/qlora/qlora.py", line 751, in on_evaluate
    refs += [abcd_idx.index(label) for label in labels.tolist()]
  File "/Workspace/Repos/toshiya.imoto@japan-d2.com/qlora/qlora.py", line 751, in <listcomp>
    refs += [abcd_idx.index(label) for label in labels.tolist()]
ValueError: 29879 is not in list

I found that:

abcd_idx:  [319, 350, 315, 360]

labels tensor([  319, 29879,   350, 29879,   319, 29879,   315, 29879,   315, 29879,
          360, 29879,   350, 29879,   360, 29879], device='cuda:0')

Can anyone have an idea how to sort this out?

[atillabasaran]

same issue

update

Change model.config.pad_token_id to 0 should fix this problem but may harm to training.

        tokenizer.add_special_tokens(
            {
                "eos_token": tokenizer.convert_ids_to_tokens(model.config.eos_token_id),
                "bos_token": tokenizer.convert_ids_to_tokens(model.config.bos_token_id),
                "unk_token": tokenizer.convert_ids_to_tokens(0),
            }
        )

I wonder what exactly does this change provide?

[mofanv]

same issue

[hemangjoshi37a]

@amdnsr,

Based on the error message you shared, it appears that there is an "OverflowError: out of range integral type conversion attempted" when converting token IDs during tokenization. To address this issue, we recommend the following solution:

1. Update your code as follows:

from transformers import LlamaTokenizer

tokenizer = LlamaTokenizer.from_pretrained("decapoda-research/llama-13b-hf")
tokenizer.add_special_tokens({"pad_token": "[PAD]"})

# Rest of your code...

By using the LlamaTokenizer from the transformers library and adding the [PAD] token as a special token, you can resolve the "out of range integral type conversion" error.

Best regards, @hemangjoshi37a

[saxenarohit]

Setting pad_token_id in the model config worked for me.

For example for vicuna model.config.pad_token_id = tokenizer.eos_token_id