Open artemdinaburg opened 3 months ago
@artemdinaburg is there a dataset you can share to reproduce the issue? I tried codellama with the alpaca dataset and it tokenized without any issues, so it's likely something specific to the dataset you are using.
Hi Wing,
Thanks for taking a look! I did some binary searching and I have a three-line JSONL that reproduces a very similar error (the lengths are different).
Invocation:
python -m axolotl.cli.preprocess debug.yml --debug
[2024-05-20 11:37:56,299] [INFO] [datasets.<module>:58] [PID:1195709] PyTorch version 2.1.2 available.
dP dP dP
88 88 88
.d8888b. dP. .dP .d8888b. 88 .d8888b. d8888P 88
88' `88 `8bd8' 88' `88 88 88' `88 88 88
88. .88 .d88b. 88. .88 88 88. .88 88 88
`88888P8 dP' `dP `88888P' dP `88888P' dP dP
****************************************
**** Axolotl Dependency Versions *****
accelerate: 0.30.1
peft: 0.10.0
transformers: 4.40.2
trl: 0.8.5
torch: 2.1.2
bitsandbytes: 0.43.1
****************************************
[2024-05-20 11:37:57,317] [WARNING] [axolotl.utils.config.models.input.hint_trust_remote_code:276] [PID:1195709] [RANK:0] `trust_remote_code` is set to true. Please make sure that you reviewed the remote code/model.
[2024-05-20 11:37:57,317] [DEBUG] [axolotl.normalize_config:79] [PID:1195709] [RANK:0] bf16 support detected, enabling for this configuration.
/home/artem/anaconda3/envs/sol-eval/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
warnings.warn(
[2024-05-20 11:37:57,523] [INFO] [axolotl.normalize_config:182] [PID:1195709] [RANK:0] GPU memory usage baseline: 0.000GB (+0.216GB misc)
[2024-05-20 11:37:58,318] [DEBUG] [axolotl.load_tokenizer:280] [PID:1195709] [RANK:0] EOS: 2 / <|endoftext|>
[2024-05-20 11:37:58,318] [DEBUG] [axolotl.load_tokenizer:281] [PID:1195709] [RANK:0] BOS: 2 / <|endoftext|>
[2024-05-20 11:37:58,318] [DEBUG] [axolotl.load_tokenizer:282] [PID:1195709] [RANK:0] PAD: 2 / <|endoftext|>
[2024-05-20 11:37:58,318] [DEBUG] [axolotl.load_tokenizer:283] [PID:1195709] [RANK:0] UNK: 0 / <unk>
[2024-05-20 11:37:58,318] [INFO] [axolotl.load_tokenizer:294] [PID:1195709] [RANK:0] No Chat template selected. Consider adding a chat template for easier inference.
[2024-05-20 11:37:58,319] [INFO] [axolotl.load_tokenized_prepared_datasets:183] [PID:1195709] [RANK:0] Unable to find prepared dataset in last_run_prepared/884cc22e60a6d1b0682e05f4340d9a4d
[2024-05-20 11:37:58,319] [INFO] [axolotl.load_tokenized_prepared_datasets:184] [PID:1195709] [RANK:0] Loading raw datasets...
[2024-05-20 11:37:58,319] [INFO] [axolotl.load_tokenized_prepared_datasets:193] [PID:1195709] [RANK:0] No seed provided, using default seed of 42
Generating train split: 3 examples [00:00, 1508.56 examples/s]
num_proc must be <= 3. Reducing num_proc to 3 for dataset of size 3.
[2024-05-20 11:37:58,672] [WARNING] [datasets.arrow_dataset.map:3087] [PID:1195709] num_proc must be <= 3. Reducing num_proc to 3 for dataset of size 3.
Tokenizing Prompts (num_proc=3): 67%|████████████████████████████████████████ | 2/3 [00:00<00:00, 7.60 examples/s]
multiprocess.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/artem/anaconda3/envs/sol-eval/lib/python3.10/site-packages/multiprocess/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/home/artem/anaconda3/envs/sol-eval/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 678, in _write_generator_to_queue
for i, result in enumerate(func(**kwargs)):
File "/home/artem/anaconda3/envs/sol-eval/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 3570, in _map_single
writer.write_batch(batch)
File "/home/artem/anaconda3/envs/sol-eval/lib/python3.10/site-packages/datasets/arrow_writer.py", line 571, in write_batch
pa_table = pa.Table.from_arrays(arrays, schema=schema)
File "pyarrow/table.pxi", line 4642, in pyarrow.lib.Table.from_arrays
File "pyarrow/table.pxi", line 3922, in pyarrow.lib.Table.validate
File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Column 1 named token_type_ids expected length 2 but got length 1
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/artem/anaconda3/envs/sol-eval/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/artem/anaconda3/envs/sol-eval/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/home/artem/git/axolotl/src/axolotl/cli/preprocess.py", line 82, in <module>
fire.Fire(do_cli)
File "/home/artem/anaconda3/envs/sol-eval/lib/python3.10/site-packages/fire/core.py", line 143, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/home/artem/anaconda3/envs/sol-eval/lib/python3.10/site-packages/fire/core.py", line 477, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/home/artem/anaconda3/envs/sol-eval/lib/python3.10/site-packages/fire/core.py", line 693, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/home/artem/git/axolotl/src/axolotl/cli/preprocess.py", line 72, in do_cli
load_datasets(cfg=parsed_cfg, cli_args=parsed_cli_args)
File "/home/artem/git/axolotl/src/axolotl/cli/__init__.py", line 403, in load_datasets
train_dataset, eval_dataset, total_num_steps, prompters = prepare_dataset(
File "/home/artem/git/axolotl/src/axolotl/utils/data/sft.py", line 66, in prepare_dataset
train_dataset, eval_dataset, prompters = load_prepare_datasets(
File "/home/artem/git/axolotl/src/axolotl/utils/data/sft.py", line 460, in load_prepare_datasets
dataset, prompters = load_tokenized_prepared_datasets(
File "/home/artem/git/axolotl/src/axolotl/utils/data/sft.py", line 399, in load_tokenized_prepared_datasets
dataset_wrapper, dataset_prompter = get_dataset_wrapper(
File "/home/artem/git/axolotl/src/axolotl/utils/data/sft.py", line 553, in get_dataset_wrapper
dataset_wrapper = TokenizedPromptDataset(
File "/home/artem/git/axolotl/src/axolotl/datasets.py", line 43, in __init__
self.process(dataset).data,
File "/home/artem/git/axolotl/src/axolotl/datasets.py", line 55, in process
return dataset.map(
File "/home/artem/anaconda3/envs/sol-eval/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 602, in wrapper
out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
File "/home/artem/anaconda3/envs/sol-eval/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 567, in wrapper
out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
File "/home/artem/anaconda3/envs/sol-eval/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 3248, in map
for rank, done, content in iflatmap_unordered(
File "/home/artem/anaconda3/envs/sol-eval/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 718, in iflatmap_unordered
[async_result.get(timeout=0.05) for async_result in async_results]
File "/home/artem/anaconda3/envs/sol-eval/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 718, in <listcomp>
[async_result.get(timeout=0.05) for async_result in async_results]
File "/home/artem/anaconda3/envs/sol-eval/lib/python3.10/site-packages/multiprocess/pool.py", line 774, in get
raise self._value
pyarrow.lib.ArrowInvalid: Column 1 named token_type_ids expected length 2 but got length 1
My config (debug.yml):
base_model: Qwen/CodeQwen1.5-7B
trust_remote_code: true
load_in_8bit: true
load_in_4bit: false
strict: false
datasets:
- path: ./minimum_error_repro.jsonl
type: completion
dataset_prepared_path: ./last_run_prepared
output_dir: ./codeqwen-7b-lora/
val_set_size: 0.05
sequence_len: 1024
sample_packing: false
adapter: lora
lora_model_dir:
lora_r: 256
lora_alpha: 512
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:
wandb_project: solidity
wandb_name: codeqwen-7b-solidity-lora
wandb_watch:
wandb_log_model:
gradient_accumulation_steps: 4
micro_batch_size: 2
num_epochs: 1
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.0003
train_on_inputs: false
group_by_length: false
bf16: auto
fp16: false
tf32: false
gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
auto_resume_from_checkpoints: true
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
s2_attention:
warmup_steps: 10
eval_step: 0.10
save_strategy: steps
save_steps: 100
save_total_limit: 5
debug:
deepspeed:
weight_decay: 0.1
fsdp:
fsdp_config:
special_tokens:
pad_token: "<|endoftext|>"
As an update, the first two lines seem irrelevant and each can be replaced with {"text": "A"}
but the third line is extremely sensitive to any kind of modification; adding or removing tokens makes the error disappear.
@winglian I tracked it down to here: https://github.com/OpenAccess-AI-Collective/axolotl/blob/main/src/axolotl/prompt_tokenizers.py#L80-L86
where a token is added to input_ids
and attention_mask
but not token_type_ids
I believe the Qwen tokenizer is one of the few that returns token_type_ids
.
The resolution would be to either set return_token_type_ids=False
in the call to tokenizer, or to add an entry for token_type_ids
. I am not sure which is best.
Validated that adding:
result["attention_mask"].append(1)
if "token_type_ids" in result:
result["token_type_ids"].append(result["token_type_ids"][-1])```
Seems to fix it for me.
Please check that this issue hasn't been reported before.
Expected Behavior
I am attempting to finetune CodeQwen 1.5 7B on a custom dataset; I expect the finetuning to start; the same exact dataset works for CodeLlama and DeepSeek Coder.
Current behaviour
Possible solution
My gut tells me this has something to do with Qwen's weird tokenizer.
Which Operating Systems are you using?
Python Version
3.10
axolotl branch-commit
docker winglian/axolotl:main-latest b7c4ce815c7d2e77aae54390747c06ec99ac0409a8d989e9b3abf2aa9023c9d1
Acknowledgements