orpo脚本NoneType问题

[songyao199681]

什么都没改执行结果如下： (songyao_MedicalGPT) root@jkha-W580-G20:/software/MedicalGPT# CUDA_VISIBLE_DEVICES=0,1 python orpo_training.py \ --model_type auto \ --model_name_or_path Qwen/Qwen1.5-0.5B-Chat \ --train_file_dir ./data/reward \ --validation_file_dir ./data/reward \ --per_device_train_batch_size 4 \ --per_device_eval_batch_size 1 \ --do_train \ --do_eval \ --use_peft True \ --max_train_samples 1000 \ --max_eval_samples 10 \ --max_steps 100 \ --eval_steps 20 \ --save_steps 50 \ --max_source_length 128 \ --max_target_length 128 \ --output_dir outputs-orpo-qwen-v1 \ --target_modules all \ --lora_rank 8 \ --lora_alpha 16 \ --lora_dropout 0.05 \ --torch_dtype float16 \ --fp16 True \ --device_map auto \ --report_to tensorboard \ --remove_unused_columns False \ --gradient_checkpointing True \ --orpo_beta 0.1 \ --cache_dir ./cache /root/anaconda3/envs/songyao_MedicalGPT/lib/python3.10/site-packages/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import d eepspeed modules directly from transformers.integrations warnings.warn( [2024-04-22 16:02:43,162] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect) [WARNING] async_io requires the dev libaio .so object and headers but these were not found. [WARNING] async_io: please install the libaio-dev package with apt [WARNING] If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. [WARNING] Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH [WARNING] NVIDIA Inference is only supported on Ampere and newer architectures [WARNING] sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.2 [WARNING] using untested triton version (2.2.0), only 1.0.0 is known to be compatible 2024-04-22 16:02:43.869 | INFO | main:main:238 - Parse args: ScriptArguments(model_type='auto', model_name_or_path='Qwen/Qwen1.5-0.5B-Chat', tokenizer_name_or_path=None, load_in_8bit=False, load_in_4bit=Fal se, cache_dir='./cache', use_fast_tokenizer=False, torch_dtype='float16', device_map='auto', trust_remote_code=True, dataset_name=None, dataset_config_name=None, train_file_dir='./data/reward', validation_file_dir= './data/reward', template_name='vicuna', per_device_train_batch_size=4, per_device_eval_batch_size=1, max_source_length=128, max_target_length=128, min_target_length=4, max_train_samples=1000, max_eval_samples=10, overwrite_cache=False, validation_split_percentage=1, preprocessing_num_workers=4, use_peft=True, qlora=False, target_modules='all', lora_rank=8, lora_dropout=0.05, lora_alpha=16.0, peft_path=None, do_train=True, d o_eval=True, beta=0.1, learning_rate=0.0005, lr_scheduler_type='cosine', warmup_steps=100, weight_decay=0.05, optim='adamw_hf', fp16=True, bf16=False, gradient_checkpointing=True, gradient_accumulation_steps=4, sav e_steps=50, eval_steps=20, logging_steps=1, output_dir='outputs-orpo-qwen-v1', max_steps=100, eval_strategy='steps', remove_unused_columns=False, report_to='tensorboard', orpo_beta=0.1) tokenizer_config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.29k/1.29k [00:00<00:00, 4.82MB/s] vocab.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2.78M/2.78M [00:01<00:00, 2.45MB/s] merges.txt: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.67M/1.67M [00:00<00:00, 11.9MB/s] tokenizer.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7.03M/7.03M [00:01<00:00, 6.95MB/s] Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. 2024-04-22 16:02:52.707 | INFO | main:main:282 - train files: ./data/reward/test.json 2024-04-22 16:02:52.707 | INFO | main:main:287 - eval files: ./data/reward/test.json 2024-04-22 16:02:54.092 | INFO | main:main:308 - Raw datasets: DatasetDict({ train: Dataset({ features: ['question', 'response_chosen', 'response_rejected'], num_rows: 100 }) validation: Dataset({ features: ['question', 'response_chosen', 'response_rejected'], num_rows: 100 }) }) 2024-04-22 16:02:54.095 | DEBUG | main:main:326 - Example train_dataset[0]: {'question': '肛门病变可能是什么疾病的症状?', 'response_chosen': '食管克罗恩病', 'response_rejected': '肛门病变可能与多种不同类型的 病症有关。'} Running tokenizer on dataset (num_proc=4): 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:00<00:00, 645.92 examples/s] Filter: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:00<00:00, 8588.55 examples/s] 2024-04-22 16:02:54.388 | DEBUG | main:main:339 - Num train_samples: 68 2024-04-22 16:02:54.389 | DEBUG | main:main:340 - First train example: 2024-04-22 16:02:54.390 | DEBUG | main:main:341 - Question: 晚期先天性梅毒的西医治疗

Answer: 治疗：普鲁卡因青霉素G，每日5万u/kg体重，肌注，连续10天为一疗程(不超过成人剂量)。8岁以下儿童禁用四环素。先天梅毒对青霉素过敏者可用红霉素治疗，每日～12.5mg/kg体重，分4次服，连服30天。预后：经过迟缓，病程较长 ，对抗梅毒疗法有抗拒性，抗梅毒疗法难控制其进行，预后难定，患儿年龄较小，且身体健康较好，治疗充分者预后较好，否则可致盲、耳聋等。 2024-04-22 16:02:54.394 | DEBUG | main:main:353 - Example eval_dataset[0]: {'question': '肛门病变可能是什么疾病的症状?', 'response_chosen': '食管克罗恩病', 'response_rejected': '肛门病变可能与多种不同类型的 病症有关。'} 2024-04-22 16:02:54.426 | DEBUG | main:main:366 - Num eval_samples: 8 2024-04-22 16:02:54.426 | DEBUG | main:main:367 - First eval example: 2024-04-22 16:02:54.427 | DEBUG | main:main:368 - Question: 肛门病变可能是什么疾病的症状?

Answer: 食管克罗恩病 2024-04-22 16:02:54.427 | INFO | main:main:380 - Device map: auto config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 661/661 [00:00<00:00, 2.32MB/s] model.safetensors: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.24G/1.24G [01:31<00:00, 13.6MB/s] generation_config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 206/206 [00:00<00:00, 1.09MB/s] 2024-04-22 16:04:30.927 | INFO | main:main:443 - Fine-tuning method: LoRA(PEFT) 2024-04-22 16:04:30.928 | INFO | main:main:447 - Peft target_modules: ['down_proj', 'gate_proj', 'k_proj', 'o_proj', 'q_proj', 'up_proj', 'v_proj'] /root/anaconda3/envs/songyao_MedicalGPT/lib/python3.10/site-packages/trl/trainer/orpo_trainer.py:209: UserWarning: max_length is not set in the ORPOConfig's init it will default to 512 by default, but you shoul d do it yourself in the future. warnings.warn( /root/anaconda3/envs/songyao_MedicalGPT/lib/python3.10/site-packages/trl/trainer/orpo_trainer.py:218: UserWarning: max_prompt_length is not set in the ORPOConfig's init it will default to 128 by default, but yo u should do it yourself in the future. warnings.warn( Map: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 68/68 [00:00<00:00, 184.54 examples/s] Map: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:00<00:00, 173.37 examples/s] trainable params: 3784704 || all params: 467772416 || trainable%: 0.8090908891900116 2024-04-22 16:04:39.080 | INFO | main:main:470 - Train /root/anaconda3/envs/songyao_MedicalGPT/lib/python3.10/site-packages/transformers/optimization.py:457: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTo rch implementation torch.optim.AdamW instead, or set no_deprecation_warning=True to disable this warning warnings.warn( 0%| | 0/100 [00:00<?, ?it/s] Traceback (most recent call last): File "/software/MedicalGPT/orpo_training.py", line 496, in main() File "/software/MedicalGPT/orpo_training.py", line 471, in main train_result = trainer.train() File "/root/anaconda3/envs/songyao_MedicalGPT/lib/python3.10/site-packages/transformers/trainer.py", line 1780, in train return inner_training_loop( File "/root/anaconda3/envs/songyao_MedicalGPT/lib/python3.10/site-packages/transformers/trainer.py", line 2085, in _inner_training_loop for step, inputs in enumerate(epoch_iterator): File "/root/anaconda3/envs/songyao_MedicalGPT/lib/python3.10/site-packages/accelerate/data_loader.py", line 452, in iter current_batch = next(dataloader_iter) File "/root/anaconda3/envs/songyao_MedicalGPT/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 631, in next data = self._next_data() File "/root/anaconda3/envs/songyao_MedicalGPT/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 675, in _next_data data = self._dataset_fetcher.fetch(index) # may raise StopIteration File "/root/anaconda3/envs/songyao_MedicalGPT/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 54, in fetch return self.collate_fn(data) File "/root/anaconda3/envs/songyao_MedicalGPT/lib/python3.10/site-packages/trl/trainer/utils.py", line 338, in call to_pad = [torch.LongTensor(ex[k]) for ex in features] File "/root/anaconda3/envs/songyao_MedicalGPT/lib/python3.10/site-packages/trl/trainer/utils.py", line 338, in to_pad = [torch.LongTensor(ex[k]) for ex in features] TypeError: 'NoneType' object cannot be interpreted as an integer 0%|

[huyidu]

同样的问题。请问解决没？

已经解决 需要手动设置 tokenizer.bos_token_id 默认 tokenizer.bos_token_id=None tokenizer.bos_token_id = 151643 # 千问

[Codingfarmer-hkl]

同样的问题。请问解决没？

因为orpo_trainer.py 443行， prompt_tokens["prompt_input_ids"] = [self.tokenizer.bos_token_id] + prompt_tokens["prompt_input_ids"]中的tokenizer.bos_token_id为None 判断一下这个tokenizer.bos_token_id是否存在，不存在手动设置一下，默认为None

[songyao199681]

同样的问题。请问解决没？

没有解决，我在研究LLaMA-Factory，带图形界面的微调了

[songyao199681]

同样的问题。请问解决没？

因为orpo_trainer.py 443行， prompt_tokens["prompt_input_ids"] = [self.tokenizer.bos_token_id] + prompt_tokens["prompt_input_ids"]中的tokenizer.bos_token_id为None 判断一下这个tokenizer.bos_token_id是否存在，不存在手动设置一下，默认为None

if tokenizer.pad_token_id is None: tokenizer.pad_token_id = 0 # set as the token 麻烦说具体点，编程水平有限，多谢

[Codingfarmer-hkl]

同样的问题。请问解决没？

因为orpo_trainer.py 443行， prompt_tokens["prompt_input_ids"] = [self.tokenizer.bos_token_id] + prompt_tokens["prompt_input_ids"]中的tokenizer.bos_token_id为None 判断一下这个tokenizer.bos_token_id是否存在，不存在手动设置一下，默认为None

if tokenizer.pad_token_id is None: tokenizer.pad_token_id = 0 # set as the token 麻烦说具体点，编程水平有限，多谢 看generation_config.json文件里的pad_token_id是多少，把0改成对应的数值。你上面改的应该不会报NoneType的错误把

[shibing624]

fixed, need add bos_token.