About ValueError:

[LWZWTWLWZ]

尊敬的作者您好，我在复现您的代码的过程中出现了如下问题，我是将llama2-7b-hf下载到本地通过本地调用实现代码的运行。在运行过程中出现了中断，显示输入有NaN。请问您有好的解决方案吗？ (Tallrec) ubuntu@ubuntu:~/0522231063/Githubfuxian/TALLRec/TALLRec-main$ bash ./shell/instruct_7B.sh 1 42 1, 42 lr: 1e-4, dropout: 0.05 , seed: 42, sample: 64 Training Alpaca-LoRA model with params: base_model: /home/ubuntu/llama2-7b-hf train_data_path: /home/ubuntu/0522231063/Githubfuxian/TALLRec/TALLRec-main/data/book/train.json val_data_path: /home/ubuntu/0522231063/Githubfuxian/TALLRec/TALLRec-main/data/book/valid.json sample: 64 seed: 42 output_dir: /home/ubuntu/0522231063/Githubfuxian/TALLRec/TALLRec-main/model_42_64 batch_size: 128 micro_batch_size: 32 num_epochs: 200 learning_rate: 0.0001 cutoff_len: 512 lora_r: 8 lora_alpha: 16 lora_dropout: 0.05 lora_target_modules: ['q_proj', 'v_proj'] train_on_inputs: True group_by_length: True wandb_project: wandb_run_name: wandb_watch: wandb_log_model: resume_from_checkpoint: /home/ubuntu/0522231063/Githubfuxian/TALLRec/TALLRec-main/alpaca-lora-7B/adapter_config.json

Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:04<00:00, 2.23s/it] Checkpoint /home/ubuntu/0522231063/Githubfuxian/TALLRec/TALLRec-main/alpaca-lora-7B/adapter_config.json/adapter_model.bin not found trainable params: 4194304 || all params: 6742609920 || trainable%: 0.06220594176090199 Map: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 64/64 [00:00<00:00, 1383.30 examples/s] Map: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2427/2427 [00:01<00:00, 1593.21 examples/s] Using the WANDB_DISABLED environment variable is deprecated and will be removed in v5. Use the --report_to flag to control the integrations used for logging result (for instance --report_to none). 0%| | 0/200 [00:00<?, ?it/s]/home/ubuntu/anaconda3/envs/Tallrec/lib/python3.10/site-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants. warnings.warn( {'loss': 1.0435, 'learning_rate': 4e-05, 'epoch': 8.0}
{'eval_loss': 2.0803046226501465, 'eval_auc': 0.499365194424907, 'eval_runtime': 162.0148, 'eval_samples_per_second': 14.98, 'eval_steps_per_second': 1.876, 'epoch': 10.0}
5%|███████▊ | 10/200 [05:12<47:31, 15.01s/it] /home/ubuntu/anaconda3/envs/Tallrec/lib/python3.10/site-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants. warnings.warn( {'loss': 1.0116, 'learning_rate': 8e-05, 'epoch': 16.0}
{'eval_loss': 1.836759328842163, 'eval_auc': 0.5092683767063315, 'eval_runtime': 162.8768, 'eval_samples_per_second': 14.901, 'eval_steps_per_second': 1.866, 'epoch': 20.0}
10%|███████████████▌ | 20/200 [10:25<51:03, 17.02s/it] /home/ubuntu/anaconda3/envs/Tallrec/lib/python3.10/site-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants. warnings.warn( {'loss': 0.9067, 'learning_rate': 9.777777777777778e-05, 'epoch': 24.0}
{'eval_loss': 1.3795101642608643, 'eval_auc': 0.5749956938005082, 'eval_runtime': 162.5594, 'eval_samples_per_second': 14.93, 'eval_steps_per_second': 1.87, 'epoch': 30.0}
15%|███████████████████████▍ | 30/200 [15:38<48:25, 17.09s/it] /home/ubuntu/anaconda3/envs/Tallrec/lib/python3.10/site-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants. warnings.warn( {'loss': 0.7264, 'learning_rate': 9.333333333333334e-05, 'epoch': 32.0}
{'loss': 0.5474, 'learning_rate': 8.888888888888889e-05, 'epoch': 40.0}
20%|███████████████████████████████▏ | 40/200 [18:09<45:34, 17.09s/it] Traceback (most recent call last):██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 304/304 [02:41<00:00, 2.19it/s] File "/home/ubuntu/0522231063/Githubfuxian/TALLRec/TALLRec-main/finetune_rec.py", line 325, in fire.Fire(train) File "/home/ubuntu/anaconda3/envs/Tallrec/lib/python3.10/site-packages/fire/core.py", line 141, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "/home/ubuntu/anaconda3/envs/Tallrec/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire component, remaining_args = _CallAndUpdateTrace( File "/home/ubuntu/anaconda3/envs/Tallrec/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace component = fn(*varargs, *kwargs) File "/home/ubuntu/0522231063/Githubfuxian/TALLRec/TALLRec-main/finetune_rec.py", line 292, in train trainer.train(resume_from_checkpoint=resume_from_checkpoint) File "/home/ubuntu/anaconda3/envs/Tallrec/lib/python3.10/site-packages/transformers/trainer.py", line 1662, in train return inner_training_loop( File "/home/ubuntu/anaconda3/envs/Tallrec/lib/python3.10/site-packages/transformers/trainer.py", line 2006, in _inner_training_loop self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval) File "/home/ubuntu/anaconda3/envs/Tallrec/lib/python3.10/site-packages/transformers/trainer.py", line 2287, in _maybe_log_save_evaluate metrics = self.evaluate(ignore_keys=ignore_keys_for_eval) File "/home/ubuntu/anaconda3/envs/Tallrec/lib/python3.10/site-packages/transformers/trainer.py", line 2993, in evaluate output = eval_loop( File "/home/ubuntu/anaconda3/envs/Tallrec/lib/python3.10/site-packages/transformers/trainer.py", line 3281, in evaluation_loop metrics = self.compute_metrics(EvalPrediction(predictions=all_preds, label_ids=all_labels)) File "/home/ubuntu/0522231063/Githubfuxian/TALLRec/TALLRec-main/finetune_rec.py", line 222, in compute_metrics auc = roc_auc_score(pre[1], pre[0]) File "/home/ubuntu/anaconda3/envs/Tallrec/lib/python3.10/site-packages/sklearn/utils/_param_validation.py", line 213, in wrapper return func(args, **kwargs) File "/home/ubuntu/anaconda3/envs/Tallrec/lib/python3.10/site-packages/sklearn/metrics/_ranking.py", line 606, in roc_auc_score y_score = check_array(y_score, ensure_2d=False) File "/home/ubuntu/anaconda3/envs/Tallrec/lib/python3.10/site-packages/sklearn/utils/validation.py", line 1003, in check_array _assert_all_finite( File "/home/ubuntu/anaconda3/envs/Tallrec/lib/python3.10/site-packages/sklearn/utils/validation.py", line 126, in _assert_all_finite _assert_all_finite_element_wise( File "/home/ubuntu/anaconda3/envs/Tallrec/lib/python3.10/site-packages/sklearn/utils/validation.py", line 175, in _assert_all_finite_element_wise raise ValueError(msg_err) ValueError: Input contains NaN. 20%|██████████████████████████████▊ | 40/200 [20:51<1:23:27, 31.30s/it]