hiyouga
commented
10 months ago 训练命令？
Closed makeukus closed 10 months ago
hiyouga
commented
10 months ago 训练命令？
makeukus
commented
10 months ago CUDA_VISIBLE_DEVICES=0 python src/train_web.py 数据集 self_cognition 计算类型 bf16 学习率1e-4 最大样本数 1000 其余没动
训练命令？
hiyouga
commented
10 months ago 用 fp16 试试
makeukus
commented
10 months ago 用 fp16 试试
fp16 直接报错呢。
C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\site-packages\bitsandbytes\cextension.py:34: UserWarn ing: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable. warn("The installed version of bitsandbytes was compiled without GPU support. " 'NoneType' object has no attribute 'cadam32bit_grad_fp32' Running on local URL: http://0.0.0.0:7860
To create a public link, set share=True in launch().
Exception in thread Thread-6 (run_exp):
Traceback (most recent call last):
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\threading.py", line 1016, in _bootstrap_inner
self.run()
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\threading.py", line 953, in run
self._target(*self._args, self._kwargs)
File "D:\BaiduNetdiskDownload\ChatGLM3\LLaMA-Factory\src\llmtuner\tuner\tune.py", line 20, in run_exp
model_args, data_args, training_args, finetuning_args, generating_args = get_train_args(args)
File "D:\BaiduNetdiskDownload\ChatGLM3\LLaMA-Factory\src\llmtuner\tuner\core\parser.py", line 78, in get_train_args
model_args, data_args, training_args, finetuning_args, generating_args = parse_train_args(args)
File "D:\BaiduNetdiskDownload\ChatGLM3\LLaMA-Factory\src\llmtuner\tuner\core\parser.py", line 49, in parse_train_args
return _parse_args(parser, args)
File "D:\BaiduNetdiskDownload\ChatGLM3\LLaMA-Factory\src\llmtuner\tuner\core\parser.py", line 24, in _parse_args
return parser.parse_dict(args)
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\site-packages\transformers\hf_argparser.py",
line 373, in parse_dict
obj = dtype(inputs)
File "--fp16) and FP16 half precision evaluation (--fp16_full_e val) can only be used on CUDA or NPU devices or certain XPU devices (with IPEX).
hiyouga
commented
10 months ago cuda 环境没装成功
最终报错： [INFO|training_args.py:1345] 2023-11-12 17:25:17,617 >> Found safetensors installation, but --save_safetensors=False. Safetensors should be a preferred weights saving format due to security and performance reasons. If your model cannot be saved by safetensors please feel free to open an issue at https://github.com/huggingface/safetensors! [INFO|training_args.py:1798] 2023-11-12 17:25:17,617 >> PyTorch: setting up devices [INFO|trainer.py:1760] 2023-11-12 17:25:17,744 >> Running training [INFO|trainer.py:1761] 2023-11-12 17:25:17,744 >> Num examples = 348 [INFO|trainer.py:1762] 2023-11-12 17:25:17,744 >> Num Epochs = 3 [INFO|trainer.py:1763] 2023-11-12 17:25:17,745 >> Instantaneous batch size per device = 4 [INFO|trainer.py:1766] 2023-11-12 17:25:17,745 >> Total train batch size (w. parallel, distributed & accumulation) = 16 [INFO|trainer.py:1767] 2023-11-12 17:25:17,745 >> Gradient Accumulation steps = 4 [INFO|trainer.py:1768] 2023-11-12 17:25:17,745 >> Total optimization steps = 63 [INFO|trainer.py:1769] 2023-11-12 17:25:17,746 >> Number of trainable parameters = 1,949,696 Exception in thread Thread-9 (run_exp): Traceback (most recent call last): File "D:\ProgramData\anaconda3\envs\llama_factory\lib\threading.py", line 1016, in _bootstrap_inner self.run() File "D:\ProgramData\anaconda3\envs\llama_factory\lib\threading.py", line 953, in run self._target(self._args, self._kwargs) File "D:\BaiduNetdiskDownload\ChatGLM3\LLaMA-Factory\src\llmtuner\tuner\tune.py", line 26, in run_exp run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks) File "D:\BaiduNetdiskDownload\ChatGLM3\LLaMA-Factory\src\llmtuner\tuner\sft\workflow.py", line 67, in run_sft train_result = trainer.train(resume_from_checkpoint=training_args.resume_from_checkpoint) File "D:\ProgramData\anaconda3\envs\llama_factory\lib\site-packages\transformers\trainer.py", line 1591, in train return inner_training_loop( File "D:\ProgramData\anaconda3\envs\llama_factory\lib\site-packages\transformers\trainer.py", line 1892, in _inner_training_loop tr_loss_step = self.training_step(model, inputs) File "D:\ProgramData\anaconda3\envs\llama_factory\lib\site-packages\transformers\trainer.py", line 2776, in training_step loss = self.compute_loss(model, inputs) File "D:\ProgramData\anaconda3\envs\llama_factory\lib\site-packages\transformers\trainer.py", line 2801, in compute_loss outputs = model(inputs) File "D:\ProgramData\anaconda3\envs\llama_factory\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(args, kwargs) File "D:\ProgramData\anaconda3\envs\llama_factory\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, *kwargs) File "D:\ProgramData\anaconda3\envs\llama_factory\lib\site-packages\accelerate\utils\operations.py", line 659, in forward return model_forward(args, kwargs) File "D:\ProgramData\anaconda3\envs\llama_factory\lib\site-packages\accelerate\utils\operations.py", line 647, in call return convert_to_fp32(self.model_forward(*args, kwargs)) File "D:\ProgramData\anaconda3\envs\llama_factory\lib\site-packages\torch\amp\autocast_mode.py", line 16, in decorate_autocast return func(*args, *kwargs) File "D:\ProgramData\anaconda3\envs\llama_factory\lib\site-packages\peft\peft_model.py", line 977, in forward return self.base_model( File "D:\ProgramData\anaconda3\envs\llama_factory\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(args, kwargs) File "D:\ProgramData\anaconda3\envs\llama_factory\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, kwargs) File "D:\ProgramData\anaconda3\envs\llama_factory\lib\site-packages\peft\tuners\tuners_utils.py", line 106, in forward return self.model.forward(*args, *kwargs) File "C:\Users\Administrator/.cache\huggingface\modules\transformers_modules\THUDM_chatglm3-6b\modeling_chatglm.py", line 937, in forward transformer_outputs = self.transformer( File "D:\ProgramData\anaconda3\envs\llama_factory\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(args, kwargs) File "D:\ProgramData\anaconda3\envs\llama_factory\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, kwargs) File "C:\Users\Administrator/.cache\huggingface\modules\transformers_modules\THUDM_chatglm3-6b\modeling_chatglm.py", line 830, in forward hidden_states, presents, all_hidden_states, all_self_attentions = self.encoder( File "D:\ProgramData\anaconda3\envs\llama_factory\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "D:\ProgramData\anaconda3\envs\llama_factory\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(args, kwargs) File "C:\Users\Administrator/.cache\huggingface\modules\transformers_modules\THUDM_chatglm3-6b\modeling_chatglm.py", line 631, in forward layer_ret = torch.utils.checkpoint.checkpoint( File "D:\ProgramData\anaconda3\envs\llama_factory\lib\site-packages\torch_compile.py", line 24, in inner return torch._dynamo.disable(fn, recursive)(*args, kwargs) File "D:\ProgramData\anaconda3\envs\llama_factory\lib\site-packages\torch_dynamo\eval_frame.py", line 328, in _fn return fn(*args, *kwargs) File "D:\ProgramData\anaconda3\envs\llama_factory\lib\site-packages\torch_dynamo\external_utils.py", line 17, in inner return fn(args, kwargs) File "D:\ProgramData\anaconda3\envs\llama_factory\lib\site-packages\torch\utils\checkpoint.py", line 451, in checkpoint return CheckpointFunction.apply(function, preserve, args) File "D:\ProgramData\anaconda3\envs\llama_factory\lib\site-packages\torch\autograd\function.py", line 539, in apply return super().apply(args, kwargs) # type: ignore[misc] File "D:\ProgramData\anaconda3\envs\llama_factory\lib\site-packages\torch\utils\checkpoint.py", line 230, in forward outputs = run_function(args) File "D:\ProgramData\anaconda3\envs\llama_factory\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(args, kwargs) File "D:\ProgramData\anaconda3\envs\llama_factory\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, kwargs) File "C:\Users\Administrator/.cache\huggingface\modules\transformers_modules\THUDM_chatglm3-6b\modeling_chatglm.py", line 544, in forward attention_output, kv_cache = self.self_attention( File "D:\ProgramData\anaconda3\envs\llama_factory\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "D:\ProgramData\anaconda3\envs\llama_factory\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(args, kwargs) File "C:\Users\Administrator/.cache\huggingface\modules\transformers_modules\THUDM_chatglm3-6b\modeling_chatglm.py", line 441, in forward context_layer = self.core_attention(query_layer, key_layer, value_layer, attention_mask) File "D:\ProgramData\anaconda3\envs\llama_factory\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "D:\ProgramData\anaconda3\envs\llama_factory\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(args, **kwargs) File "C:\Users\Administrator/.cache\huggingface\modules\transformers_modules\THUDM_chatglm3-6b\modeling_chatglm.py", line 231, in forward context_layer = torch.nn.functional.scaled_dot_product_attention(query_layer, key_layer, value_layer, RuntimeError: Expected query, key, and value to have the same dtype, but got query.dtype: float key.dtype: float and value.dtype: struct c10::BFloat16 instead.
运行日志：
[INFO|training_args.py:1345] 2023-11-12 17:25:07,101 >> Found safetensors installation, but --save_safetensors=False. Safetensors should be a preferred weights saving format due to security and performance reasons. If your model cannot be saved by safetensors please feel free to open an issue at https://github.com/huggingface/safetensors!
[INFO|training_args.py:1798] 2023-11-12 17:25:07,102 >> PyTorch: setting up devices
[INFO|training_args.py:1519] 2023-11-12 17:25:07,103 >> The default value for the training argument --report_to will change in v5 (from all installed integrations to none). In v5, you will need to use --report_to all to get the same behavior as now. You should start updating your code and make this info disappear :-).
[WARNING|parser.py:159] 2023-11-12 17:25:07,107 >> ddp_find_unused_parameters needs to be set as False for LoRA in DDP training.
[INFO|training_args.py:1345] 2023-11-12 17:25:07,108 >> Found safetensors installation, but --save_safetensors=False. Safetensors should be a preferred weights saving format due to security and performance reasons. If your model cannot be saved by safetensors please feel free to open an issue at https://github.com/huggingface/safetensors!
[INFO|training_args.py:1798] 2023-11-12 17:25:07,109 >> PyTorch: setting up devices
[INFO|parser.py:189] 2023-11-12 17:25:07,109 >> Process rank: 0, device: cpu, n_gpu: 0 distributed training: True, compute dtype: torch.bfloat16
[INFO|parser.py:193] 2023-11-12 17:25:07,110 >> Training/evaluation parameters Seq2SeqTrainingArguments( _n_gpu=0, adafactor=False, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, auto_find_batch_size=False, bf16=True, bf16_full_eval=False, data_seed=None, dataloader_drop_last=False, dataloader_num_workers=0, dataloader_pin_memory=True, ddp_backend=None, ddp_broadcast_buffers=None, ddp_bucket_cap_mb=None, ddp_find_unused_parameters=False, ddp_timeout=1800, debug=[], deepspeed=None, disable_tqdm=True, dispatch_batches=None, do_eval=False, do_predict=False, do_train=True, eval_accumulation_steps=None, eval_delay=0, eval_steps=None, evaluation_strategy=no, fp16=False, fp16_backend=auto, fp16_full_eval=False, fp16_opt_level=O1, fsdp=[], fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_grad_ckpt': False}, fsdp_min_num_params=0, fsdp_transformer_layer_cls_to_wrap=None, full_determinism=False, generation_config=None, generation_max_length=None, generation_num_beams=None, gradient_accumulation_steps=4, gradient_checkpointing=False, greater_is_better=None, group_by_length=False, half_precision_backend=auto, hub_always_push=False, hub_model_id=None, hub_private_repo=False, hub_strategy=every_save, hub_token=, ignore_data_skip=False, include_inputs_for_metrics=False, include_tokens_per_second=False, jit_mode_eval=False, label_names=None, label_smoothing_factor=0.0, learning_rate=0.0001, length_column_name=length, load_best_model_at_end=False, local_rank=0, log_level=passive, log_level_replica=warning, log_on_each_node=True, logging_dir=saves\ChatGLM3-6B-Chat\lora\2023-11-12-17-20-02\runs\Nov12_17-25-07_DESKTOP-ARVDQJM, logging_first_step=False, logging_nan_inf_filter=True, logging_steps=5, logging_strategy=steps, lr_scheduler_type=cosine, max_grad_norm=1.0, max_steps=-1, metric_for_best_model=None, mp_parameters=, no_cuda=False, num_train_epochs=3.0, optim=adamw_torch, optim_args=None, output_dir=saves\ChatGLM3-6B-Chat\lora\2023-11-12-17-20-02, overwrite_output_dir=False, past_index=-1, per_device_eval_batch_size=8, per_device_train_batch_size=4, predict_with_generate=False, prediction_loss_only=False, push_to_hub=False, push_to_hub_model_id=None, push_to_hub_organization=None, push_to_hub_token=, ray_scope=last, remove_unused_columns=True, report_to=[], resume_from_checkpoint=None, run_name=saves\ChatGLM3-6B-Chat\lora\2023-11-12-17-20-02, save_on_each_node=False, save_safetensors=False, save_steps=100, save_strategy=steps, save_total_limit=None, seed=42, sharded_ddp=[], skip_memory_metrics=True, sortish_sampler=False, tf32=None, torch_compile=False, torch_compile_backend=None, torch_compile_mode=None, torchdynamo=None, tpu_metrics_debug=False, tpu_num_cores=None, use_cpu=False, use_ipex=False, use_legacy_prediction_loop=False, use_mps_device=False, warmup_ratio=0.0, warmup_steps=0, weight_decay=0.0, )
[INFO|loader.py:25] 2023-11-12 17:25:07,112 >> Loading dataset self_cognition.json...
[WARNING|utils.py:37] 2023-11-12 17:25:07,112 >> Checksum failed: mismatched SHA-1 hash value at data\self_cognition.json.
[INFO|builder.py:573] 2023-11-12 17:25:09,905 >> Using custom data configuration default-ed6af288972ae3a8
[INFO|info.py:437] 2023-11-12 17:25:09,905 >> Loading Dataset Infos from D:\ProgramData\anaconda3\envs\llama_factory\lib\site-packages\datasets\packaged_modules\json
[INFO|builder.py:419] 2023-11-12 17:25:09,907 >> Overwrite dataset info from restored data version if exists.
[INFO|info.py:352] 2023-11-12 17:25:09,907 >> Loading Dataset info from C:\Users\Administrator.cache\huggingface\datasets\json\default-ed6af288972ae3a8\0.0.0\8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96
[INFO|builder.py:882] 2023-11-12 17:25:09,915 >> Found cached dataset json (C:/Users/Administrator/.cache/huggingface/datasets/json/default-ed6af288972ae3a8/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96)
[INFO|info.py:352] 2023-11-12 17:25:09,915 >> Loading Dataset info from C:/Users/Administrator/.cache/huggingface/datasets/json/default-ed6af288972ae3a8/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96
[INFO|tokenization_utils_base.py:2013] 2023-11-12 17:25:09,926 >> loading file tokenizer.model
[INFO|tokenization_utils_base.py:2013] 2023-11-12 17:25:09,926 >> loading file added_tokens.json
[INFO|tokenization_utils_base.py:2013] 2023-11-12 17:25:09,926 >> loading file special_tokens_map.json
[INFO|tokenization_utils_base.py:2013] 2023-11-12 17:25:09,927 >> loading file tokenizer_config.json
[INFO|tokenization_utils_base.py:2013] 2023-11-12 17:25:09,927 >> loading file tokenizer.json
[INFO|configuration_utils.py:713] 2023-11-12 17:25:10,023 >> loading configuration file models\THUDM_chatglm3-6b\config.json
[INFO|configuration_utils.py:713] 2023-11-12 17:25:10,025 >> loading configuration file models\THUDM_chatglm3-6b\config.json
[INFO|configuration_utils.py:775] 2023-11-12 17:25:10,025 >> Model config ChatGLMConfig { "_name_or_path": "models\THUDM_chatglm3-6b", "add_bias_linear": false, "add_qkv_bias": true, "apply_query_key_layer_scaling": true, "apply_residual_connection_post_layernorm": false, "architectures": [ "ChatGLMModel" ], "attention_dropout": 0.0, "attention_softmax_in_fp32": true, "auto_map": { "AutoConfig": "configuration_chatglm.ChatGLMConfig", "AutoModel": "modeling_chatglm.ChatGLMForConditionalGeneration", "AutoModelForCausalLM": "modeling_chatglm.ChatGLMForConditionalGeneration", "AutoModelForSeq2SeqLM": "modeling_chatglm.ChatGLMForConditionalGeneration", "AutoModelForSequenceClassification": "modeling_chatglm.ChatGLMForSequenceClassification" }, "bias_dropout_fusion": true, "classifier_dropout": null, "eos_token_id": 2, "ffn_hidden_size": 13696, "fp32_residual_connection": false, "hidden_dropout": 0.0, "hidden_size": 4096, "kv_channels": 128, "layernorm_epsilon": 1e-05, "model_type": "chatglm", "multi_query_attention": true, "multi_query_group_num": 2, "num_attention_heads": 32, "num_layers": 28, "original_rope": true, "pad_token_id": 0, "padded_vocab_size": 65024, "post_layer_norm": true, "pre_seq_len": null, "prefix_projection": false, "quantization_bit": 0, "rmsnorm": true, "seq_length": 8192, "tie_word_embeddings": false, "torch_dtype": "float16", "transformers_version": "4.34.1", "use_cache": true, "vocab_size": 65024 }
[INFO|modeling_utils.py:2990] 2023-11-12 17:25:10,029 >> loading weights file models\THUDM_chatglm3-6b\pytorch_model.bin.index.json
[INFO|modeling_utils.py:1220] 2023-11-12 17:25:10,030 >> Instantiating ChatGLMForConditionalGeneration model under default dtype torch.bfloat16.
[INFO|configuration_utils.py:770] 2023-11-12 17:25:10,030 >> Generate config GenerationConfig { "eos_token_id": 2, "pad_token_id": 0 }