search

hiyouga / LLaMA-Factory

Unified Efficient Fine-Tuning of 100+ LLMs (ACL 2024)
https://arxiv.org/abs/2403.13372
Apache License 2.0
33.07k stars 4.07k forks source link

是否支持yuan2 #3260

Closed bgtii closed 6 months ago

bgtii commented 6 months ago

Reminder

Reproduction

脚本 CUDA_VISIBLE_DEVICES=4 python src/train_bash.py \ --stage sft \ --model_name_or_path /root/.cache/modelscope/hub/YuanLLM/Yuan2-2B-Janus-hf \ --dataset 肾科ai生成1000问 \ --template yuan \ --finetuning_type lora \ --lora_target q_proj,v_proj \ --output_dir /root/workspace/LLaMA-Factory-main/output_checkpoint/yuan_肾科千问 \ --overwrite_cache \ --per_device_train_batch_size 2 \ --gradient_accumulation_steps 4 \ --lr_scheduler_type cosine \ --logging_steps 10 \ --save_steps 100 \ --learning_rate 5e-4 \ --num_train_epochs 100.0 \ --plot_loss \ --fp16 \

日志： [2024-04-13 17:22:34,193] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect) 04/13/2024 17:22:35 - INFO - llmtuner.hparams.parser - Process rank: 0, device: cuda:0, n_gpu: 1, distributed training: False, compute dtype: torch.float16 [INFO|tokenization_utils_base.py:2082] 2024-04-13 17:22:35,681 >> loading file tokenizer.model [INFO|tokenization_utils_base.py:2082] 2024-04-13 17:22:35,681 >> loading file added_tokens.json [INFO|tokenization_utils_base.py:2082] 2024-04-13 17:22:35,681 >> loading file special_tokens_map.json [INFO|tokenization_utils_base.py:2082] 2024-04-13 17:22:35,681 >> loading file tokenizer_config.json [INFO|tokenization_utils_base.py:2082] 2024-04-13 17:22:35,681 >> loading file tokenizer.json [WARNING|logging.py:329] 2024-04-13 17:22:35,681 >> You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This is expected, and simply means that the legacy (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set legacy=False. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 04/13/2024 17:22:36 - INFO - llmtuner.data.template - Replace eos token: 04/13/2024 17:22:36 - INFO - llmtuner.data.template - Add pad token: 04/13/2024 17:22:36 - INFO - llmtuner.data.template - Cannot add this chat template to tokenizer. 04/13/2024 17:22:36 - INFO - llmtuner.data.loader - Loading dataset 肾科ai生成1000问.json... 04/13/2024 17:22:36 - WARNING - llmtuner.data.utils - Checksum failed: missing SHA-1 hash value in dataset_info.json. Converting format of dataset: 100%|██████████████████████████████████████████████████████████████████████████████| 997/997 [00:00<00:00, 46877.13 examples/s] Running tokenizer on dataset: 100%|███████████████████████████████████████████████████████████████████████████████| 997/997 [00:00<00:00, 2181.24 examples/s] input_ids: [29871, 37768, 36513, 33315, 30882, 77187, 29871, 36513, 33315, 34747, 34855, 48807, 30503, 34855, 41329, 31429, 30687, 39616, 30214, 43025, 34439, 47580, 30214, 35709, 32796, 32004, 53836, 49788, 30210, 38466, 33809, 30267, 43559, 36513, 75267, 36248, 30503, 32369, 32280, 38095, 33793, 33111, 33982, 30383, 29896, 29889, 36513, 75267, 30783, 39290, 31436, 34421, 30210, 51386, 30952, 32617, 31608, 29906, 29889, 50687, 30783, 30716, 30330, 51048, 32960, 30330, 58311, 34434, 31436, 43661, 35220, 32369, 35823, 31608, 29941, 29889, 36513, 75267, 67955, 32369, 36326, 30267, 44933, 36513, 75267, 36513, 33315, 31558, 32227, 32725, 32672, 30214, 32227, 31101, 48327, 37966, 40112, 30214, 37792, 47390, 47502, 30267, 32110, 30417, 32169, 36106, 30214, 39290, 36106, 30214, 53836, 43063, 31436, 53294, 30214, 34390, 33707, 42818, 33870, 36513, 32369, 33394, 30267, 32271, 43025, 30417, 44933, 34414, 74042, 41558, 70383, 43677, 30210, 30658, 37167, 34942, 30911, 30214, 36314, 30505, 34942, 30822, 29896, 29899, 29946, 31109, 30822, 41329, 30267, 77185] inputs: 什么是肾炎？ 肾炎是由多种病因和多种发病机理引起的，病例类型各异，临床表现又常有重叠的一组疾病。患者的肾小球形态和功能发生损伤具备以下特点：1.肾小球对蛋白及细胞的通透性改变；2.肾脏对水、电解质、酸碱平衡及血压调节功能障碍；3.肾小球滤过功能损害。急性肾小球肾炎起病较急，病程一般在三个月以内，病情轻重不一。一般有血尿，蛋白尿，常有高血压及水肿，有时会有短暂性的肾功能下降。部分病例有急性链球菌或其他病原微生物的前驱感染史，多数在感染后1-4周后发病。 label_ids: [-100, -100, -100, -100, -100, -100, 29871, 36513, 33315, 34747, 34855, 48807, 30503, 34855, 41329, 31429, 30687, 39616, 30214, 43025, 34439, 47580, 30214, 35709, 32796, 32004, 53836, 49788, 30210, 38466, 33809, 30267, 43559, 36513, 75267, 36248, 30503, 32369, 32280, 38095, 33793, 33111, 33982, 30383, 29896, 29889, 36513, 75267, 30783, 39290, 31436, 34421, 30210, 51386, 30952, 32617, 31608, 29906, 29889, 50687, 30783, 30716, 30330, 51048, 32960, 30330, 58311, 34434, 31436, 43661, 35220, 32369, 35823, 31608, 29941, 29889, 36513, 75267, 67955, 32369, 36326, 30267, 44933, 36513, 75267, 36513, 33315, 31558, 32227, 32725, 32672, 30214, 32227, 31101, 48327, 37966, 40112, 30214, 37792, 47390, 47502, 30267, 32110, 30417, 32169, 36106, 30214, 39290, 36106, 30214, 53836, 43063, 31436, 53294, 30214, 34390, 33707, 42818, 33870, 36513, 32369, 33394, 30267, 32271, 43025, 30417, 44933, 34414, 74042, 41558, 70383, 43677, 30210, 30658, 37167, 34942, 30911, 30214, 36314, 30505, 34942, 30822, 29896, 29899, 29946, 31109, 30822, 41329, 30267, 77185] labels: 肾炎是由多种病因和多种发病机理引起的，病例类型各异，临床表现又常有重叠的一组疾病。患者的肾小球形态和功能发生损伤具备以下特点：1.肾小球对蛋白及细胞的通透性改变；2.肾脏对水、电解质、酸碱平衡及血压调节功能障碍；3.肾小球滤过功能损害。急性肾小球肾炎起病较急，病程一般在三个月以内，病情轻重不一。一般有血尿，蛋白尿，常有高血压及水肿，有时会有短暂性的肾功能下降。部分病例有急性链球菌或其他病原微生物的前驱感染史，多数在感染后1-4周后发病。 [INFO|configuration_utils.py:724] 2024-04-13 17:22:37,719 >> loading configuration file /root/.cache/modelscope/hub/YuanLLM/Yuan2-2B-Janus-hf/config.json [INFO|configuration_utils.py:724] 2024-04-13 17:22:37,721 >> loading configuration file /root/.cache/modelscope/hub/YuanLLM/Yuan2-2B-Janus-hf/config.json [INFO|configuration_utils.py:789] 2024-04-13 17:22:37,722 >> Model config YuanConfig { "_from_model_config": true, "_name_or_path": "/root/.cache/modelscope/hub/YuanLLM/Yuan2-2B-Janus-hf", "architectures": [ "YuanForCausalLM" ], "auto_map": { "AutoConfig": "configuration_yuan.YuanConfig", "AutoModelForCausalLM": "yuan_hf_model.YuanForCausalLM" }, "bos_token_id": 77185, "causal_mask": true, "dropout": 0.1, "eod_token": 77185, "eod_token_id": 77185, "eos_token_id": 77185, "hidden_act": "silu", "hidden_size": 2048, "initializer_range": 0.02, "intermediate_size": 8192, "mask_token_id": 77185, "max_position_embeddings": 8192, "model_max_length": 8192, "model_type": "yuan", "num_attention_heads": 32, "num_hidden_layers": 24, "pad_token_id": 77185, "reset_attention_mask": true, "reset_position_ids": true, "rms_norm_eps": 1e-06, "sep_token": 77187, "sep_token_id": 77185, "tokenizer_class": "YuanTokenizer", "torch_dtype": "bfloat16", "transformers_version": "4.39.2", "use_cache": true, "use_flash_attention": true, "use_loss_mask": false, "vocab_size": 135040 }

04/13/2024 17:22:37 - INFO - llmtuner.model.patcher - Using KV cache for faster generation. [INFO|modeling_utils.py:3280] 2024-04-13 17:22:37,754 >> loading weights file /root/.cache/modelscope/hub/YuanLLM/Yuan2-2B-Janus-hf/pytorch_model.bin [INFO|modeling_utils.py:1417] 2024-04-13 17:22:37,793 >> Instantiating YuanForCausalLM model under default dtype torch.float16. [INFO|configuration_utils.py:928] 2024-04-13 17:22:37,794 >> Generate config GenerationConfig { "bos_token_id": 77185, "eos_token_id": 77185, "pad_token_id": 77185 }

[INFO|modeling_utils.py:4024] 2024-04-13 17:22:40,336 >> All model checkpoint weights were used when initializing YuanForCausalLM.

[INFO|modeling_utils.py:4032] 2024-04-13 17:22:40,337 >> All the weights of YuanForCausalLM were initialized from the model checkpoint at /root/.cache/modelscope/hub/YuanLLM/Yuan2-2B-Janus-hf. If your task is similar to the task the model of the checkpoint was trained on, you can already use YuanForCausalLM for predictions without further training. [INFO|configuration_utils.py:881] 2024-04-13 17:22:40,339 >> loading configuration file /root/.cache/modelscope/hub/YuanLLM/Yuan2-2B-Janus-hf/generation_config.json [INFO|configuration_utils.py:928] 2024-04-13 17:22:40,340 >> Generate config GenerationConfig { "bos_token_id": 77185, "eos_token_id": 77185, "pad_token_id": 77185 }

04/13/2024 17:22:40 - INFO - llmtuner.model.adapter - Adapter is not found at evaluation, load the base model. 04/13/2024 17:22:40 - INFO - llmtuner.model.loader - all params: 2088724480 /root/miniconda3/envs/llama_factory/lib/python3.10/site-packages/accelerate/accelerator.py:432: FutureWarning: Passing the following arguments to Accelerator is deprecated and will be removed in version 1.0 of Accelerate: dict_keys(['dispatch_batches', 'split_batches', 'even_batches', 'use_seedable_sampler']). Please pass an accelerate.DataLoaderConfiguration instead: dataloader_config = DataLoaderConfiguration(dispatch_batches=None, split_batches=False, even_batches=True, use_seedable_sampler=True) warnings.warn( [INFO|trainer.py:607] 2024-04-13 17:22:40,383 >> Using auto half precision backend

Expected behavior

No response

System Info

No response

Others

No response

marko1616 commented 6 months ago

Yes?

bgtii commented 6 months ago

不清楚为什么跑不起来