Open 18065013 opened 1 year ago
(vicuna_training_310) root@ubuntu-3090x2:/home/huwei/training/Chinese-Vicuna# python finetune_chat.py --data_path sample/merge_split_s1.json --model_path decapoda-research/llama-7b-hf --test_size=10
The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization. The tokenizer class you load from this checkpoint is 'LLaMATokenizer'. The class this function is called from is 'LlamaTokenizer'. normalizer.cc(51) LOG(INFO) precompiled_charsmap is empty. use identity normalization. The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization. The tokenizer class you load from this checkpoint is 'LLaMATokenizer'. The class this function is called from is 'LlamaTokenizer'. normalizer.cc(51) LOG(INFO) precompiled_charsmap is empty. use identity normalization. Using custom data configuration default-9e71a4e6f9c8d1a3 Found cached dataset json (/root/.cache/huggingface/datasets/json/default-9e71a4e6f9c8d1a3/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51) 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 759.84it/s] 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 191.43ex/s] Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 33/33 [00:10<00:00, 3.13it/s] normalizer.cc(51) LOG(INFO) precompiled_charsmap is empty. use identity normalization. normalizer.cc(51) LOG(INFO) precompiled_charsmap is empty. use identity normalization.
normalizer.cc(51) LOG(INFO) precompiled_charsmap is empty. use identity normalization.
The following columns in the training set don't have a corresponding argument in PeftModelForCausalLM.forward
and have been ignored: input, output. If input, output are not expected by PeftModelForCausalLM.forward
, you can safely ignore this message.████████████████████████████████████████▉ | 608/625 [00:03<00:00, 202.11ex/s]
/root/anaconda3/envs/vicuna_training_310/lib/python3.10/site-packages/transformers/optimization.py:391: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set no_deprecation_warning=True
to disable this warning00:00, 209.69ex/s]
warnings.warn(
Running training
Num examples = 2500
Num Epochs = 3
Instantaneous batch size per device = 4
Total train batch size (w. parallel, distributed & accumulation) = 256
Gradient Accumulation steps = 32
Total optimization steps = 19
Number of trainable parameters = 19988480
0%| | 0/19 [00:00<?, ?it/s]/root/anaconda3/envs/vicuna_training_310/lib/python3.10/site-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
warnings.warn('Was asked to gather along dimension 0, but all '
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/huwei/training/Chinese-Vicuna/finetune_chat.py:273 in
单卡情况下能跑通并且正常训练,未改过代码在多卡时报错,求解 mat1 and mat2 shapes cannot be multiplied (1024x2 and 1x11008)