Open wangyin717 opened 2 months ago
是解决了吗,没有的话可以发一下配置的yaml文件,然后也可以检查下权重是否下载完全
这是我的sft.yaml配置文件,在这里我尝试用单卡A100训练,所以改了run下面的参数,训练运行 CUDA_VISIBLE_DEVICES=0 python train.py --cfg-path lavis/projects/pp_qwen7b_video/sft.yaml
然后会出现上述Missing keys的问题,最后会报错显存不足(单张A100肯定显存不够)。此外model.pth权重检查过应该没问题,所以Missing keys按理说不应该出现。
sft.yaml:
model:
arch: minigpt4qwen
model_type: qwen7b_chat
load_finetuned: True
load_pretrained: True
# pretrained: "https://storage.googleapis.com/sfr-vision-language-research/LAVIS/models/InstructBLIP/blip2_pretrained_flant5xxl.pth"
pretrained: "/data/MiniGPT4Qwen/cache/ckpt/blip2/blip2_pretrained_flant5xxl.pth"
# finetuned: ""
finetuned: "/data/MiniGPT4Qwen/lavis/output/pp_7b_video/pretrain/global_step2181/model.pth"
# vit encoder
vit_model: "eva_clip_g"
image_size: 224
drop_path_rate: 0
use_grad_checkpoint: True
vit_precision: "fp16" # 如果你要打开vit进行训练,这里需要调整成fp32,否则如果开启amp混合精度训练会有问题(在scaler处报错,因为没有实现一个fp16的AdamW)
freeze_vit: True
unfreeze_pos_embed: False
# Q-Former
num_query_token: 32
qformer_text_input: False
freeze_qformer: True
freeze_queries: True
# projection
freeze_proj: False
# path to Vicuna checkpoint
llm_model: "/data/MiniGPT4Qwen/cache/ckpt/Qwen7B-chat"
# unfreeze LLM for better chat
freeze_llm: False
# lora config
get_lora: False
lora_alpha: 32
lora_r: 8
lora_dropout: 0.05
# text length when training
max_txt_len: 1536 # 512
# enable autocast of vit
enable_autocast: False
datasets:
llava_instruct_156k: # name of the dataset builder
vis_processor:
train:
name: "blip2_image_train"
image_size: 224
text_processor:
train:
name: "base_instruction"
max_words: 200
videochatgpt_100k: # name of the dataset builder
vis_processor:
train:
name: "blip2_image_train"
image_size: 224
text_processor:
train:
name: "base_instruction"
max_words: 200
# run:
# output_dir: "lavis/output/pp_7b_video/sft_video/"
# # task: deepspeed_image_text_pretrain
# task: deepspeed_image_text_pretrain
# num_workers: 4
# seed: 42
# world_size: 1
# dist_url: "env://"
# distributed: True
# max_epoch: 1
# log_freq: 10
# lr_sched: "linear_warmup_cosine_lr_step-wise"
# warmup_lr: 0
# init_lr: 2e-5
# min_lr: 0
# warmup_ratio: 0.1
# deepspeed_config:
# # global batch = 128 = n_ranks * grad_acc_steps * micro_batch_size = (4//2) * 64 * 1
# # 8 x 3090
# # pp=8 dp=1 nproc=pp*dp=8
# gradient_accumulation_steps: 128 # 128 // dp(=1) // bs_per_gpu(=1) = 128
# train_micro_batch_size_per_gpu: 1
# gradient_clipping: 1.
# steps_per_print: 10
# wall_clock_breakdown: false
# dump_state: False
# fp16:
# enabled: false
# loss_scale: 0
# loss_scale_window: 1000
# initial_scale_power: 16
# hysteresis: 2
# min_loss_scale: 1
# bf16:
# enabled: true
# optimizer:
# type: "AdamW"
# params:
# lr: 2e-5
# betas: [0.9,0.99]
# eps: 1e-7
# weight_decay: 0.
# zero_optimization:
# stage: 0
# # offload_optimizer:
# # device: "cpu"
# # pin_memory: true
# allgather_partitions: true
# allgather_bucket_size: 2e8
# overlap_comm: true
# reduce_scatter: true
# reduce_bucket_size: 2e8
# contiguous_gradients: true
run:
task: image_text_pretrain
# optimizer
lr_sched: "linear_warmup_cosine_lr"
init_lr: 2e-5
min_lr: 1e-6
warmup_lr: 0
warmup_steps: 500
weight_decay: 0.05
grad_norm_clip: 1.
max_epoch: 10 #5
batch_size_train: 1 #16
batch_size_eval: 1
num_workers: 4
accum_grad_iters: 16 #1
seed: 42
output_dir: "lavis/output/pp_7b_video/pretrain/"
log_freq: 64
amp: True
autocast_dtype: "bf16" # ['bf16','fp16']
loss_scale: False # 默认为True,如果autocast_dtype为float16,请设置为True,为bfloat16,建议设置为False
resume_ckpt_path: null
evaluate: False
train_splits: ["train"]
# valid_splits: ["val"]
# test_splits: ["test"]
device: "cuda"
world_size: 1
dist_url: "env://"
distributed: True
# fsdp: False
看了下,应该不是报错,https://github.com/Coobiw/MiniGPT4Qwen/blob/master/lavis/models/base_model.py#L53
msg = self.load_state_dict(state_dict, strict=False)
logging.info("Missing keys {}".format(msg.missing_keys))
这里只是一个log,因为model.pth里只有中间的projection层的参数(第一步pretrain stage只训练中间projection层),其他都用的eva,blip2_qformer和qwen-7B的参数,所以会有一个INFO的提醒,按照 https://github.com/Coobiw/MiniGPT4Qwen/blob/master/WEIGHT.md 中下载权重就好
担心的话先训几步看看loss,显存不够的话可以把freeze_llm
设为True
先看下
报错信息: Using downloaded and verified file: /data/MiniGPT4Qwen/lavis/../cache/dataset/llava_instruct/llava_instruction_156k.json 2024-07-02 14:04:21,365 [INFO] Building datasets... Using downloaded and verified file: /data/MiniGPT4Qwen/lavis/../cache/dataset/videochatgpt/videochatgpt_instruction_100k.json 2024-07-02 14:04:22,514 [INFO] Building datasets... Finishing Initializing Vision-Encoder... 2024-07-02 14:04:35,207 [INFO] freeze vision encoder Finishing Loading Q-former Initializing Config... Finishing Initializing Q-former... 2024-07-02 14:04:35,917 [INFO] no text input for q-former Loading LLM:/data/MiniGPT4Qwen/cache/ckpt/Qwen7B-chat... 2024-07-02 14:04:36,396 [WARNING] The model is automatically converting to bf16 for faster inference. If you want to disable the automatic precision, please manually add bf16/fp16/fp32=True to "AutoModelForCausalLM.from_pretrained". 2024-07-02 14:04:36,397 [WARNING] Try importing flash-attention for faster inference... 2024-07-02 14:04:36,397 [WARNING] Warning: import flash_attn rotary fail, please install FlashAttention rotary to get higher efficiency https://github.com/Dao-AILab/flash-attention/tree/main/csrc/rotary 2024-07-02 14:04:36,397 [WARNING] Warning: import flash_attn rms_norm fail, please install FlashAttention layer_norm to get higher efficiency https://github.com/Dao-AILab/flash-attention/tree/main/csrc/layer_norm 2024-07-02 14:04:36,397 [WARNING] Warning: import flash_attn fail, please install FlashAttention to get higher efficiency https://github.com/Dao-AILab/flash-attention Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:06<00:00, 1.16it/s] Unfreeze LLM!!! Start loading pretrained model: /data/MiniGPT4Qwen/cache/ckpt/blip2/blip2_pretrained_flant5xxl.pth Loading the File Named: /data/MiniGPT4Qwen/cache/ckpt/blip2/blip2_pretrained_flant5xxl.pth... 2024-07-02 14:04:43,919 [INFO] load checkpoint from /data/MiniGPT4Qwen/cache/ckpt/blip2/blip2_pretrained_flant5xxl.pth Start loading finetuned model: /data/MiniGPT4Qwen/lavis/output/ckpt-and-data/pretrain/global_step2181/model.pth Checkpoint: /data/MiniGPT4Qwen/lavis/output/ckpt-and-data/pretrain/global_step2181/model.pth
################################################### 在这里读取预训练模型model.pth时,报错提示Missing keys ###################################################
2024-07-02 14:04:43,958 [INFO] Missing keys ['query_tokens', 'visual_encoder.cls_token', 'visual_encoder.pos_embed', 'visual_encoder.patch_embed.proj.weight', 'visual_encoder.patch_embed.proj.bias', 'visual_encoder.blocks.0.norm1.weight', 'visual_encoder.blocks.0.norm1.bias', 'visual_encoder.blocks.0.attn.q_bias', 'visual_encoder.blocks.0.attn.v_bias', 'visual_encoder.blocks.0.attn.qkv.weight', 'visual_encoder.blocks.0.attn.proj.weight', 'visual_encoder.blocks.0.attn.proj.bias', 'visual_encoder.blocks.0.norm2.weight', ......后面还有很多
请问是什么原因导致的,谢谢!