search

microsoft / SwinBERT

Research code for CVPR 2022 paper "SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning"
https://arxiv.org/abs/2111.13196
MIT License
237 stars 35 forks source link

gpu memory requirements? #27

Closed mhyeonsoo closed 2 years ago

mhyeonsoo commented 2 years ago

Hi, thanks for the great source.

I am trying to do train with the youcook dataset, and facing the error related to gpu resource. I am using 3080Ti 10GiB single gpu, and modify the training command by chaing batch size as below.

python src/tasks/run_caption_VidSwinBert.py \
        --config src/configs/VidSwinBert/youcook2_8frm_default.json \
        --train_yaml YouCook2/training_32frames.yaml \
        --val_yaml YouCook2/validation_32frames.yaml \
        --per_gpu_train_batch_size 1 \
        --per_gpu_eval_batch_size 1 \
        --num_train_epochs 40 \
        --learning_rate 0.0003 \
        --max_num_frames 32 \
        --pretrained_2d 0 \
        --backbone_coef_lr 0.05 \
        --mask_prob 0.5 \
        --max_masked_token 45 \
        --zero_opt_stage 1 \
        --mixed_precision_method deepspeed \
        --deepspeed_fp16 \
        --gradient_accumulation_steps 4 \
        --learn_mask_enabled \
        --loss_sparse_w 0.5 \
        --output_dir ./output

Although I set up a batch size as 1 for training, the command returns the error.

CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 9.78 GiB total capacity; 7.25 GiB already allocated; 40.00 MiB free; 7.44 GiB reserved in total by PyTorch)

Can I ask you what is the gpu resource minimum requirements? Thanks,

tiesanguaixia commented 1 year ago

Hi, thanks for the great source.

I am trying to do train with the youcook dataset, and facing the error related to gpu resource. I am using 3080Ti 10GiB single gpu, and modify the training command by chaing batch size as below.

python src/tasks/run_caption_VidSwinBert.py \
        --config src/configs/VidSwinBert/youcook2_8frm_default.json \
        --train_yaml YouCook2/training_32frames.yaml \
        --val_yaml YouCook2/validation_32frames.yaml \
        --per_gpu_train_batch_size 1 \
        --per_gpu_eval_batch_size 1 \
        --num_train_epochs 40 \
        --learning_rate 0.0003 \
        --max_num_frames 32 \
        --pretrained_2d 0 \
        --backbone_coef_lr 0.05 \
        --mask_prob 0.5 \
        --max_masked_token 45 \
        --zero_opt_stage 1 \
        --mixed_precision_method deepspeed \
        --deepspeed_fp16 \
        --gradient_accumulation_steps 4 \
        --learn_mask_enabled \
        --loss_sparse_w 0.5 \
        --output_dir ./output

Although I set up a batch size as 1 for training, the command returns the error.

CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 9.78 GiB total capacity; 7.25 GiB already allocated; 40.00 MiB free; 7.44 GiB reserved in total by PyTorch)

Can I ask you what is the gpu resource minimum requirements? Thanks,

Hi! May I ask how to download the raw videos of YouCook2? I found that in SwinBERT annotation, the video path is like datasets/YouCook2/training/A68bBM_5wUY_11.mp4, but the raw video of YouCook2 does not has a clip id like _11. So how to generate the .mp4 videos for YouCook2 training? I really look forward to your reply, thank you in advance!