v-iashin
commented
3 years ago Hi, thank you for the warm words!
- I think it is not a problem to use videos with a different language of narration. Audio information contains not only speech but also sounds of actions, background, etc. Therefore, I wouldn't worry about it too much unless I run an experiment.
- To use visual features only, you may specify
--model transformer(which by default isav_trasformer). Here is the script we used for the ablation study (use it as guidance because I don't expect it to work without some error – I refactored code a bit after that, so some arguments' names might be different etc)
Pre-training the captioning module
$env_python scripts/train_captioning_module.py \
--train_meta_path ./data/train_meta.csv \
--val_1_meta_path ./data/val_1_meta.csv \
--val_2_meta_path ./data/val_2_meta.csv \
--video_feature_name i3d \
--video_features_path ./data/i3d_25fps_stack64step64_2stream_npy \
--log_dir ./logs/ablation/V_only_cap \
--d_vid 1024 \
--d_model_video 1024 \
--d_model_caps 300 \
--d_model 1024 \
--model transformer \
--modality video \
--optimizer adam \
--dout_p 0.1 \
--N 2 \
--smoothing 0.7 \
--lr 5e-5 \
--B 32 \
--one_by_one_starts_at 1 \
--betas 0.9 0.999 \
--H 4 \
--word_emb_caps glove.840B.300d \
--device_ids 0Training proposal generator with pre-trained encoder from the captioning module
$env_python scripts/train_proposal_generator.py \
--train_json_path ./data/train.json \
--train_meta_path ./data/train_meta.csv \
--val_1_meta_path ./data/val_1_meta.csv \
--val_2_meta_path ./data/val_2_meta.csv \
--log_dir ./logs/ablation/V_only_with_pretr_on_cap_V \
--video_feature_name i3d \
--video_features_path ./data/i3d_25fps_stack64step64_2stream_npy \
--feature_timespan_in_fps 64 \
--fps_at_extraction 25 \
--modality video \
--pretrained_cap_model_path ./logs/ablation/V_only_cap/best_model.pt \
--loc_model transformer \
--d_vid 1024 \
--d_model_caps 1024 \
--d_model 1024 \
--dout_p 0.1 \
--noobj_coeff 100 \
--smoothing 0.7 \
--epoch_num 70 \
--obj_coeff 1 \
--one_by_one_starts_at 5000 \
--N 2 \
--B 16 \
--inf_B_coeff 2 \
--lr 5e-5 \
--pad_video_feats_up_to 300 \
--max_prop_per_vid 100 \
--optimizer adam \
--betas 0.9 0.999 \
--anchors_num_video 128 \
--kernel_sizes_video 1 5 9 13 19 25 35 45 61 79 \
--conv_layers_video 512 512 \
--device_ids 0
amil-rp-work
Hey @v-iashin Thanks for open sourcing such an awesome work!!! Kudos to you on this and MDVC. I was wondering since my videos are not of English language but I do require captions in the English language, how do I go about utilizing your work here? Is there a possibility to totally ignore audio features and just image features?