search

qinghuannn / PAMFN

[TIP 2024] Official Implementation of Progressive Adaptive Multimodal Fusion Network (PAMFN)
MIT License
6 stars 0 forks source link

Feature extraction question #2

Open lexcynthia opened 3 days ago

lexcynthia commented 3 days ago

Could you please provide the files for extracting RGB, optical flow, and audio features?

qinghuannn commented 3 days ago

The final code for extracting all features has been lost. However, we managed to locate the earlier versions of the code for extracting VST, I3D, and AST features. We hope this will be helpful to you.

Please note that you may need to remove unused model imports, download the VST code separately, and ensure that all required checkpoints are downloaded.

Below are the commands for feature extraction:

# RG dataset
# extract RGB features
CUDA_VISIBLE_DEVICES=6 OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python tools/extract_features.py --feat V --dataset RG --save_path {/save_path/extracted_features/rgb/VST_clip32} --stride 1 --rgb_model VST --batch_size 16
# extract optical flow features
CUDA_VISIBLE_DEVICES=6 OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python tools/extract_features.py --feat F --dataset RG --save_path  {/save_path/extracted_features/flow/I3D_clip32} --stride 1 --flow_model I3D --batch_size 16
# extract audio features
CUDA_VISIBLE_DEVICES=7 OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python tools/extract_features.py --feat A --dataset RG --save_path  {/save_path/extracted_features/audio/AST16_tdim41} --stride 1 --rgb_model VST --flow_model I3D --audio_model AST --batch_size 16 --tm_dim 47 --audio_sr 16000

# FISV dataset
# extract RGB features
CUDA_VISIBLE_DEVICES=6 OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python tools/extract_features.py --feat V --dataset FISV --save_path  {/save_path/extracted_features/rgb/VST_clip32} --stride 1 --rgb_model VST --batch_size 16
# extract optical flow features
CUDA_VISIBLE_DEVICES=6 OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python tools/extract_features.py --feat F --dataset FISV --save_path  {/save_path/extracted_features/flow/I3D_clip32} --stride 1 --flow_model I3D --batch_size 16
# extract audio features
CUDA_VISIBLE_DEVICES=6 OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python tools/extract_features.py --feat A --dataset FISV --save_path  {/save_path/extracted_features/audio/AST16_tdim41} --stride 1 --rgb_model VST --flow_model I3D --audio_model AST --batch_size 16 --tm_dim 41 --audio_sr 16000