Issues in AVSlowFast demo

[ybkim95]

Hello,

thanks to you guys I've succeeded in fine-tuning AVSlowFast on my own dataset and I now have the train, test results with the parameters from the training.

Now that I have the model params, I wanted to do the demos but it seems like you guys didn't provide demo script for the AVSlowFast model.

So when I follow your guidelines and run the demo, the script complains about not providing the audio sources.

Can you provide a demo script for AVSlowFast?

Best, Yubin

[ERROR]

Traceback (most recent call last): File "tools/run_net.py", line 42, in main() File "tools/run_net.py", line 37, in main demo(cfg) File "/workspace/Documents/Tony/AVSlowFast/tools/demo_net.py", line 119, in demo for frames in tqdm.tqdm(run_demo(cfg, frame_provider)): File "/usr/local/lib/python3.6/dist-packages/tqdm/std.py", line 1174, in iter for obj in iterable: File "/workspace/Documents/Tony/AVSlowFast/tools/demo_net.py", line 79, in run_demo model.put(task) File "/workspace/Documents/Tony/AVSlowFast/slowfast/visualization/predictor.py", line 138, in put task = self.predictor(task) File "/workspace/Documents/Tony/AVSlowFast/slowfast/visualization/predictor.py", line 77, in call inputs = process_cv2_inputs(frames, self.cfg) File "/workspace/Documents/Tony/AVSlowFast/slowfast/visualization/utils.py", line 321, in process_cv2_inputs inputs = [inp.unsqueeze(0) for inp in inputs] File "/workspace/Documents/Tony/AVSlowFast/slowfast/visualization/utils.py", line 321, in inputs = [inp.unsqueeze(0) for inp in inputs] AttributeError: 'NoneType' object has no attribute 'unsqueeze'

[ybkim95]

What I've tried is as below :

python3 tools/run_net.py --cfg demo/Kinetics/AVSLOWFAST_4x16_R50.yaml DEMO.LABEL_FILE_PATH /u/ybkim95/AVSlowFast/labels/demo.json NUM_GPUS 1 DEMO.INPUT_VIDEO /u/ybkim95/AVSlowFast/demo_videos/clip_106.mp4 DEMO.ENABLE True NUM_SHARDS 1

in the AVSlowFast directory.

[ybkim95]

Could you guys leave comments on where should I change to do this? or provide extra scripts?

[ybkim95]

Here is the AVSLOWFAST_4X16_R50.yaml file

TRAIN: ENABLE: False DATASET: kinetics BATCH_SIZE: 32 EVAL_PERIOD: 10 CHECKPOINT_PERIOD: 1 AUTO_RESUME: True DATA: USE_BGR_ORDER: False # False NUM_FRAMES: 32 SAMPLING_RATE: 2 TRAIN_JITTER_SCALES: [256, 320] TRAIN_CROP_SIZE: 224 TEST_CROP_SIZE: 256 INPUT_CHANNEL_NUM: [3, 3, 1] USE_AUDIO: True GET_MISALIGNED_AUDIO: True AUDIO_SAMPLE_RATE: 16000 AUDIO_WIN_SZ: 32 AUDIO_STEP_SZ: 16 AUDIO_FRAME_NUM: 128 AUDIO_MEL_NUM: 80 AUDIO_MISALIGNED_GAP: 32 # half second LOGMEL_MEAN: -7.03 # -7.03, -24.227 LOGMEL_STD: 4.66 # 4.66, 1.0 EASY_NEG_RATIO: 0.75 MIX_NEG_EPOCH: 96 SLOWFAST: ALPHA: 8 BETA_INV: 8 FUSION_CONV_CHANNEL_RATIO: 2 FUSION_KERNEL_SZ: 5 AU_ALPHA: 32 AU_BETA_INV: 2 AU_FUSION_CONV_CHANNEL_MODE: ByDim # ByDim, ByRatio AU_FUSION_CONV_CHANNEL_RATIO: 0.25 AU_FUSION_CONV_CHANNEL_DIM: 64 AU_FUSION_KERNEL_SZ: 5 AU_FUSION_CONV_NUM: 2 AU_REDUCE_TF_DIM: True FS_FUSION: [False, False, True, True] AFS_FUSION: [False, False, True, True] AVS_FLAG: [False, False, True, True, True] AVS_PROJ_DIM: 64 AVS_VAR_THRESH: 0.01 AVS_DUPLICATE_THRESH: 0.99999 DROPPATHWAY_RATE: 0.8 # 0.8 RESNET: ZERO_INIT_FINAL_BN: True WIDTH_PER_GROUP: 64 NUM_GROUPS: 1 DEPTH: 50 TRANS_FUNC: bottleneck_transform AUDIO_TRANS_FUNC: tf_bottleneck_transform_v1 AUDIO_TRANS_NUM: 2 STRIDE_1X1: False NUM_BLOCK_TEMP_KERNEL: [[3, 3, 3], [4, 4, 4], [6, 6, 6], [3, 3, 3]] SPATIAL_DILATIONS: [[1, 1, 1], [1, 1, 1], [1, 1, 1], [1, 1, 1]] NONLOCAL: LOCATION: [[[], [], []], [[], [], []], [[], [], []], [[], [], []]] GROUP: [[1, 1, 1], [1, 1, 1], [1, 1, 1], [1, 1, 1]] POOL: [ [[1, 2, 2], [1, 2, 2], [1, 2, 2]], [[1, 2, 2], [1, 2, 2], [1, 2, 2]], [[1, 2, 2], [1, 2, 2], [1, 2, 2]], [[1, 2, 2], [1, 2, 2], [1, 2, 2]], ] INSTANTIATION: dot_product BN: USE_PRECISE_STATS: True NUM_BATCHES_PRECISE: 200 WEIGHT_DECAY: 0.0 SOLVER: BASE_LR: 0.1 # 0.1 LR_POLICY: cosine MAX_EPOCH: 196 MOMENTUM: 0.9 WEIGHT_DECAY: 1e-4 WARMUP_EPOCHS: 34.0 # 34.0 WARMUP_START_LR: 0.01 # 0.01 OPTIMIZING_METHOD: sgd MODEL: NUM_CLASSES: 400 MODEL_NAME: AVSlowFast ARCH: avslowfast LOSS_FUNC: cross_entropy DROPOUT_RATE: 0.5 TEST: ENABLE: False DATASET: kinetics BATCH_SIZE: 64 DATA_LOADER: NUM_WORKERS: 4 # 8 PIN_MEMORY: True DEMO: ENABLE: True INPUT_VIDEO: '/u/ybkim95/AVSlowFast/demo_videos/clip_108.mp4' NUM_GPUS: 1 NUM_SHARDS: 1 RNG_SEED: 0 OUTPUT_DIR: ./output/AVSlowFast-R50-4x16