Demo programme on camera input

[anhminh3105]

Hi,

I find this project very interesting and thanks for open-sourcing it.

I am trying to make a demo programme to load and run the models (e.g. SlowFast) and infer it using input from a USB camera to visually evaluate the accuracy and performance and I wonder if it would it be possible? if so could you briefly elaborate on which modules should be used and how it could be implemented?

Thanks in advance.

[Indigo6]

@anhminh3105 Hi! Thank you for your cool demo first! I successfully run the ava demo on my videos, but I encountered a confusing situation. The outputted labels and confidences(I modified the demo_net.py to show confidences of each label) seemed to be not so right (；´д｀)ゞ . I wonder what the cause may be ? Here is my cfg and some uploaded outputted frame samples:

TRAIN:
  ENABLE: False
  DATASET: ava
  BATCH_SIZE: 16
  EVAL_PERIOD: 1
  CHECKPOINT_PERIOD: 1
  AUTO_RESUME: True
  CHECKPOINT_FILE_PATH: "/home/data3/fanglin/demoSlowFast/pkls/AVA/SLOWFAST_32x2_R101_50_50.pkl" #path to pretrain model
  CHECKPOINT_TYPE: pytorch
DATA:
  NUM_FRAMES: 32
  SAMPLING_RATE: 2
  TRAIN_JITTER_SCALES: [256, 320]
  TRAIN_CROP_SIZE: 224
  TEST_CROP_SIZE: 256
  INPUT_CHANNEL_NUM: [3, 3]
DETECTION:
  ENABLE: True
  ALIGNED: False
AVA:
  BGR: False
  DETECTION_SCORE_THRESH: 0.8
  TEST_PREDICT_BOX_LISTS: ["person_box_67091280_iou90/ava_detection_val_boxes_and_labels.csv"]
SLOWFAST:
  ALPHA: 4
  BETA_INV: 8
  FUSION_CONV_CHANNEL_RATIO: 2
  FUSION_KERNEL_SZ: 5
RESNET:
  ZERO_INIT_FINAL_BN: True
  WIDTH_PER_GROUP: 64
  NUM_GROUPS: 1
  DEPTH: 101
  TRANS_FUNC: bottleneck_transform
  STRIDE_1X1: False
  NUM_BLOCK_TEMP_KERNEL: [[3, 3], [4, 4], [6, 6], [3, 3]]
  SPATIAL_DILATIONS: [[1, 1], [1, 1], [1, 1], [2, 2]]
  SPATIAL_STRIDES: [[1, 1], [2, 2], [2, 2], [1, 1]]
NONLOCAL:
  LOCATION: [[[], []], [[], []], [[6, 13, 20], []], [[], []]]
  GROUP: [[1, 1], [1, 1], [1, 1], [1, 1]]
  INSTANTIATION: dot_product
  POOL: [[[2, 2, 2], [2, 2, 2]], [[2, 2, 2], [2, 2, 2]], [[2, 2, 2], [2, 2, 2]], [[2, 2, 2], [2, 2, 2]]]
BN:
  USE_PRECISE_STATS: False
  NUM_BATCHES_PRECISE: 200
  MOMENTUM: 0.1
  WEIGHT_DECAY: 0.0
SOLVER:
  MOMENTUM: 0.9
  WEIGHT_DECAY: 1e-7
  OPTIMIZING_METHOD: sgd
MODEL:
  NUM_CLASSES: 80
  ARCH: slowfast
  LOSS_FUNC: bce
  DROPOUT_RATE: 0.5
TEST:
  ENABLE: False
  DATASET: ava
  BATCH_SIZE: 8
DATA_LOADER:
  NUM_WORKERS: 2
  PIN_MEMORY: True
DEMO:
  ENABLE: True
  LABEL_FILE_PATH: "./demo/AVA/ava.names"
  DATA_SOURCE: "/home/data3/fanglin/demoSlowFast/demo/videos/far.mp4"
  # DISPLAY_WIDTH: 640
  # DISPLAY_HEIGHT: 480
  DETECTRON2_OBJECT_DETECTION_MODEL_CFG: "COCO-Detection/faster_rcnn_R_50_FPN_3x.yaml"
  DETECTRON2_OBJECT_DETECTION_MODEL_WEIGHTS: "detectron2://COCO-Detection/faster_rcnn_R_50_FPN_3x/137849458/model_final_280758.pkl"
NUM_GPUS: 1
NUM_SHARDS: 1
RNG_SEED: 0
OUTPUT_DIR: "/home/data3/fanglin/demoSlowFast/demo/imgs/"

outputted frame sample: hand wave

Looking forward to your reply, and thank you for it in advance!

[yosagaf]

Hi, I could run demo for kinectics (demo_prediction branch) with success but whenever I run demo for AVA (demo_app branch) I got the following error. Thanks you for u help

Traceback (most recent call last): File "tools/run_net.py", line 7, in from slowfast.utils.misc import launch_job ImportError: cannot import name 'launch_job' from 'slowfast.utils.misc' (/home/fsa/SlowFast/slowfast/utils/misc.py)

[voldemortX]

Thanks for your nice work . I have question ,

python3 tools/run_net.py --predict_source ./sample_input.mp4

I try run this code and I get below error

ASSERT: "false" in file qasciikey.cpp, line 501
Traceback (most recent call last):
  File "tools/run_net.py", line 178, in <module>
    main()
  File "tools/run_net.py", line 172, in main
    daemon=False,
  File "/home/a/Downloads/test/venv/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 171, in spawn
    while not spawn_context.join():
  File "/home/a/Downloads/test/venv/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 107, in join
    (error_index, name)
Exception: process 0 terminated with signal SIGABRT

This is my config File - SLOWFAST_4x16_R50.yaml

TRAIN:
  ENABLE: False
  DATASET: kinetics
  BATCH_SIZE: 64
  EVAL_PERIOD: 10
  CHECKPOINT_PERIOD: 1
  CHECKPOINT_TYPE: caffe2
  CHECKPOINT_FILE_PATH: "/home/a/Downloads/WEB/test/SlowFast/checkpoints/SLOWFAST_4x16_R50.pkl"
  AUTO_RESUME: True
DATA:
  NUM_FRAMES: 32
  SAMPLING_RATE: 2
  TRAIN_JITTER_SCALES: [256, 320]
  TRAIN_CROP_SIZE: 224
  TEST_CROP_SIZE: 256
  INPUT_CHANNEL_NUM: [3, 3]
SLOWFAST:
  ALPHA: 8
  BETA_INV: 8
  FUSION_CONV_CHANNEL_RATIO: 2
  FUSION_KERNEL_SZ: 5
RESNET:
  ZERO_INIT_FINAL_BN: True
  WIDTH_PER_GROUP: 64
  NUM_GROUPS: 1
  DEPTH: 50
  TRANS_FUNC: bottleneck_transform
  STRIDE_1X1: False
  NUM_BLOCK_TEMP_KERNEL: [[3, 3], [4, 4], [6, 6], [3, 3]]
NONLOCAL:
  LOCATION: [[[], []], [[], []], [[], []], [[], []]]
  GROUP: [[1, 1], [1, 1], [1, 1], [1, 1]]
  INSTANTIATION: dot_product
BN:
  USE_PRECISE_STATS: True
  NUM_BATCHES_PRECISE: 200
  MOMENTUM: 0.1
  WEIGHT_DECAY: 0.0
SOLVER:
  BASE_LR: 0.1
  LR_POLICY: cosine
  MAX_EPOCH: 196
  MOMENTUM: 0.9
  WEIGHT_DECAY: 1e-4
  WARMUP_EPOCHS: 34
  WARMUP_START_LR: 0.01
  OPTIMIZING_METHOD: sgd
MODEL:
  NUM_CLASSES: 400
  ARCH: slowfast
  LOSS_FUNC: cross_entropy
  DROPOUT_RATE: 0.5
TEST:
  ENABLE: False
  DATASET: kinetics
  CHECKPOINT_TYPE: caffe2
  CHECKPOINT_FILE_PATH: "/home/a/Downloads/WEB/test/SlowFast/checkpoints/SLOWFAST_4x16_R50.pkl"
  BATCH_SIZE: 64
DATA_LOADER:
  NUM_WORKERS: 8
  PIN_MEMORY: True
NUM_GPUS: 2
NUM_SHARDS: 1
RNG_SEED: 0
OUTPUT_DIR: .

How can I solve this problem,?thank you!

additionaly I use torch 1.4.0 torchvision 0.5.0 this version

Man, I got the same error. Then I changed the resolution to 320x240 and it worked. GPU: 1 x RTX 2080 Ti (11G)

[lizituo12]

ava_demo.gif but I hope it satis

Thanks for your job,but it seems can not detect person ,do u have a version that include detectron2？

[wwdok]

outputted frame sample: hand wave

@indigo6 Why my outputed video looks like this, do you configure something to filter the top7 action classes, and how to show the FPS on the top-left corner ? and i feel my FPS is slower than you...

[Mathuran-Mang]

@anhminh3105 I am trying to run PySlowFast on real-time webcam video, I am of the understanding that you have been working on it. Do you have the source code or demo of how to run PySlowFast on a webcam real-time? I have been going through this entire page but I am a little confused and overwhelmed by how much has been done, so I was wondering if you could help me with trying to run Slowfast on live video real-time

[sainivedh]

@Mathuran-Mang Hi, any updates on how to make it work in real time using webcam.

Thanks