I want to use the realsense camera in webcam_demo.py, but it can only show the image one by one, not the video.

Prerequisite

[X] I have searched Issues and Discussions but cannot get the expected help.
[X] I have read the FAQ documentation but cannot get the expected help.
[X] The bug has not been fixed in the latest version (master) or latest version (3.x).

💬 Describe the reimplementation questions

I had used the webcam_demo.py, and the config is yolov3_mobilenetv2_320_300e_coco.py, the checkpoint is yolov3_mobilenetv2_320_300e_coco_20210719_215349-d18dff72.pth

Environment

_libgcc_mutex             0.1                        main  
_openmp_mutex             5.1                       1_gnu  
addict                    2.4.0                    pypi_0    pypi
blas                      1.0                         mkl  
brotlipy                  0.7.0           py38h27cfd23_1003  
bzip2                     1.0.8                h7b6447c_0  
ca-certificates           2022.10.11           h06a4308_0  
certifi                   2022.9.24        py38h06a4308_0  
cffi                      1.15.1           py38h5eee18b_2  
charset-normalizer        2.0.4              pyhd3eb1b0_0  
click                     8.1.3                    pypi_0    pypi
colorama                  0.4.6                    pypi_0    pypi
commonmark                0.9.1                    pypi_0    pypi
contourpy                 1.0.6                    pypi_0    pypi
cryptography              38.0.1           py38h9ce1e76_0  
cudatoolkit               10.2.89              hfd86e86_1  
cycler                    0.11.0                   pypi_0    pypi
ffmpeg                    4.3                  hf484d3e_0    pytorch
fonttools                 4.38.0                   pypi_0    pypi
freetype                  2.12.1               h4a9f257_0  
giflib                    5.2.1                h7b6447c_0  
gmp                       6.2.1                h295c915_3  
gnutls                    3.6.15               he1e5248_0  
idna                      3.4              py38h06a4308_0  
importlib-metadata        5.1.0                    pypi_0    pypi
intel-openmp              2021.4.0          h06a4308_3561  
jpeg                      9e                   h7f8727e_0  
kiwisolver                1.4.4                    pypi_0    pypi
lame                      3.100                h7b6447c_0  
lcms2                     2.12                 h3be6417_0  
ld_impl_linux-64          2.38                 h1181459_1  
lerc                      3.0                  h295c915_0  
libdeflate                1.8                  h7f8727e_5  
libffi                    3.4.2                h6a678d5_6  
libgcc-ng                 11.2.0               h1234567_1  
libgomp                   11.2.0               h1234567_1  
libiconv                  1.16                 h7f8727e_2  
libidn2                   2.3.2                h7f8727e_0  
libpng                    1.6.37               hbc83047_0  
libstdcxx-ng              11.2.0               h1234567_1  
libtasn1                  4.16.0               h27cfd23_0  
libtiff                   4.4.0                hecacb30_2  
libunistring              0.9.10               h27cfd23_0  
libwebp                   1.2.4                h11a3e52_0  
libwebp-base              1.2.4                h5eee18b_0  
lz4-c                     1.9.3                h295c915_1  
markdown                  3.4.1                    pypi_0    pypi
matplotlib                3.6.2                    pypi_0    pypi
mkl                       2021.4.0           h06a4308_640  
mkl-service               2.4.0            py38h7f8727e_0  
mkl_fft                   1.3.1            py38hd3c417c_0  
mkl_random                1.2.2            py38h51133e4_0  
mmcv-full                 1.7.0                    pypi_0    pypi
mmdet                     2.26.0                    dev_0    <develop>
model-index               0.1.11                   pypi_0    pypi
ncurses                   6.3                  h5eee18b_3  
nettle                    3.7.3                hbbd107a_1  
numpy                     1.23.4           py38h14f4228_0  
numpy-base                1.23.4           py38h31eccc5_0  
opencv-python             4.6.0.66                 pypi_0    pypi
openh264                  2.1.1                h4ff587b_0  
openmim                   0.3.3                    pypi_0    pypi
openssl                   1.1.1s               h7f8727e_0  
ordered-set               4.1.0                    pypi_0    pypi
packaging                 21.3                     pypi_0    pypi
pandas                    1.5.2                    pypi_0    pypi
pillow                    9.2.0            py38hace64e9_1  
pip                       22.2.2           py38h06a4308_0  
pycocotools               2.0.6                    pypi_0    pypi
pycparser                 2.21               pyhd3eb1b0_0  
pygments                  2.13.0                   pypi_0    pypi
pyopenssl                 22.0.0             pyhd3eb1b0_0  
pyparsing                 3.0.9                    pypi_0    pypi
pyrealsense2              2.51.1.4348              pypi_0    pypi
pysocks                   1.7.1            py38h06a4308_0  
python                    3.8.15               h7a1cb2a_2  
python-dateutil           2.8.2                    pypi_0    pypi
pytorch                   1.12.1          py3.8_cuda10.2_cudnn7.6.5_0    pytorch
pytorch-mutex             1.0                        cuda    pytorch
pytz                      2022.6                   pypi_0    pypi
pyyaml                    6.0                      pypi_0    pypi
readline                  8.2                  h5eee18b_0  
requests                  2.28.1           py38h06a4308_0  
rich                      12.6.0                   pypi_0    pypi
scipy                     1.9.3                    pypi_0    pypi
setuptools                65.5.0           py38h06a4308_0  
six                       1.16.0             pyhd3eb1b0_1  
sqlite                    3.40.0               h5082296_0  
tabulate                  0.9.0                    pypi_0    pypi
terminaltables            3.1.10                   pypi_0    pypi
tk                        8.6.12               h1ccaba5_0  
torchaudio                0.12.1               py38_cu102    pytorch
torchvision               0.13.1               py38_cu102    pytorch
typing_extensions         4.3.0            py38h06a4308_0  
urllib3                   1.26.12          py38h06a4308_0  
wheel                     0.37.1             pyhd3eb1b0_0  
xz                        5.2.6                h5eee18b_0  
yapf                      0.32.0                   pypi_0    pypi
zipp                      3.11.0                   pypi_0    pypi
zlib                      1.2.13               h5eee18b_0  
zstd                      1.5.2                ha4553b6_0

Expected results

I think it can show the video, not the image one by one.

Additional information

I want to used the realsense camera, so i had make some modify in the webcam_demo.py:

 # Copyright (c) OpenMMLab. All rights reserved.
import argparse
import pyrealsense2 as rs
import numpy as np
import cv2
import torch

from mmdet.apis import inference_detector, init_detector

pipeline = rs.pipeline()

config = rs.config()
config.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 30)
config.enable_stream(rs.stream.color, 640, 480, rs.format.bgr8, 30)

profile = pipeline.start(config)

device = profile.get_device()
depth_sensor = device.first_depth_sensor()
device.hardware_reset()

depth_sensor = profile.get_device().first_depth_sensor()
depth_scale = depth_sensor.get_depth_scale()
print("Depth Scale is: ", depth_scale)

clipping_distance_in_meters = 1
clipping_distance = clipping_distance_in_meters / depth_scale

align_to = rs.stream.color
align = rs.align(align_to)

def parse_args():
    parser = argparse.ArgumentParser(description='MMDetection webcam demo')
    parser.add_argument('config', help='test config file path')
    parser.add_argument('checkpoint', help='checkpoint file')
    parser.add_argument(
        '--device', type=str, default='cuda:0', help='CPU/CUDA device option')
    # parser.add_argument(
    #     '--camera-id', type=int, default=0, help='camera device id')
    parser.add_argument(
        '--score-thr', type=float, default=0.5, help='bbox score threshold')
    args = parser.parse_args()
    return args

def main():
    args = parse_args()

    device = torch.device(args.device)

    model = init_detector(args.config, args.checkpoint, device=device)

    # camera = cv2.VideoCapture(args.camera_id)

    print('Press "Esc", "q" or "Q" to exit.')
    while True:
        frames = pipeline.wait_for_frames()

        # aligned_frames = align.process(frames)

        # aligned_depth_frame = aligned_frames.get_depth_frame()
        # color_frame = aligned_frames.get_color_frame()
        depth_frame = frames.get_depth_frame()
        color_frame = frames.get_color_frame()

        # if not aligned_depth_frame or not color_frame:
        #     continue
        if not depth_frame or not color_frame:
            continue

        # depth_image = np.asanyarray(depth_frame.get_data())
        color_image = np.asanyarray(color_frame.get_data())

        result = inference_detector(model, color_image)
        #cv2.namedWindow('Figure 1', cv2.WINDOW_AUTOSIZE)
        # cv2.imshow('Figure 1', color_image)

        ch = cv2.waitKey(1)

        if ch == 27 or ch == ord('q') or ch == ord('Q'):
            break

        model.show_result(
            color_image, result, score_thr=args.score_thr, wait_time=0.5,show=True)

if __name__ == '__main__':
    main()

Thank you for your help!!!!!!

open-mmlab / mmdetection

I want to use the realsense camera in webcam_demo.py, but it can only show the image one by one, not the video. #9425

Prerequisite

💬 Describe the reimplementation questions

Environment

Expected results

Additional information