open-mmlab / mmaction2

OpenMMLab's Next Generation Video Understanding Toolbox and Benchmark
https://mmaction2.readthedocs.io
Apache License 2.0
4.32k stars 1.25k forks source link

'UniformSampleFrames is not in the transform registry. #2467

Open damienzh opened 1 year ago

damienzh commented 1 year ago

Branch

main branch (1.x version, such as v1.0.0, or dev-1.x branch)

Prerequisite

Environment

installed mmaction2 following the installation instruction using mim inside a new virtual environment

mmaction2                1.0.0
mmcv                     2.0.0
mmdet                    3.0.0
mmengine                 0.7.2
mmpose                   1.0.0

Describe the bug

Loads checkpoint by local backend from path: /home/futong/Projects/mmlab/mmaction2/work_dirs/posec3d_iclr/xuji_keypoint_v1_01/best_acc_top1_epoch_45.pth
Loads checkpoint by http backend from path: http://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_2x_coco/faster_rcnn_r50_fpn_2x_coco_bbox_mAP-0.384_20200504_210434-a5d8aa15.pth 
Performing Human Detection for each frame
received 200 frames
camera feed stopped
Loads checkpoint by http backend from path: https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w32_coco_256x192-c78dce93_20200708.pth 
Performing Human Pose Estimation for each frame
Traceback (most recent call last):
  File "xuji_skeleton_realtimev2.py", line 335, in <module>
    main()
  File "xuji_skeleton_realtimev2.py", line 309, in main
    results = inference_recognizer(model, fake_anno)
  File "/home/futong/.virtualenvs/mmlabv2/lib/python3.8/site-packages/mmaction/apis/inference.py", line 76, in inference_recognizer
    test_pipeline = Compose(test_pipeline_cfg)
  File "/home/futong/.virtualenvs/mmlabv2/lib/python3.8/site-packages/mmengine/dataset/base_dataset.py", line 37, in __init__
    transform = TRANSFORMS.build(transform)
  File "/home/futong/.virtualenvs/mmlabv2/lib/python3.8/site-packages/mmengine/registry/registry.py", line 545, in build
    return self.build_func(cfg, *args, **kwargs, registry=self)
  File "/home/futong/.virtualenvs/mmlabv2/lib/python3.8/site-packages/mmengine/registry/build_functions.py", line 100, in build_from_cfg
    raise KeyError(
KeyError: 'UniformSampleFrames is not in the transform registry. Please check whether the value of `UniformSampleFrames` is correct or it was registered as expected. More details can be found at https://mmengine.readthedocs.io/en/latest/advanced_tutorials/config.html#import-the-custom-module'

got this error when adopting demo_skeleton to a webcam inference, the test_pipeline as follow

test_pipeline = [
    dict(
        type='UniformSampleFrames', clip_len=48, num_clips=1, test_mode=True),
    dict(type='PoseDecode'),
    dict(type='PoseCompact', hw_ratio=1., allow_imgpad=True),
    dict(type='Resize', scale=(-1, 56)),
    dict(type='CenterCrop', crop_size=56),
    dict(
        type='GeneratePoseTarget',
        sigma=0.6,
        use_score=True,
        with_kp=True,
        with_limb=False,
        double=True,
        left_kp=left_kp,
        right_kp=right_kp),
    dict(type='FormatShape', input_format='NCTHW_Heatmap'),
    dict(type='PackActionInputs')
]

I tried the demo_skeleton.py in readme, there's no error.

Reproduces the problem - code sample

imports

import argparse
import os
import os.path as osp
import shutil
from collections import deque
from threading import Thread
from time import sleep

import cv2
import mmcv
import mmengine
import numpy as np
import torch
from mmengine import DictAction

import mmaction
from mmaction.apis import inference_recognizer, init_recognizer

try:
    from mmdet.apis import inference_detector, init_detector
except (ImportError, ModuleNotFoundError):
    raise ImportError('Failed to import `inference_detector` and '
                      '`init_detector` form `mmdet.apis`. These apis are '
                      'required in this demo! ')

try:
    from mmpose.apis import inference_topdown, init_model
    from mmpose.structures import PoseDataSample, merge_data_samples
except (ImportError, ModuleNotFoundError):
    raise ImportError('Failed to import `inference_top_down_pose_model`, '
                      '`init_pose_model`, and `vis_pose_result` form '
                      '`mmpose.apis`. These apis are required in this demo! ')

try:
    import moviepy.editor as mpy
except ImportError:
    raise ImportError('Please install moviepy to enable output file')

inference part


model = init_recognizer(config, args.checkpoint, args.device)
det_results = detection_inference(args, frame_buffer_det)
    torch.cuda.empty_cache()

    pose_results = pose_inference(args, frame_buffer_det, det_results)
    torch.cuda.empty_cache()
fake_anno = dict(
        frame_dir='',
        label=-1,
        img_shape=(h, w),
        original_shape=(h, w),
        start_index=0,
        modality='Pose',
        total_frames=clip_len)
    num_person = max([len(x['keypoints']) for x in pose_results])

    num_keypoint = 17
    keypoint = np.zeros((clip_len, num_person, num_keypoint, 2),
                        dtype=np.float16)
    keypoint_score = np.zeros((clip_len, num_person, num_keypoint),
                              dtype=np.float16)
    for i, poses in enumerate(pose_results):
        keypoint[i] = poses['keypoints']
        keypoint_score[i] = poses['keypoint_scores']

    fake_anno['keypoint'] = keypoint.transpose((1, 0, 2, 3))
    fake_anno['keypoint_score'] = keypoint_score.transpose((1, 0, 2))

    results = inference_recognizer(model, fake_anno)

### Reproduces the problem - command or script

_No response_

### Reproduces the problem - error message

_No response_

### Additional information

_No response_
cir7 commented 1 year ago

The bug is due to the shift of default scope when calling detection_inference and pose_inference, and the incorrect scope would fail to build a module in mmaction2 registry, e.g. UniformSampleFrames in your case.
A walkaround solution is to put init_recognizer and inference_recognizer together:

model = init_recognizer(config, args.checkpoint, args.device)
results = inference_recognizer(model, fake_anno)

or call init_scope before inference recognizer:

from mmengine.registry import init_default_scope
init_default_scope('mmaction')

we will fix it asap.

What's more, the webcam demo would call detection_inference and pose_inference many times, due to we build models in the inference API, which would rebuild the model repeatedly, which would slow down the inference speed.
We will improve the code asap.