Multi-modal algorithm inference demo error

[andyluo7]

Thanks for your error report and we appreciate it a lot.

Checklist

1. I have searched related issues but cannot get the expected help.
2. I have read the FAQ documentation but cannot get the expected help.
3. The bug has not been fixed in the latest version.

Describe the bug Ran into the following error of the Multi-modal algorithm inference demo. mmdet was installed from source. The image and webcam demo works.

(openmmlab) user@puget-253012:~/openmmlab/mmdetection$ python demo/image_demo.py demo/demo.jpg glip_tiny_a_mmdet-b3654169.pth --texts bench The model is a weight file, automatically assign the model to --weights Loads checkpoint by local backend from path: glip_tiny_a_mmdet-b3654169.pth Traceback (most recent call last): File "/home/user/anaconda3/envs/openmmlab/lib/python3.8/site-packages/mmengine/infer/infer.py", line 462, in _init_model cfg_string = checkpoint['message_hub']['runtime_info']['cfg'] KeyError: 'message_hub'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "demo/image_demo.py", line 192, in main() File "demo/image_demo.py", line 179, in main inferencer = DetInferencer(**init_args) File "/home/user/openmmlab/mmdetection/mmdet/apis/det_inferencer.py", line 99, in init super().init( File "/home/user/anaconda3/envs/openmmlab/lib/python3.8/site-packages/mmengine/infer/infer.py", line 180, in init self.model = self._init_model(cfg, weights, device) # type: ignore File "/home/user/anaconda3/envs/openmmlab/lib/python3.8/site-packages/mmengine/infer/infer.py", line 464, in _init_model assert 'meta' in checkpoint, ( AssertionError: If model(config) is not provided, the checkpoint mustcontain the config string in meta or message_hub, but both meta and message_hub are not found in the checkpoint.

Reproduction

python demo/image_demo.py demo/demo.jpg glip_tiny_a_mmdet-b3654169.pth --texts bench

A placeholder for the command.

2. Did you make any modifications on the code or config? Did you understand what you have modified?
3. What dataset did you use?

Environment

1. Please run python mmdet/utils/collect_env.py to collect necessary environment information and paste it here. sys.platform: linux Python: 3.8.18 (default, Sep 11 2023, 13:40:15) [GCC 11.2.0] CUDA available: True MUSA available: False numpy_random_seed: 2147483648 GPU 0: NVIDIA GeForce RTX 4090 CUDA_HOME: /usr/local/cuda-12.2 NVCC: Cuda compilation tools, release 12.2, V12.2.140 GCC: gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 PyTorch: 2.2.0+cu121 PyTorch compiling details: PyTorch built with:
   • GCC 9.3
   • C++ Version: 201703
   • Intel(R) oneAPI Math Kernel Library Version 2022.2-Product Build 20220804 for Intel(R) 64 architecture applications
   • Intel(R) MKL-DNN v3.3.2 (Git Hash 2dc95a2ad0841e29db8b22fbccaf3e5da7992b01)
   • OpenMP 201511 (a.k.a. OpenMP 4.5)
   • LAPACK is enabled (usually provided by MKL)
   • NNPACK is enabled
   • CPU capability usage: AVX2
   • CUDA Runtime 12.1
   • NVCC architecture flags: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90
   • CuDNN 8.9.2
   • Magma 2.6.1
   • Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=8.9.2, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.2.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF,

TorchVision: 0.17.0+cu121 OpenCV: 4.7.0 MMEngine: 0.10.3 MMDetection: 3.3.0+44ebd17

3. You may add addition that may be helpful for locating the problem, such as
   • How you installed PyTorch [e.g., pip, conda, source]
   • Other environment variables that may be related (such as $PATH, $LD_LIBRARY_PATH, $PYTHONPATH, etc.)

Error traceback If applicable, paste the error trackback here.

A placeholder for trackback.

Bug fix If you have already identified the reason, you can provide the information here. If you are willing to create a PR to fix it, please also leave a comment here and that would be much appreciated!

[hhaAndroid]

@andyluo7 Since you are not using the weights trained by mmdetection, you should use the configuration file name for inference, for example

python demo/image_demo.py demo/demo.jpg \
        glip_atss_swin-t_a_fpn_dyhead_pretrain_obj365
        --texts 'bench . car .' -c

[AkashahS]

I encountered the same error and was able to resolve it by using the following instead:

python demo/image_demo.py demo/demo.jpg configs/glip/glip_atss_swin-t_a_fpn_dyhead_pretrain_obj365.py --weights glip_tiny_a_mmdet-b3654169.pth --texts 'bench. car'

hope it helps!

[Colorfu1]

The same issue to me. :(

[Cloud65000]

@andyluo7 Since you are not using the weights trained by mmdetection, you should use the configuration file name for inference, for example

python demo/image_demo.py demo/demo.jpg \
        glip_atss_swin-t_a_fpn_dyhead_pretrain_obj365
        --texts 'bench . car .' -c

Hi! I use the groundingdino_swinb_cogcoor_mmdet-55949c9c.pth and in-house inference code which can directly use the xml annotation files. But when I tested the model's performance using this pth file, this issue occurred but when I use the finetuned gdino_b's pth file to run the test process, it worked. I wonder the reason why this happened and how the solution will be.