Perform inference on a video using custom weights

[LLH-Harward]

Hello, thank you for your outstanding work! I would like to perform video inference directly using yolo_world, and I have used Roboflow Inference and Supervision, but they only provide some benchmark models, such as l, x, v2-x, v2-l. The model performance of the Yolo World Hugging Face model (https://huggingface.co/spaces/stevengrove/YOLO-World) is better for my purposes than the standard inference one, "yolo_world/v2-x" for example. I would like to use the weights from Hugging Face, such as x-1280, for inference. Could you please provide the necessary support? Or is it possible to directly input videos? I would greatly appreciate it.

[wondervictor]

Could you provide more details/clues about why the HF version (L-640) of YOLO-World is better than the GitHub version (X-1280)? BTW, the HuggingFace demo only uses L-640.

[LLH-Harward]

Thank you for your response. I apologize if I wasn't clear earlier. I meant to point out that on Hugging Face, the models with a 1280 input seem to be more effective at detecting small objects. While Roboflow Inference and Supervision does support video processing, it currently only offers basic models like v2-l, v2-x, and lacks access to other 1280 models. Could you kindly inform me if there's a method to directly use custom weights (for instance, those obtained from training) for video inference?

[wondervictor]

Sure, I saw many requests for inferencing with videos and I'll increase the priority of it. And I'll notify you if I make it, not too long.

[LLH-Harward]

Thank you so much.

[wondervictor]

Hi @LLH-Harward, the latest update has supported video inference. Please check demo/video_demo.py.

[LLH-Harward]

Thank you so much！ I'll try it later.

[LLH-Harward]

hello,When I used video_demo.py for inference, the following error occurred, showing that there is no "data/coco/lvis/lvis_v1_minival_inserted_image_name.json" I found the relevant content in the model's configuration file "yolo_world_v2_x_vlpan_bn_2e-3_100e_4x8gpus_obj365v1_goldg_train_1280ft_lvis_minival.py". How to solve this problem? Can you give relevant guidance? Related configuration: torch+cu118==2.1.1 torchvision+cu118==0.16.1 mmcv==2.0.0rc4 mmdet==3.0.0 mmengine==0.10.3 mmyolo==0.6.0

BUG: inputs: python video_demo.py D:\YOLO-World-master\configs\pretrain\yolo_world_v2_x_vlpan_bn_2e-3_100e_4x8gpus_obj365v1_goldg_train_1280ft_lvis_minival.py D:\YOLO-World-master\pretrained_weights\ yolo_world_v2_x_obj365v1_goldg_cc3mlite_pretrain_1280ft-14996a36.pth D:\YOLO-World-master\result.mp4 "people,laptop,book,bottle,pen,phone" --out out111

bin C:\Users\714\AppData\Roaming\Python\Python39\site-packages\bitsandbytes\libbitsandbytes_cuda118.dll Loads checkpoint by local backend from path: D:\YOLO-World-master\pretrained_weights\yolo_world_v2_x_obj365v1_goldg_cc3mlite_pretrain_1280ft-14996a36.pth Traceback (most recent call last): File "D:\YOLO-World-master\demo\video_demo.py", line 109, in main() File "D:\YOLO-World-master\demo\video_demo.py", line 56, in main model = init_detector(args.config, args.checkpoint, device=args.device) File "F:\Anaconda\envs\yoloworld\lib\site-packages\mmdet\apis\inference.py", line 97, in init_detector metainfo = DATASETS.build(test_dataset_cfg).metainfo File "F:\Anaconda\envs\yoloworld\lib\site-packages\mmengine\registry\registry.py", line 570, in build return self.build_func(cfg, args, kwargs, registry=self) File "F:\Anaconda\envs\yoloworld\lib\site-packages\mmengine\registry\build_functions.py", line 121, in build_from_cfg obj = obj_cls(args) # type: ignore File "D:\YOLO-World-master\yolo_world\datasets\mm_dataset.py", line 25, in init self.dataset = DATASETS.build(dataset) File "F:\Anaconda\envs\yoloworld\lib\site-packages\mmengine\registry\registry.py", line 570, in build return self.build_func(cfg, args, kwargs, registry=self) File "F:\Anaconda\envs\yoloworld\lib\site-packages\mmengine\registry\build_functions.py", line 121, in build_from_cfg obj = obj_cls(args) # type: ignore File "F:\Anaconda\envs\yoloworld\lib\site-packages\mmyolo\datasets\yolov5_coco.py", line 19, in init super().init(*args, *kwargs) File "F:\Anaconda\envs\yoloworld\lib\site-packages\mmdet\datasets\base_det_dataset.py", line 40, in init super().init(args, **kwargs) File "F:\Anaconda\envs\yoloworld\lib\site-packages\mmengine\dataset\base_dataset.py", line 247, in init self.full_init() File "F:\Anaconda\envs\yoloworld\lib\site-packages\mmyolo\datasets\yolov5_coco.py", line 27, in full_init self.data_list = self.load_data_list() File "F:\Anaconda\envs\yoloworld\lib\site-packages\mmdet\datasets\lvis.py", line 605, in load_data_list self.lvis = LVIS(local_path) File "F:\Anaconda\envs\yoloworld\lib\site-packages\lvis\lvis.py", line 27, in init self.dataset = self._load_json(annotation_path) File "F:\Anaconda\envs\yoloworld\lib\site-packages\lvis\lvis.py", line 35, in _load_json with open(path, "r") as f: FileNotFoundError: [Errno 2] No such file or directory: 'data/coco/lvis/lvis_v1_minival_inserted_image_name.json'

[LLH-Harward]

New developments: After I completed the json file and its path required in the configuration file "yolo_world_v2_x_vlpan_bn_2e-3_100e_4x8gpus_obj365v1_goldg_train_1280ft_lvis_minival.py", and modified for frame in track_iter_progress(video_reader): in video_demo.py to for frame in video_reader:, the code can now run normally and produce results.

However, the running speed is quite slow. Is it because the mmcv framework loads slowly?

[wondervictor]

The visualization takes time to draw objects in frames.

[LLH-Harward]

OK, thank you for your help!