Can i train a "videocutler"?

[so45jj45]

First of all, thank you for your great words.

I am trying to train a "videocutler." After preparing all the necessary datasets and running the code, I encounter the following error:

Exception has occurred: KeyError 'video_id' File "/home/sujung/repo/CutLER/videocutler/mask2former_video/data_video/datasets/ytvis_api/ytvos.py", line 75, in createIndex vidToAnns[ann['video_id']].append(ann) File "/home/sujung/repo/CutLER/videocutler/mask2former_video/data_video/datasets/ytvis_api/ytvos.py", line 66, in init self.createIndex() File "/home/sujung/repo/CutLER/videocutler/mask2former_video/data_video/datasets/ytvis.py", line 173, in load_ytvis_json ytvis_api = YTVOS(json_file) File "/home/sujung/repo/CutLER/videocutler/mask2former_video/data_video/datasets/ytvis.py", line 310, in DatasetCatalog.register(name, lambda: load_ytvis_json(json_file, image_root, name)) File "/home/sujung/repo/CutLER/detectron2/detectron2/data/catalog.py", line 58, in get return f() File "/home/sujung/repo/CutLER/videocutler/mask2former_video/data_video/build.py", line 92, in dataset_dicts = [DatasetCatalog.get(dataset_name) for dataset_name in dataset_names] File "/home/sujung/repo/CutLER/videocutler/mask2former_video/data_video/build.py", line 92, in get_detection_dataset_dicts dataset_dicts = [DatasetCatalog.get(dataset_name) for dataset_name in dataset_names] File "/home/sujung/repo/CutLER/videocutler/train_net_video.py", line 84, in build_train_loader dataset_dict = get_detection_dataset_dicts( File "/home/sujung/repo/CutLER/videocutler/mask2former_video/engine/defaults.py", line 391, in init data_loader = self.build_train_loader(cfg) File "/home/sujung/repo/CutLER/videocutler/train_net_video.py", line 305, in main trainer = Trainer(cfg) File "/home/sujung/repo/CutLER/detectron2/detectron2/engine/launch.py", line 84, in launch main_func(*args) File "/home/sujung/repo/CutLER/videocutler/train_net_video.py", line 313, in launch( KeyError: 'video_id'

Upon examining the problematic part, I noticed that the 'ann' dictionary does not contain the 'video_id' key. So, I changed 'video_id' to 'image_id' on lines 75 and 88 of 'ytvos.py.' However, when I tried running it again, I encountered the following error:

Exception has occurred: AssertionError Dataset 'imagenet_video_train_cls_agnostic' is empty! File "/home/sujung/repo/CutLER/videocutler/mask2former_video/data_video/build.py", line 94, in get_detection_dataset_dicts assert len(dicts), "Dataset '{}' is empty!".format(dataset_name) File "/home/sujung/repo/CutLER/videocutler/train_net_video.py", line 84, in build_train_loader dataset_dict = get_detection_dataset_dicts( File "/home/sujung/repo/CutLER/videocutler/mask2former_video/engine/defaults.py", line 391, in init data_loader = self.build_train_loader(cfg) File "/home/sujung/repo/CutLER/videocutler/train_net_video.py", line 305, in main trainer = Trainer(cfg) File "/home/sujung/repo/CutLER/detectron2/detectron2/engine/launch.py", line 84, in launch main_func(*args) File "/home/sujung/repo/CutLER/videocutler/train_net_video.py", line 313, in launch( AssertionError: Dataset 'imagenet_video_train_cls_agnostic' is empty!

WARNING [11/06 11:08:38 mask2former_video.data_video.datasets.ytvis]: /home/sujung/repo/CutLER/videocutler/DETECTRON2_DATASETS/imagenet/annotations/video_imagenet_train_fixsize480_tau0.15_N3.json contains 1933347 annotations, but only 0 of them match to images in the file. [11/06 11:08:38 mask2former_video.data_video.datasets.ytvis]: Loaded 0 videos in YTVIS format from /home/sujung/repo/CutLER/videocutler/DETECTRON2_DATASETS/imagenet/annotations/video_imagenet_train_fixsize480_tau0.15_N3.json

I would greatly appreciate any advice on how to resolve this issue.

[so45jj45]

Additionaly, there is problem in train_net_video.py

if args.test_dataset != "": cfg.DATASETS.TEST = ((args.test_dataset),) if args.train_dataset != "": cfg.DATASETS.TRAIN = ((args.train_dataset),) if args.steps != 0: cfg.SOLVER.STEPS = (int(args.steps),)

args don't have the key like test_dataset, train_dataset, steps.

so i modified that code like below:

cfg.DATASETS.TEST = (("ytvis_2019_train"),) cfg.DATASETS.TRAIN = (("imagenet_video_train_cls_agnostic"),) cfg.SOLVER.STEPS = (int(79999),)

[frank-xwang]

Hey! Did you follow the [INSTALL.md]() to prepare the necessary VideoCutLER packages and follow [datasets]() to prepare the necessary datasets? If you follow these steps, you should be able to run the training codes. Since we have defined the imagenet_video_train_cls_agnostic at this line, you may don't need to modify the codes. Also, are you using this annotation file?

[so45jj45]

Hey! Did you follow the INSTALL.md to prepare the necessary VideoCutLER packages and follow datasets to prepare the necessary datasets? If you follow these steps, you should be able to run the training codes. Since we have defined the imagenet_video_train_cls_agnostic at this line, you may don't need to modify the codes. Also, are you using this annotation file?

Thank you fore quick reply! In videocutler code page, you mentioned "You can either use the pre-generated json file directly by downloading it from [here]". So i clicked [here] but page is redirecting itself and can't download it. So i renamed imagenet_train_fixsize480_tau0.15_N3.json to video_imagenet_train_fixsize480_tau0.15_N3.json.

I think this may occur the errors I'll try it right away! Thanks again!

[so45jj45]

I solved problem that i mentioned just before.

but i'm not sure the training is work fine.

Command Line Args: Namespace(config_file='/home/sujung/repo/CutLER/videocutler/configs/imagenet_video/video_mask2former_R50_cls_agnostic.yaml', dist_url='tcp://127.0.0.1:50154', eval_only=False, machine_rank=0, num_gpus=1, num_machines=1, opts=['SOLVER.BASE_LR', '0.000005', 'SOLVER.IMS_PER_BATCH', '16', 'MODEL.MASK_FORMER.DROPOUT', '0.3', 'OUTPUT_DIR', '/home/sujung/repo/CutLER/videocutler/'], resume=False)

train_net_video.py line 264~266 if args.test_dataset != "": cfg.DATASETS.TEST = ((args.test_dataset),) if args.train_dataset != "": cfg.DATASETS.TRAIN = ((args.train_dataset),) if args.steps != 0: cfg.SOLVER.STEPS = (int(args.steps),)

There is no test_dataset, train_dataset, steps in args so i comment out them.

Next is detectron2/data/build.py (line 333)

prefetch_factor=prefetch_factor was occur the error so i comment out that and solved.

Next is detectron2/data/detection_utils.py

ERROR [11/07 08:51:54 d2.engine.train_loop]: Exception during training: Traceback (most recent call last): File "/home/sujung/repo/CutLER/detectron2/detectron2/engine/train_loop.py", line 155, in train self.run_step() File "/home/sujung/repo/CutLER/videocutler/mask2former_video/engine/defaults.py", line 508, in run_step self._trainer.run_step() File "/home/sujung/repo/CutLER/videocutler/mask2former_video/engine/train_loop.py", line 370, in run_step data = next(self._data_loader_iter) File "/home/sujung/repo/CutLER/detectron2/detectron2/data/common.py", line 329, in iter for d in self.dataset: File "/home/sujung/anaconda3/envs/videocutler/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 652, in next data = self._next_data() File "/home/sujung/anaconda3/envs/videocutler/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 692, in _next_data data = self._dataset_fetcher.fetch(index) # may raise StopIteration File "/home/sujung/anaconda3/envs/videocutler/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 32, in fetch data.append(next(self.dataset_iter)) File "/home/sujung/repo/CutLER/detectron2/detectron2/data/common.py", line 296, in iter yield self.dataset[idx] File "/home/sujung/repo/CutLER/detectron2/detectron2/data/common.py", line 125, in getitem data = self._map_func(self._dataset[cur_idx]) File "/home/sujung/repo/CutLER/detectron2/detectron2/utils/serialize.py", line 26, in call return self._obj(*args, **kwargs) File "/home/sujung/repo/CutLER/videocutler/mask2former_video/data_video/dataset_mapper.py", line 224, in call utils.check_image_size(dataset_dict, image) File "/home/sujung/repo/CutLER/detectron2/detectron2/data/detection_utils.py", line 196, in check_image_size raise SizeMismatchError( detectron2.data.detection_utils.SizeMismatchError: Mismatched image shape, got (600, 450), expect (450, 600). Please check the width/height in your annotation.

line 193 i modified the code "image_wh = (image.shape[1], image.shape[0]) to image_wh = (image.shape[0], image.shape[1]) and solved (may not be correct)

Did i missed something else or it's correct setting you mentioned in below?

NOTE: also need to change detectron2/detectron2/enginee/defaults.py (not modified)

NOTE: need to change detectron2/detectron2/data/detection_utils.py (modified)

NOTE: need to change detectron2/detectron2/data/transforms/transform.py (not modified)

[frank-xwang]

I think your changes should be okay as long as the size mismatch error disappeared.