copy_and_paste code error

[so45jj45]

Thank you for your appreciation of our research.

When I read your paper, I understood that it uses the copy-and-paste technique on images to create synthetic videos. It appears that this code operates within the copy_and_paste(self, labeled_data, unlabeled_data) function of the train_loop.py.

The part keep = iou_matrix.max(1)[0] < 0.0 seems to always return false, leading me to believe that the current code does not copy instances from other images but rather copies the existing target frame for training.

When changing it to keep = iou_matrix.max(1)[0] < 0.5, it results in an error. I would appreciate any assistance in resolving this issue.

[02/16 08:27:16 mask2former_video.data_video.build]: Using training sampler TrainingSampler [02/16 08:27:16 d2.data.common]: Serializing the dataset using: <class 'detectron2.data.common._TorchSerializedList'> [02/16 08:27:16 d2.data.common]: Serializing 100 elements to byte tensors and concatenating them all ... [02/16 08:27:16 d2.data.common]: Serialized dataset takes 0.69 MiB [02/16 08:27:16 d2.data.build]: Making batched data loader with batch_size=2 INFO: use AMPTrainer. copy_paste hyper-params: True 1.0 False [02/16 08:27:36 d2.checkpoint.detection_checkpoint]: [DetectionCheckpointer] Loading from /workspace/CutLER/videocutler/pretrain/cutler_m2f_rn50.pth ... [02/16 08:27:36 fvcore.common.checkpoint]: [Checkpointer] Loading from /workspace/CutLER/videocutler/pretrain/cutler_m2f_rn50.pth ... [02/16 08:27:37 d2.engine.train_loop]: Starting training from iteration 0 ERROR [02/16 08:28:22 d2.engine.train_loop]: Exception during training: Traceback (most recent call last): File "/workspace/CutLER/detectron2/detectron2/engine/train_loop.py", line 155, in train self.run_step() File "/workspace/CutLER/CutLER/videocutler/mask2former_video/engine/defaults.py", line 508, in run_step self._trainer.run_step() File "/workspace/CutLER/CutLER/videocutler/mask2former_video/engine/train_loop.py", line 374, in run_step data = self.copy_and_paste(copy.deepcopy(data[::-1]), data) File "/workspace/CutLER/CutLER/videocutler/mask2former_video/engine/train_loop.py", line 212, in copy_and_paste copied_instances.gt_masks = copied_masks File "/workspace/CutLER/detectron2/detectron2/structures/instances.py", line 62, in setattr self.set(name, val) File "/workspace/CutLER/detectron2/detectron2/structures/instances.py", line 78, in set assert ( AssertionError: Adding a field of length 2 to a Instances of length 1 [02/16 08:28:22 d2.engine.hooks]: Total training time: 0:00:44 (0:00:00 on hooks) [02/16 08:28:22 d2.utils.events]: iter: 0 lr: N/A max_mem: 171M Traceback (most recent call last): File "/opt/conda/lib/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/opt/conda/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/root/.vscode-server/extensions/ms-python.python-2024.0.1/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/main.py", line 39, in cli.main() File "/root/.vscode-server/extensions/ms-python.python-2024.0.1/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430, in main run() File "/root/.vscode-server/extensions/ms-python.python-2024.0.1/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 284, in run_file runpy.run_path(target, run_name="main") File "/root/.vscode-server/extensions/ms-python.python-2024.0.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 321, in run_path return _run_module_code(code, init_globals, run_name, File "/root/.vscode-server/extensions/ms-python.python-2024.0.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 135, in _run_module_code _run_code(code, mod_globals, init_globals, File "/root/.vscode-server/extensions/ms-python.python-2024.0.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code exec(code, run_globals) File "/workspace/CutLER/CutLER/videocutler/train_net_video.py", line 314, in launch( File "/workspace/CutLER/detectron2/detectron2/engine/launch.py", line 84, in launch main_func(*args) File "/workspace/CutLER/CutLER/videocutler/train_net_video.py", line 305, in main return trainer.train() File "/workspace/CutLER/CutLER/videocutler/mask2former_video/engine/defaults.py", line 498, in train super().train(self.start_iter, self.max_iter) File "/workspace/CutLER/detectron2/detectron2/engine/train_loop.py", line 155, in train self.run_step() File "/workspace/CutLER/CutLER/videocutler/mask2former_video/engine/defaults.py", line 508, in run_step self._trainer.run_step() File "/workspace/CutLER/CutLER/videocutler/mask2former_video/engine/train_loop.py", line 374, in run_step data = self.copy_and_paste(copy.deepcopy(data[::-1]), data) File "/workspace/CutLER/CutLER/videocutler/mask2former_video/engine/train_loop.py", line 212, in copy_and_paste copied_instances.gt_masks = copied_masks File "/workspace/CutLER/detectron2/detectron2/structures/instances.py", line 62, in setattr self.set(name, val) File "/workspace/CutLER/detectron2/detectron2/structures/instances.py", line 78, in set assert ( AssertionError: Adding a field of length 2 to a Instances of length 1

[frank-xwang]

Hi, as I've left Meta and no longer have the permissions to make direct changes to this repository, I've implemented the bug fix in my local fork available at: https://github.com/frank-xwang/CutLER. Please let me know if you encounter any additional issues. Thank you!

update: got the push permission. Have updated the codes. :-)

[YangYafei1998]

Hi Frank, I have encountered another error when with video_mask2former_R50_cls_agnostic.yaml. The error comes from the copy_paste function, below is the error msg: Traceback (most recent call last): File "/home/anaconda3/envs/cutler/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 68, in _wrap fn(i, args) File "/media/HDD1/detectron2/detectron2/engine/launch.py", line 123, in _distributed_worker main_func(args) File "/media/HDD1/Video_CutLER/videocutler/train_net_video.py", line 312, in main return trainer.train() File "/media/HDD1/Video_CutLER/videocutler/mask2former_video/engine/defaults.py", line 498, in train super().train(self.start_iter, self.max_iter) File "/media/HDD1/detectron2/detectron2/engine/train_loop.py", line 155, in train self.run_step() File "/media/HDD1/Video_CutLER/videocutler/mask2former_video/engine/defaults.py", line 508, in run_step self._trainer.run_step() File "/media/HDD1/Video_CutLER/videocutler/mask2former_video/engine/train_loop.py", line 367, in run_step data = self.copy_and_paste(copy.deepcopy(data[::-1]), data) File "/media/HDD1/Video_CutLER/videocutler/mask2former_video/engine/train_loop.py", line 182, in copy_and_paste masks_new = F.interpolate(copied_masks.tensor[None, ...].float(), size=(h_new, w_new), mode="bilinear", align_corners=False).bool().squeeze(0) File "/home/anaconda3/envs/cutler/lib/python3.8/site-packages/torch/nn/functional.py", line 4038, in interpolate return torch._C._nn.upsample_bilinear2d(input, output_size, align_corners, scale_factors) RuntimeError: Non-empty 4D data tensor expected but got a tensor with sizes [1, 0, 480, 513]

It seems like the copied masks sometimes get 0 instances. Could you advise how to resolve this issue?

Thanks a lot! Appreciate your great work!

[frank-xwang]

Hi! Here's my take on it, I guess the issues might be: 1) Some instances are too small, therefore were filtered out. or

2) Some videos/frames don't have any instances detected by MaskCut or other pseudo-labeling methods.

You may want to double check your annotation file.

Hope it helps!

[YangYafei1998]

Thanks for your quick reply!

For the annotation file, I directly use video_imagenet_train_fixsize480_tau0.15_N3.json downloaded from https://drive.google.com/file/d/1gllHvrZQNVXphnk-IQxMcXh87Qs86ofT/view. Is there any preprocessing that needs to be performed on these annotations before training?

I notice that at CutLER/videocutler/mask2former_video/engine/train_loop.py, between line 150 and 262, the copied instances are changed along the target frames. Based on my understanding, is it supposed to use the same copied instances for each frame, and then scale and shift copied instances differently and independently for each frame?

Looking forward to your further advice. Really appreciate your help!