segment problem when using custom dataset

[Lin239522]

To build the semantic segmentation dataset, I ran scripts/mask2former_infer/infer.sh

export CUDA_VISIBLE_DEVICES=0
/home/ubuntu/anaconda3/envs/rome/bin/python3.8 /home/ubuntu/Project/RoMe/scripts/mask2former_infer/inference.py  \
  --config-file /home/ubuntu/Project/Mask2Former/configs/mapillary-vistas/semantic-segmentation/swin/maskformer2_swin_large_IN21k_384_bs16_300k.yaml \
  --opts MODEL.WEIGHTS /home/ubuntu/Project/Mask2Former/configs/mapillary-vistas/semantic-segmentation/swin/model_final_90ee2d.pkl

And in inference.py, change the folder path to the path of the custom dataset

However, there are noises similar to the boundary on render_gt_seg_x.png . Can you tell me whether this will affect the rendering result? How can I remove these noises? render_gt_seg: seg_sequences: sequences:

[DRosemei]

@Lin239522 You may need use cv2.INTER_NEAREST to resize semantic images.

[Lin239522]

@Lin239522 You may need use cv2.INTER_NEAREST to resize semantic images.

@DRosemei Thanks for the quick response！ However, I did not modify the resize code in the getitem function in RoMe.：RGB image resize uses cv2.INTER_LINEAR, and semantic segmentation image resize uses cv2.INTER_NEAREST.

 K = self.cameras_K[idx]
 D = self.distortion
 h, w = input_image.shape[:2]
 input_image = cv2.fisheye.undistortImage(input_image, K, D, Knew=K, new_size=(w, h))
 resized_image = cv2.resize(input_image, dsize=self.resized_image_size, interpolation=cv2.INTER_LINEAR)
…
 label = cv2.fisheye.undistortImage(label, K, D, Knew=K, new_size=(w, h))
 resized_label = cv2.resize(label, dsize=self.resized_image_size, interpolation=cv2.INTER_NEAREST)

In addition, in the configuration file, I set the image size to be the same as the input image (because I saw that you did this in the kitti configuration)

image_width: 3840
image_height: 1536

Is there anything wrong with my above operation?

[Lin239522]

When I reduce the image size in config to 960*384, the problem still exists

[DRosemei]

@Lin239522
label = cv2.fisheye.undistortImage(label, K, D, Knew=K, new_size=(w, h)) may cause the problem.

[Lin239522]

@Lin239522 label = cv2.fisheye.undistortImage(label, K, D, Knew=K, new_size=(w, h)) may cause the problem. Thank you for your suggestion！！ ·v· This is the result without de-distortion. You can see that the lane lines on the road are curved. but the same problem occurred QAQ I am wondering that，in the visualized_output of mask2former， the boundaries of each category are drawn . Could this be the reason for the result?

[DRosemei]

@Lin239522 You can visualize label image before and after "cv2.fisheye.undistortImage" in source resolution to check.

[DRosemei]

@Lin239522 Please visualize label image before "cv2.fisheye.undistortImage", that is visualizing fisheye image by colormap. "cv2.fisheye.undistortImage" will interpolate labels and use cv2.remap instead

[Lin239522]

@DRosemei This photo is label image before "cv2.fisheye.undistortImage", I just ran infer.sh, and then visualizing fisheye image by colormap. Could this be due to the limitations of the official model, which cannot identify domestic roads?

[Lin239522]

Dear @DRosemei ,Thank you again for your enthusiastic answer. I have another question. What is the reason for the box-like defects that appear in the rendered image?

eval_117-render.png: eval_117-vis_gt_seg.png: eval_117-vis_seg.png: eval_117-blend.png:

[Lin239522]

Hey @DRosemei , you are right! The cv2.fisheye.undistortImage function did cause the problem. How can I solve it? Should I use FishEyeCameras to render the mesh instead of using cv2.fisheye.undistortImage on the input images?

[DRosemei]

@Lin239522

1. "box-like defects" are caused by pytorch3D, it may happen somtimes
2. Use cv2.remap to undistort fisheye label by cv2.INTER_NEAREST

   
   cv_image = cv2.imread(image_path, -1)

mapx, mapy = cv2.initUndistortRectifyMap(k, d, None, k, (cv_image.shape[1], cv_image.shape[0]), cv2.CV_32FC1)

cv_image_undistorted = cv2.remap(cv_image, mapx, mapy, cv2.INTER_NEAREST)

[Lin239522]

thanks! it works ·v·