search

HuajianUP / Photo-SLAM

[CVPR 2024] Photo-SLAM: Real-time Simultaneous Localization and Photorealistic Mapping for Monocular, Stereo, and RGB-D Cameras
GNU General Public License v3.0
295 stars 25 forks source link

how to render the image from random viewpoint #15

Closed vvcatstar closed 1 week ago

vvcatstar commented 2 weeks ago

I have successfully run your code; it’s a nice work. However, I have some issues when applying it and hope to get your help.

  1. The final results only include the render image and camera outputs; the GS results are not found. If I want to input a random camera pose and render the image from that viewpoint, how should I proceed?
  2. I have test on my own iphone camera data use monocular mode, but the result is not well, here is the results. The left image is input image and right image is rendered image. I used the Monocular/Replica/iffuce0.yaml gaussian_mapper parameters. Which parameters can I adjust to improve the results? image
  3. I have test on my fisheye camera(gopro9) data use monocular mode, but the code was segmentation fault. I set the camera parameters use Camera.type: "KannalaBrandt8". 
    
# The camera key parameters is
Camera.type: "KannalaBrandt8"
# Camera calibration and distortion parameters (OpenICC https://github.com/urbste/OpenImuCameraCalibrator) 
Camera1.fx: 460.2889442852628 # 190.97847715128717
Camera1.fy: 460.2889442852628 # 190.9733070521226
Camera1.cx: 489.79113984156373 # 254.93170605935475
Camera1.cy: 364.921271706051 # 256.8974428996504

Camera1.k1: 0.038476731454432096 # 0.0034823894022493434 Camera1.k2: 0.07948296976400737 # 0.0007150348452162257 Camera1.k3: -0.09249199643907653 # -0.0020532361418706202 Camera1.k4: 0.03459679710611075 # 0.00020293673591811182

Camera resolution

Camera.width: 960 Camera.height: 720

The log is:

CUDA available! Training on GPU.

ORB-SLAM3 Copyright (C) 2017-2020 Carlos Campos, Richard Elvira, Juan J. Gómez, José M.M. Montiel and Juan D. Tardós, University of Zaragoza. ORB-SLAM2 Copyright (C) 2014-2016 Raúl Mur-Artal, José M.M. Montiel and Juan D. Tardós, University of Zaragoza. This program comes with ABSOLUTELY NO WARRANTY; This is free software, and you are welcome to redistribute it under certain conditions. See LICENSE.txt.

Input sensor was set to: Monocular Loading settings from ./cfg/ORB_SLAM3/Monocular/gopro/gopro9_L.yaml -Loaded camera 1 Camera.newHeight optional parameter does not exist... Camera.newWidth optional parameter does not exist... -Loaded image info -Loaded ORB settings Viewer.imageViewScale optional parameter does not exist... -Loaded viewer settings System.LoadAtlasFromFile optional parameter does not exist... System.SaveAtlasToFile optional parameter does not exist... -Loaded Atlas settings System.thFarPoints optional parameter does not exist... -Loaded misc parameters

SLAM settings: -Camera 1 parameters (Kannala-Brandt): [ 460.289 460.289 489.791 364.921 0.0384767 0.079483 -0.092492 0.0345968 ] -Original image size: [ 960 , 720 ] -Current image size: [ 960 , 720 ] -Sequence FPS: 60 -Features per image: 2000 -ORB scale factor: 1.2 -ORB number of scales: 8 -Initial FAST threshold: 20 -Min FAST threshold: 7

Loading ORB Vocabulary. This could take a while... Vocabulary loaded!

Initialization of Atlas from scratch Creation of new map with id: 0 Creation of new map with last KF id: 0 Seq. Name: There are 1 cameras in the atlas Camera 0 is fisheye [Gaussian Mapper]CUDA available! Training on GPU. [Gaussian Mapper]Reading parameters from "./cfg/gaussian_mapper/Monocular/gopro/gopro9_L.yaml" [1] 3580226 segmentation fault (core dumped) ./bin/gopro_mono ./ORB-SLAM3/Vocabulary/ORBvoc.txt



Thank you.
HuajianUP commented 2 weeks ago

Hi, Thank you for your interest and testing our code.

1) The default path to store the PLY file is `SEQ_RESULTS/xxxxx_shutdown/ply/point_cloud/iteration_xxxxx/point_cloud.ply. You can use any viewers compatible with GS visualization to load the PLY file and check the results. Moreover, you can also refer to our evaluation code.

2) You should ensure the intrinsic model of the iPhone camera is calibrated. As an online system, it will stop optimization once tracking is finished. When we continue to visit new places, the newly added Keyframes and primitives have not been thoroughly optimized. A simple solution is to keep optimizing after processing all frames. In addition, you can adjust the densification parameters for fine-tuning.

3) We rectify the input images to undistorted perspective images before photorealistic mapping. Currently, the code is not compatible with fisheye images yet, refer to camera.h. To address this issue, you can add an OpenCV fisheye model there, cv::fisheye::initUndistortRectifyMap. And we will update our code and support it later.

vvcatstar commented 2 weeks ago

Thank you for your reply, I will calibrate my camera and then to test. I want to use /bin/view_result to show the ply result, but error. The run code is:

./bin/view_result ./cfg/gaussian_mapper/Monocular/Replica/replica_mono.yaml ./cfg/ORB_SLAM3/Monocular/Replica/office0.yaml ./results/replica_mono_0/office0/5481_shutdown/ply/point_cloud/iteration_5481/point_cloud.ply

The result is:

terminate called after throwing an instance of 'std::runtime_error'
  what():  [Gaussian Mapper]Unsupported camera model: /home/zyw/data/repo_common/Photo-SLAM/cfg/ORB_SLAM3/Monocular/Replica/office0.yaml

The camera.type in office0.yaml is 'Pinhole', which is not 'Pinhole' in gaussian_mapper.cpp. So I change the office0.yaml 'PinHole' to 'Pinhole' and run again. But error again. Here is the log:

CUDA available! Training on GPU.
[Gaussian Mapper]CUDA available! Training on GPU.
[Gaussian Mapper]Reading parameters from "./cfg/gaussian_mapper/Monocular/Replica/replica_mono.yaml"
        [ply_header] Type: binary
        [ply_header] element: vertex (80535)
        [ply_header]    property: x (type=float)
        [ply_header]    property: y (type=float)
        [ply_header]    property: z (type=float)
        [ply_header]    property: nx (type=float)
        [ply_header]    property: ny (type=float)
        [ply_header]    property: nz (type=float)
        [ply_header]    property: f_dc_0 (type=float)
        [ply_header]    property: f_dc_1 (type=float)
        [ply_header]    property: f_dc_2 (type=float)
        [ply_header]    property: f_rest_0 (type=float)
        [ply_header]    property: f_rest_1 (type=float)
        [ply_header]    property: f_rest_2 (type=float)
        [ply_header]    property: f_rest_3 (type=float)
        [ply_header]    property: f_rest_4 (type=float)
        [ply_header]    property: f_rest_5 (type=float)
        [ply_header]    property: f_rest_6 (type=float)
        [ply_header]    property: f_rest_7 (type=float)
        [ply_header]    property: f_rest_8 (type=float)
        [ply_header]    property: f_rest_9 (type=float)
        [ply_header]    property: f_rest_10 (type=float)
        [ply_header]    property: f_rest_11 (type=float)
        [ply_header]    property: f_rest_12 (type=float)
        [ply_header]    property: f_rest_13 (type=float)
        [ply_header]    property: f_rest_14 (type=float)
        [ply_header]    property: f_rest_15 (type=float)
        [ply_header]    property: f_rest_16 (type=float)
        [ply_header]    property: f_rest_17 (type=float)
        [ply_header]    property: f_rest_18 (type=float)
        [ply_header]    property: f_rest_19 (type=float)
        [ply_header]    property: f_rest_20 (type=float)
        [ply_header]    property: f_rest_21 (type=float)
        [ply_header]    property: f_rest_22 (type=float)
        [ply_header]    property: f_rest_23 (type=float)
        [ply_header]    property: f_rest_24 (type=float)
        [ply_header]    property: f_rest_25 (type=float)
        [ply_header]    property: f_rest_26 (type=float)
        [ply_header]    property: f_rest_27 (type=float)
        [ply_header]    property: f_rest_28 (type=float)
        [ply_header]    property: f_rest_29 (type=float)
        [ply_header]    property: f_rest_30 (type=float)
        [ply_header]    property: f_rest_31 (type=float)
        [ply_header]    property: f_rest_32 (type=float)
        [ply_header]    property: f_rest_33 (type=float)
        [ply_header]    property: f_rest_34 (type=float)
        [ply_header]    property: f_rest_35 (type=float)
        [ply_header]    property: f_rest_36 (type=float)
        [ply_header]    property: f_rest_37 (type=float)
        [ply_header]    property: f_rest_38 (type=float)
        [ply_header]    property: f_rest_39 (type=float)
        [ply_header]    property: f_rest_40 (type=float)
        [ply_header]    property: f_rest_41 (type=float)
        [ply_header]    property: f_rest_42 (type=float)
        [ply_header]    property: f_rest_43 (type=float)
        [ply_header]    property: f_rest_44 (type=float)
        [ply_header]    property: opacity (type=float)
        [ply_header]    property: scale_0 (type=float)
        [ply_header]    property: scale_1 (type=float)
        [ply_header]    property: scale_2 (type=float)
        [ply_header]    property: rot_0 (type=float)
        [ply_header]    property: rot_1 (type=float)
        [ply_header]    property: rot_2 (type=float)
        [ply_header]    property: rot_3 (type=float)
        Read 80535 total xyz 
        Read 80535 total f_dc 
        Read 80535 total f_rest 
        Read 80535 total opacity 
        Read 80535 total scales 
        Read 80535 total rot 
terminate called after throwing an instance of 'cv::Exception'
  what():  OpenCV(4.7.0) /home/zyw/data/environment/opencv470/opencv-4.7.0/modules/imgproc/src/imgwarp.cpp:1703: error: (-215:Assertion failed) !_map1.empty() in function 'remap'

[1]    2999218 abort (core dumped)  ./bin/view_result ./cfg/gaussian_mapper/Monocular/Replica/replica_mono.yaml
HuajianUP commented 2 weeks ago

1) For evaluation, please refer to Photo-SLAM-eval

2) Simply modifying the .yaml file won't work for fisheye images. We need to change the undistortion function in camera.h to support the KB camera model.