Inaccurate render result

[lys5588]

Hi, I'm testing Log with my dataset using the same config setting of feicuiwan. The render images are excellent and the validation results are good, the PSNR reaches 28. But when I use the gui.py to generate a GUI, the rendering result has lots of missing primitive and incorrectness, the input data are the campus data and are supposed to be like the realtime_demo_campus.mp4 shows.

The code is the latest version, and, the data preprocessing and training are the same as the campus dataset process in preprocess.md showed

My Cuda version is 11.8 and my torch version is 2.3.0 + cu118, the training didn't occur error.

Here's my validation image:

Here's my GUI result:

Here's my GUI result of feicuiwan dataset with the same config setting,

[lys5588]

Here's my config

dataset.yml

root: data/Yuehai_downsample
# i replece the sparse data with the space_align result
PLYNAME: data/Yuehai_downsample/sparse_align/sparse.npz
scale3d: 1.
xyz_scale: 1.
ground_height: &ground_height 1.18

#the updated template
dataset:
  module: LoG.dataset.colmap.ImageDataset
  args:
    root: $root
    pre_undis: True
    share_camera: False
    cameras: sparse_align
    scales: [1, 2, 4, 8]
    crop_size: [-1, -1]
    znear: 0.001
    zfar: 100.
    scale3d: $scale3d
    ext: .JPG

val_dataset:
  module: LoG.dataset.colmap.ImageDataset
  args:
    root: $root
    namelist:
      - DJI_0145
      - DJI_0245
      - DJI_0345
      - DJI_0445
      - DJI_0545
      - DJI_0645
      - DJI_0745
      - DJI_0845
      - DJI_0945
      - DJI_0400
      - DJI_0500
      - DJI_0185
      - DJI_0285
      - DJI_0385
      - DJI_0485
      - DJI_0585
      - DJI_0685
      - DJI_0785
      - DJI_0885
      - DJI_0985
    cache: cache_val.pkl
    pre_undis: True
    share_camera: False
    scales: [ 4 ]
    znear: 0.001
    zfar: 100.
    scale3d: $scale3d
    ext: .JPG

demo_interpolate:
  model_state:
    enable_sh: True
    log_query: False
  render_state:
    background: [1., 1., 1.]
  dataset:
    module: LoG.dataset.demo.InterpolatePath
    args:
      cameras: $PLYNAME
      scale: 4
      steps: 300
      subs:
        - DJI_0145
        - DJI_0245
        - DJI_0345
        - DJI_0445
        - DJI_0545
        - DJI_0645
        - DJI_0745
        - DJI_0845
        - DJI_0945
        - DJI_0400
        - DJI_0500
        - DJI_0185
        - DJI_0285
        - DJI_0385
        - DJI_0485
        - DJI_0585
        - DJI_0685
        - DJI_0785
        - DJI_0885
        - DJI_0985

demo_level:
  model_state:
    enable_sh: True
    log_query: False
  dataset:
    module: LoG.dataset.demo.ShowLevel
    args:
      cameras: $PLYNAME
      steps: 15000
      sub:
        - DJI_0645
      scale: 4

demo_pixel:
  model_state:
    enable_sh: True
    log_query: True
  dataset:
    module: LoG.dataset.demo.ShowLevel
    args:
      mode: pixel
      cameras: $PLYNAME
      steps: 15000
      sub:
        - DJI_0645
      scale: 4

demo_lod:
  model_state:
    enable_sh: True
    log_query: False
  dataset:
    module: LoG.dataset.demo.ZoomInOut
    args:
      cameras: $PLYNAME
      sub:
        - DJI_0645
      zranges: [-20., 1.]
      scale: 2
      use_logspace: False

level_of_gaussian.yml

max_steps: 600

model:
  module: LoG.model.level_of_gaussian.LoG
  args:
    use_view_correction: True
    gaussian:
      xyz_scale: $xyz_scale
      sh_degree: 1
      init_ply:
        scale3d: $scale3d
        filename: $PLYNAME
        init_opacity: 0.1
    optimizer:
      optimize_keys: [xyz, colors, scaling, opacity, rotation, shs]
      opt_all_levels: True # optimize all level or not
      lr_dict:
        xyz: 0.00016
        xyz_final: 0.0000016
        xyz_scale: $xyz_scale
        colors: 0.0025
        shs: 0.000125
        scaling: 0.005
        opacity: 0.05 # lr_opacity > lr_scaling
        rotation: 0.001
        max_steps: $max_steps # 30_000
    tree:
      max_child: 4
      max_level: 30
    densify_and_remove:
      # add sh degree
      upgrade_sh_iter: 10
      densify_from_iter: 1
      densify_every_iter: 1
      upgrade_repeat: 50
      # init
      init_split_method: split_by_2d
      init_radius_min: 4
      init_radius_split: 16
      init_weight_min: 0.1
      min_steps: 50
      # densify
      method: naive
      split_grad_thres: 0.0002
      radius2d_thres: 6
      remove_weights_thres: 0.005
      max_split_points: 100000
      sort_method: radii
      min_steps_split: 100
      #
      scaling_decay: 0.9

stage_8_4.yml

num_workers: &num_workers 0

NAIVE_STAGE:
  init:
    loader:
      module: iteration
      args:
        batch_size: 1
        iterations: 250
        num_workers: *num_workers
    dataset_state:
      scale: 8
    model_state: {}
  tree:
    loader:
      module: iteration
      args:
        batch_size: 1
        iterations: 800
        num_workers: *num_workers
    dataset_state:
      scale: 4
    render_state:
      render_depth: False
    model_state:
      enable_sh: True

train.yml

experiment_name: urbanbis/Yuehai_downsample   # dataset/testname
parents:
  - config/urbanbis/Yuehai_downsample/dataset.yml
  - config/urbanbis/Yuehai_downsample/level_of_gaussian.yml
  - config/urbanbis/Yuehai_downsample/stage_8_4.yml

exp: output/urbanbis/Yuehai_downsample_retest/log
gpus: [0]

log_interval: 1_000
save_interval: 100_000

max_steps: 750

RGB_RENDER_L1_SSIM:
  module: LoG.render.renderer.NaiveRendererAndLoss
  args:
    use_origin_render: False
    use_randback: True

train:
  dataset: $dataset
  render: $RGB_RENDER_L1_SSIM
  stages: $NAIVE_STAGE
  init:
    method: scale_min
    dataset_state:
      scale: 4

val:
  dataset: $val_dataset
  iteration: 10000

[chingswy]

It looks strange. Did you use the complete Campus dataset for training? I noticed that the first image includes the sky, and currently, our algorithm does not specifically process the sky, which might result in additional floating objects in the air. The public Campus dataset I used does not contain perspectives that capture the sky, so you might need to check your training data.

[dendenxu]

This looks more like there are some floaters up in the sky blocking the views instead of missing primitives. Could you try using w a s d and v c, q, e, 1 2 to try to navigate down closer to earth and check whether the rendering results get better?

[lys5588]

Thanks for the reply.I'm now retrain the scene with a totally new python environment and re-cloned code to avoid error of my modification. For the data, I'm using the urbanBIS Yuehai dataset, my image set includes 1030 images. The images were all downsampled by the cv2.pyrDown with a ratio 0.5, which I think may not be the reason of error.

My screen shot of my dataset folder is here https://github.com/lys5588/LoG/blob/main/gui_screenshot/dataset_folder.png Here are more images of my render result with the campus scene with the prefix of gui: https://github.com/lys5588/LoG/tree/main/gui_screenshot