Best config for im_undistort_lite/02_Flames.json

[remmel]

Hello Li,

I want to reproduce the _02Flames (Google Immersive dataset).

I ran 02_Flames_undist will following almost default params:

im_undistort_lite/02_Flames.json:

{
  "resolution": 2,
  "model": "ours_lite",
  "scaling_lr": 0.001,
  "trbfslinit": 4.0,
  "preprocesspoints": 3,
  "test_iteration": 25000,
  "densify": 3,
  "emsstart": 30000,
  "desicnt": 6,
  "duration": 5, #changed from 50 to 5 to fit my 12GB VRAM
  "rgbfunction": "None",
  "loader": "technicolor",
  "rdpip": "train_ours_lite"
}

python train.py --gtmask 1 --config configs/im_undistort_lite/02_Flames.json --model_path log/02_Flames_undist/01 --source_path ../dataset/immersive/02_Flames_undist/colmap_0

I get low quality result as seen on the screenshot:

However after playing with the parameters, I get better result, specifically after removing the trbfslinit :

Logs are here: GDrive - log 01-dur5 and then 01-dur5-res4-notrbfslinit

However result are not good enough, specifically comparing with _02flames from https://antimatter15.com/splaTV/ or _flamesteak using n3d_lite/flame_steak.json My guess is that there are not enough splats.

Thus, what is the best configuration you found for _im_undistort_lite/02Flames.json ? You paper is using _imdistort[lite|full] isn't it? _imundistort should reach same quality than _imdistort ? Why did not you tried to go further with _imundistort version?

I know that I'm not using full resolution, but it should be way enough comparing with my n3d tries.

[edit] train_imdist.py lite comparison

as comparison, this is the result obtained using _train_imdist.py lite_, with _resolution:4_ and _duration:5_ (but something strange seems to occurs with the time) :  ``` python train_imdist.py --config configs/im_distort_lite/02_Flames.json --model_path log/02_Flames_dist/01-lite-dur5 --source_path ../dataset/immersive/02_Flames_dist/colmap_0 ``` see log _02_Flames_dist/01-lite-dur5-res4/_ commenting https://github.com/oppo-us-research/SpacetimeGaussians/blob/7eba660bc8766528c18b87f8b4194d142de9b137/thirdparty/gaussian_splatting/utils/camera_utils.py#L121 and https://github.com/oppo-us-research/SpacetimeGaussians/blob/7eba660bc8766528c18b87f8b4194d142de9b137/thirdparty/gaussian_splatting/utils/camera_utils.py#L141

Thanks for your help!

[lizhan17]

1) for the im_undistort_lite config: we need 50 frames such that points from 50 frames can cover each other for some static points. you can think it is a video codec with 50 frames. it can be much better than the 5 frames model.

if you use 5 frames, the default trbfinit (which increase the range of each frame can cover in time compared to trbfinit=4) can also increase the quality as each frame can cover more timestamps.

for the demo model on the website, we use all the cameras to train with 50 frames.

2) for the train_imdist config:

for immersive models that are used to report results in the paper, the we are using camera super sampling (camss) to render at original resolution (to compensate the fisheye warping error). As other methods are trained with half resolution ground truth images, so i force the code to render at original resolution and use half resolution image as ground truth label.

[remmel]

Thanks, I'll thus try on a bigger VRAM machine with the 50 frames, I reopen if needed