Open TurtleZhong opened 1 month ago
Put them both at the same training and put all pictures. At last this green hause should be better. More you cover from both and more views - better will be result cus algorithms will have more precise positioning in the space. (less floaters and better interpolation)
@jaco001 despite of the green house, I found the render results of vehicles in two views are really different. In the right-view, may be more views of the vehicle, so the results of vehicle is really good even if moving render-view around. But in the front-view case, the vehicles on the road sides seems blur or floters even if the render-view is the same pose with the training poses. I also tried put them both to train but the results of vehicles are still bad…:sob:
With the front view you have more transformation like scaling and more dynamic angles. So there is more potential degradation. If you don't compensate it with eg 45 camera to the right nothing change. If you don't put late right side pictures nothing change too. You must cross each other cameras more. Right side is less 'broken' cuz you only move camera with moving vector. No scaling. If you move camera forward more - your view will be broken too like front view.
When I used the autonomous driving data set for training, I discovered a very strange phenomenon. If I use the front-view camera, the training results are very poor, but if I use the right-view image data, the training results It's going to be really good. My guess is that it is difficult to reconstruct the forward motion image itself, and there are too many point clouds in one fov. In addition, when I checked the results, I found that the points in the sky were learned very low, resulting in the appearance of Some ghosts. Below are some results I did using images from the pandaset 053 sequence. I only used 10 images with ground truth poses for testing because I used the entire sequence (a total of 80 images from one perspective) and found that the training results of the front-view camera were very bad. However, The results for the right-view image are very good.
The dataset I used is here: 053_front_10_images.zip 053_right_10_images.zip
the structure is like:
I used these commands to train:
and I got these log of front and right datasets
front-view
right-view
The following videos are visualizations of the training results of the front-view camera and the right-view camera in the same scene.
https://github.com/graphdeco-inria/gaussian-splatting/assets/19700579/27c746de-477d-4a35-af45-f3e3eb36e102
https://github.com/graphdeco-inria/gaussian-splatting/assets/19700579/857bd0aa-8564-4229-880a-82784d3a04e0
I also tried train the GS using the lidar pointclouds as prior, but I got the same results that the front-view got bad results. So what I want to know is there any way to make the training results of forward motion better?