search

LightwheelAI / street-gaussians-ns

Unofficial implementation of "Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splatting", ECCV2024.
Apache License 2.0
321 stars 31 forks source link

where is points3D_withlidar.txt #24

Open llianxu opened 4 months ago

llianxu commented 4 months ago

Thanks for your impressive work! I downloaded the preprocessed data, but when I ran train.sh, I found that there was no points3D_withlidar.txt file. I want to ask if there is a replacement file or if I missed a step.

wangyida commented 4 months ago

generate with scripts/shells/points_cloud_generate.sh

llianxu commented 4 months ago

OK，I will try this script，Thanks！

yuanhaorannnnnn commented 4 months ago

generate with scripts/shells/points_cloud_generate.sh

it will gen points3d_withlidar.bin instead txt file, it's ok for training?

1400013531 commented 3 months ago

generate with scripts/shells/points_cloud_generate.sh

it will gen points3d_withlidar.bin instead txt file, it's ok for training?

yeah, I used it for training and got the correct result. You just need to change the file name in train.sh

data_root=$1
cuda_id=$2

mkdir -p output/

CUDA_VISIBLE_DEVICES=$cuda_id sgn-train street-gaussians-ns \
    --experiment_name street-gaussians-ns \
    --output_dir output/ \
    --vis viewer+wandb \
    --viewer.quit_on_train_completion True \
    colmap-data-parser-config \
    --data $data_root \
    --colmap_path colmap/sparse/0 \
    --load_3D_points True \
    --max_2D_matches_per_3D_point 0 \
    --undistort True \
    --segments-path segs \
    --filter_camera_id 1 2 3 4 5\
    --init_points_filename points3D_withlidar.bin         # this file name
jycc1991 commented 3 months ago

generate with scripts/shells/points_cloud_generate.sh

it will gen points3d_withlidar.bin instead txt file, it's ok for training?

yeah, I used it for training and got the correct result. You just need to change the file name in train.sh

data_root=$1
cuda_id=$2

mkdir -p output/

CUDA_VISIBLE_DEVICES=$cuda_id sgn-train street-gaussians-ns \
    --experiment_name street-gaussians-ns \
    --output_dir output/ \
    --vis viewer+wandb \
    --viewer.quit_on_train_completion True \
    colmap-data-parser-config \
    --data $data_root \
    --colmap_path colmap/sparse/0 \
    --load_3D_points True \
    --max_2D_matches_per_3D_point 0 \
    --undistort True \
    --segments-path segs \
    --filter_camera_id 1 2 3 4 5\
    --init_points_filename points3D_withlidar.bin         # this file name

I set --filter_camera_id to 1 2 3 4 5,I got this error: [16:26:28] Caching / undistorting eval images sgn_datamanager.py:173 [16:26:29] Caching / undistorting train images sgn_datamanager.py:173 scripts/shells/train.sh：行 19: 82357 已杀死 CUDA_VISIBLE_DEVICES=$cuda_id sgn-train street-gaussians-ns --experiment_name street-gaussians-ns --output_dir output2/ --vis viewer+tensorboard --viewer.quit_on_train_completion True colmap-data-parser-config --data $data_root --colmap_path colmap/sparse/0 --load_3D_points True --max_2D_matches_per_3D_point 0 --undistort True --segments-path segs --filter_camera_id 1 2 3 4 5 --init_points_filename points3D_withlidar.bin Can you tell me how to set to solve this? Thanks

1400013531 commented 3 months ago

generate with scripts/shells/points_cloud_generate.sh

it will gen points3d_withlidar.bin instead txt file, it's ok for training?

yeah, I used it for training and got the correct result. You just need to change the file name in train.sh

data_root=$1
cuda_id=$2

mkdir -p output/

CUDA_VISIBLE_DEVICES=$cuda_id sgn-train street-gaussians-ns \
    --experiment_name street-gaussians-ns \
    --output_dir output/ \
    --vis viewer+wandb \
    --viewer.quit_on_train_completion True \
    colmap-data-parser-config \
    --data $data_root \
    --colmap_path colmap/sparse/0 \
    --load_3D_points True \
    --max_2D_matches_per_3D_point 0 \
    --undistort True \
    --segments-path segs \
    --filter_camera_id 1 2 3 4 5\
    --init_points_filename points3D_withlidar.bin         # this file name

I set --filter_camera_id to 1 2 3 4 5,I got this error: [16:26:28] Caching / undistorting eval images sgn_datamanager.py:173 [16:26:29] Caching / undistorting train images sgn_datamanager.py:173 scripts/shells/train.sh：行 19: 82357 已杀死 CUDA_VISIBLE_DEVICES=$cuda_id sgn-train street-gaussians-ns --experiment_name street-gaussians-ns --output_dir output2/ --vis viewer+tensorboard --viewer.quit_on_train_completion True colmap-data-parser-config --data $data_root --colmap_path colmap/sparse/0 --load_3D_points True --max_2D_matches_per_3D_point 0 --undistort True --segments-path segs --filter_camera_id 1 2 3 4 5 --init_points_filename points3D_withlidar.bin Can you tell me how to set to solve this? Thanks

Oh, sorry for my mistake. This parameter [-filter_camera_id 1 2 3 4 5] is used for choosing which cameras you are going to train with. You can set it according to your training data. For example, if you want to train with only one camera data in your training data including five cameras, just set [-filter_camera_id 1]. It seems that your error was caused by choosing non existing camera. Setting [-filter_camera_id 1] might help

yuanhaorannnnnn commented 3 months ago

generate with scripts/shells/points_cloud_generate.sh

it will gen points3d_withlidar.bin instead txt file, it's ok for training?

yeah, I used it for training and got the correct result. You just need to change the file name in train.sh

data_root=$1
cuda_id=$2

mkdir -p output/

CUDA_VISIBLE_DEVICES=$cuda_id sgn-train street-gaussians-ns \
    --experiment_name street-gaussians-ns \
    --output_dir output/ \
    --vis viewer+wandb \
    --viewer.quit_on_train_completion True \
    colmap-data-parser-config \
    --data $data_root \
    --colmap_path colmap/sparse/0 \
    --load_3D_points True \
    --max_2D_matches_per_3D_point 0 \
    --undistort True \
    --segments-path segs \
    --filter_camera_id 1 2 3 4 5\
    --init_points_filename points3D_withlidar.bin         # this file name

yes, same method and it works fine

jycc1991 commented 3 months ago

generate with scripts/shells/points_cloud_generate.sh

it will gen points3d_withlidar.bin instead txt file, it's ok for training?

yeah, I used it for training and got the correct result. You just need to change the file name in train.sh

data_root=$1
cuda_id=$2

mkdir -p output/

CUDA_VISIBLE_DEVICES=$cuda_id sgn-train street-gaussians-ns \
    --experiment_name street-gaussians-ns \
    --output_dir output/ \
    --vis viewer+wandb \
    --viewer.quit_on_train_completion True \
    colmap-data-parser-config \
    --data $data_root \
    --colmap_path colmap/sparse/0 \
    --load_3D_points True \
    --max_2D_matches_per_3D_point 0 \
    --undistort True \
    --segments-path segs \
    --filter_camera_id 1 2 3 4 5\
    --init_points_filename points3D_withlidar.bin         # this file name

yes, same method and it works fine

I download preprocessed dataset from this repo which is 10448102132863604198_472_000_492_000,and cameras.txt in /colmap shows it has 5 cameras: 1 OPENCV 1920 1280 2044.3189893365634 2044.3189893365634 966.3197697401635 624.7706323955747 0.048519645404847 -0.31855151741237414 -0.0005279507169045765 -0.0010850539443264748 2 OPENCV 1920 1280 2064.096136816171 2064.096136816171 965.2989897820795 643.2419196405554 0.04536665392932437 -0.3481585691089555 0.0023209650920834515 0.000879426586508394 3 OPENCV 1920 1280 2047.353864208166 2047.353864208166 959.6234340600746 644.2507554214233 0.03923964629713441 -0.30937484864506715 0.0016984481880601527 -0.0009807100492894205 4 OPENCV 1920 886 2052.166978577554 2052.166978577554 964.3304222948891 251.3045142973026 0.04792858845797487 -0.3115144801129193 0.00044896409868372216 0.0006301442821205557 5 OPENCV 1920 886 2060.943963183803 2060.943963183803 961.8321926331334 243.845105660424 0.04466498449403254 -0.33887981610387924 0.0015137815608006677 4.743970552947572e-05 So I think maybe is there any other settings I should set?

1400013531 commented 3 months ago

generate with scripts/shells/points_cloud_generate.sh

it will gen points3d_withlidar.bin instead txt file, it's ok for training?

yeah, I used it for training and got the correct result. You just need to change the file name in train.sh

data_root=$1
cuda_id=$2

mkdir -p output/

CUDA_VISIBLE_DEVICES=$cuda_id sgn-train street-gaussians-ns \
    --experiment_name street-gaussians-ns \
    --output_dir output/ \
    --vis viewer+wandb \
    --viewer.quit_on_train_completion True \
    colmap-data-parser-config \
    --data $data_root \
    --colmap_path colmap/sparse/0 \
    --load_3D_points True \
    --max_2D_matches_per_3D_point 0 \
    --undistort True \
    --segments-path segs \
    --filter_camera_id 1 2 3 4 5\
    --init_points_filename points3D_withlidar.bin         # this file name

yes, same method and it works fine

I download preprocessed dataset from this repo which is 10448102132863604198_472_000_492_000,and cameras.txt in /colmap shows it has 5 cameras: 1 OPENCV 1920 1280 2044.3189893365634 2044.3189893365634 966.3197697401635 624.7706323955747 0.048519645404847 -0.31855151741237414 -0.0005279507169045765 -0.0010850539443264748 2 OPENCV 1920 1280 2064.096136816171 2064.096136816171 965.2989897820795 643.2419196405554 0.04536665392932437 -0.3481585691089555 0.0023209650920834515 0.000879426586508394 3 OPENCV 1920 1280 2047.353864208166 2047.353864208166 959.6234340600746 644.2507554214233 0.03923964629713441 -0.30937484864506715 0.0016984481880601527 -0.0009807100492894205 4 OPENCV 1920 886 2052.166978577554 2052.166978577554 964.3304222948891 251.3045142973026 0.04792858845797487 -0.3115144801129193 0.00044896409868372216 0.0006301442821205557 5 OPENCV 1920 886 2060.943963183803 2060.943963183803 961.8321926331334 243.845105660424 0.04466498449403254 -0.33887981610387924 0.0015137815608006677 4.743970552947572e-05 So I think maybe is there any other settings I should set?

Yeah, I used this dataset too. So we can eliminate error with dataset. Can you provide more log or more information about environment?

jycc1991 commented 3 months ago

generate with scripts/shells/points_cloud_generate.sh

it will gen points3d_withlidar.bin instead txt file, it's ok for training?

yeah, I used it for training and got the correct result. You just need to change the file name in train.sh

data_root=$1
cuda_id=$2

mkdir -p output/

CUDA_VISIBLE_DEVICES=$cuda_id sgn-train street-gaussians-ns \
    --experiment_name street-gaussians-ns \
    --output_dir output/ \
    --vis viewer+wandb \
    --viewer.quit_on_train_completion True \
    colmap-data-parser-config \
    --data $data_root \
    --colmap_path colmap/sparse/0 \
    --load_3D_points True \
    --max_2D_matches_per_3D_point 0 \
    --undistort True \
    --segments-path segs \
    --filter_camera_id 1 2 3 4 5\
    --init_points_filename points3D_withlidar.bin         # this file name

yes, same method and it works fine

I download preprocessed dataset from this repo which is 10448102132863604198_472_000_492_000,and cameras.txt in /colmap shows it has 5 cameras: 1 OPENCV 1920 1280 2044.3189893365634 2044.3189893365634 966.3197697401635 624.7706323955747 0.048519645404847 -0.31855151741237414 -0.0005279507169045765 -0.0010850539443264748 2 OPENCV 1920 1280 2064.096136816171 2064.096136816171 965.2989897820795 643.2419196405554 0.04536665392932437 -0.3481585691089555 0.0023209650920834515 0.000879426586508394 3 OPENCV 1920 1280 2047.353864208166 2047.353864208166 959.6234340600746 644.2507554214233 0.03923964629713441 -0.30937484864506715 0.0016984481880601527 -0.0009807100492894205 4 OPENCV 1920 886 2052.166978577554 2052.166978577554 964.3304222948891 251.3045142973026 0.04792858845797487 -0.3115144801129193 0.00044896409868372216 0.0006301442821205557 5 OPENCV 1920 886 2060.943963183803 2060.943963183803 961.8321926331334 243.845105660424 0.04466498449403254 -0.33887981610387924 0.0015137815608006677 4.743970552947572e-05 So I think maybe is there any other settings I should set?

Yeah, I used this dataset too. So we can eliminate error with dataset. Can you provide more log or more information about environment?

I found my error was caused by the following code snippet in sgn_datamanager.py: if cache_images_device == "gpu": for cache in undistorted_images: cache["image"] = cache["image"].to(self.device) if "mask" in cache: cache["mask"] = cache["mask"].to(self.device) elif cache_images_device == "cpu": for cache in undistorted_images: cache["image"] = cache["image"].pin_memory() if "mask" in cache: cache["mask"] = cache["mask"].pin_memory() else: assert_never(cache_images_device) code above cause out of memory: Jul 30 10:58:44 SZH-C-0075K kernel: [90258.022147] Out of memory: Killed process 170640 (sgn-train) total-vm:43940012kB, anon-rss:18457996kB, file-rss:66412kB, shmem-rss:8151876kB, UID:1000 pgtables:55188kB oom_score_adj:0 Jul 30 10:58:44 SZH-C-0075K kernel: [90258.025371] Cannot map memory with base addr 0x7f1ed8000000 and size of 0x2000 pages I comment code above,training process can be done,but render command 'ns-render camera-path --load-config xxx/config.yml --camera-path-filename xxx/camera_paths/2024-07-29-13-15-51.json --output-path xxx.mp4' got the following error: street-gaussians-ns/street_gaussians_ns/sgn_splatfacto_scene_graph.py", line 323, in get_outputs assert camera.times is not None AssertionError Do these two things have some relationship?

1400013531 commented 3 months ago

Not sure about the relationship. I noticed the memory cost was unusual. Have you tried to modify the configuration in sgn_config.py or sgn_splatfacto.py? less iteration or less initial gaussians may help. About the render command, I used the command generated by nerfstudio, and did not meet the problem. Here is one of my camera path of dataset 10448102132863604198 which worked for me. 2024-07-26-09-38-26.json

jycc1991 commented 3 months ago

sgn_config.py

I user your json,it runs ok,but the mp4 contains nothing except black screen.Moreover,json I got from nerfstudio broswer is different from yours,it has no time info 2024-07-30-16-59-24.json What's the reason for this? Thanks

1400013531 commented 3 months ago

sgn_config.py

I user your json,it runs ok,but the mp4 contains nothing except black screen.Moreover,json I got from nerfstudio broswer is different from yours,it has no time info 2024-07-30-16-59-24.json What's the reason for this? Thanks

Oh, my camera path was generated from model trained by five cameras. If you use it on the model trained by cameras including no backward camera, you may get empty scenes.

For the timestamp in camera path, because generated by nerfstudio, I guess it is related with your nerfstudio version. gsplat to 0.1.11 and nerfstudio to 1.1.0 worked for me.

jycc1991 commented 3 months ago

sgn_config.py

I user your json,it runs ok,but the mp4 contains nothing except black screen.Moreover,json I got from nerfstudio broswer is different from yours,it has no time info 2024-07-30-16-59-24.json What's the reason for this? Thanks

Oh, my camera path was generated from model trained by five cameras. If you use it on the model trained by cameras including no backward camera, you may get empty scenes.

For the timestamp in camera path, because generated by nerfstudio, I guess it is related with your nerfstudio version. gsplat to 0.1.11 and nerfstudio to 1.1.0 worked for me.

Hi,do your render result looks good?I have the same dataset and camera 1-5 with you,but I got not so good render result,looks to https://github.com/LightwheelAI/street-gaussians-ns/issues/40

wangyida commented 3 months ago

generate with scripts/shells/points_cloud_generate.sh

it will gen points3d_withlidar.bin instead txt file, it's ok for training?

The TXT version could be triggered by colmap command line given a BIN file