Train and reconstruct EndoNeRF-medair - our dataset

guibo-tech / BIMAP-NeRF-Endoscopy-3DReconstruction

Neural Radiance Fields for 3D Reconstruction in Monoscopic Laryngeal Endoscopy. Implemented for ISBI 2024.

Apache License 2.0

0 stars 0 forks source link

Train and reconstruct EndoNeRF-medair - our dataset #11

Open guibo-tech opened 1 year ago

guibo-tech commented 1 year ago

oral1v2_training_1

Training: 20min export CUDA_VISIBLE_DEVICES=0 # Specify GPU id python3 run_endonerf.py --config configs/oral1v2.txt

Reconstruction: 20min python3 endo_pc_reconstruction.py --config_file configs/oral1v2.txt --n_frames 120 --depth_smoother --depth_smoother_d 28

Visualization python3 vis_pc.py --pc_dir logs/oral1v2_training_1/reconstructed_pcds_1000

guibo-tech commented 1 year ago

expname = oral1v2_training_5 datadir = ../oral1v5_endonerf/

Training capacity: 5% based on original version, training parameters reduced to train it faster --N_iter = 1000 depth_refine_period=4000 i_print = 2 i_testset = 500 i_weights = 40 i_video = 1000 video_fps = 2

Training: 10min export CUDA_VISIBLE_DEVICES=0 python3 run_endonerf.py --config configs/oral1.txt --no_mask_raycast

Reconstruction: 5min python3 endo_pc_reconstruction.py --config_file configs/oral1.txt --n_frames 34 --depth_smoother --depth_smoother_d 28

Visualization python3 vis_pc.py --pc_dir logs/oral1v2_training_5/reconstructed_pcds_1000

guibo-tech commented 1 year ago

expname = oral1v2_training_6 datadir = ../oral1v5_endonerf/

Training capacity: 10% based on original version, training parameters reduced to train it faster --N_iter = 10000 depth_refine_period=4000 i_print = 20 i_testset = 5000 i_weights = 400 i_video = 10000 video_fps = 2

Training: 8min export CUDA_VISIBLE_DEVICES=0 python3 run_endonerf.py --config configs/oral1.txt --no_mask_raycast

ERROR

solution: --no_depth_refine

guibo-tech commented 1 year ago

expname = oral1v2_training_6 datadir = ../oral1v5_endonerf/

Training capacity: 10% based on original version, training parameters reduced to train it faster --N_iter = 1000 depth_refine_period=40 i_print = 2 i_testset = 500 i_weights = 40 i_video = 1000 video_fps = 2

Training: 8min export CUDA_VISIBLE_DEVICES=0 python3 run_endonerf.py --config configs/oral1.txt --no_mask_raycast --no_depth_refine

Reconstruction: 4min python3 endo_pc_reconstruction.py --config_file configs/oral1.txt --n_frames 34 --depth_smoother --depth_smoother_d 28

Visualization python3 vis_pc.py --pc_dir logs/oral1v2_training_6/reconstructed_pcds_1000

guibo-tech commented 1 year ago

expname = oral1v2_training_7 datadir = ../oral1v6_endonerf/

Training capacity: 10% based on original version, training parameters reduced to train it faster

--N_iter = 10000 depth_refine_period=40000 i_print = 20 i_testset = 5000 i_weights = 400 i_video = 10000 video_fps = 2

Training: 15min export CUDA_VISIBLE_DEVICES=0 python3 run_endonerf.py --config configs/oral1.txt --no_mask_raycast

Reconstruction: 5min python3 endo_pc_reconstruction.py --config_file configs/oral1.txt --n_frames 34 --depth_smoother --depth_smoother_d 28

Visualization python3 vis_pc.py --pc_dir logs/oral1v2_training_7/reconstructed_pcds_10000

guibo-tech commented 1 year ago

expname = oral1v2_training_9 datadir = ../oral1v6_endonerf/

-N_iter = 100000 depth_refine_period=400000 i_print = 200 i_testset = 50000 i_weights = 4000 i_video = 100000 video_fps = 2

Training: 1h30 export CUDA_VISIBLE_DEVICES=0 python3 run_endonerf.py --config configs/oral1.txt --no_mask_raycast

Reconstruction: min python3 endo_pc_reconstruction.py --config_file configs/oral1.txt --n_frames 34 --depth_smoother --depth_smoother_d 28

Visualization python3 vis_pc.py --pc_dir logs/oral1v2_training_9/reconstructed_pcds_100000

Bad result, it generates something good for the firts frame, but all the other frames are completely black.

The 3D reconstruction shows a quite small thing in the middle.

guibo-tech commented 1 year ago

expname = oral1v2_training_10 datadir = ../oral1v6_endonerf/

Training capacity: 10% based on original version, training parameters reduced to train it faster

-N_iter = 100000 depth_refine_period=400000 i_print = 200 i_testset = 50000 i_weights = 4000 i_video = 100000 video_fps = 2

Training: 1h15 export CUDA_VISIBLE_DEVICES=0 python3 run_endonerf.py --config configs/oral1.txt --no_mask_raycast --no_depth_refine

Reconstruction: 4min python3 endo_pc_reconstruction.py --config_file configs/oral1.txt --n_frames 34 --depth_smoother --depth_smoother_d 28

Visualization python3 vis_pc.py --pc_dir logs/oral1v2_training_10/reconstructed_pcds_100000

guibo-tech commented 1 year ago

expname = oral1v2_training_11 datadir = ../oral1v6_endonerf/

Training capacity: 30% based on original version, training parameters reduced to train it faster

Training: 3h30 export CUDA_VISIBLE_DEVICES=0 python3 run_endonerf.py --config configs/oral1.txt --no_mask_raycast --no_depth_refine

Reconstruction: 6min python3 endo_pc_reconstruction.py --config_file configs/oral1.txt --n_frames 34 --depth_smoother --depth_smoother_d 28

Visualization python3 vis_pc.py --pc_dir logs/oral1v2_training_11/reconstructed_pcds_100000 --vis_stall 40 --save_dir logs/oral1v2_training_11/ --rec_video_fps 2

guibo-tech commented 1 year ago

Improvements

shallow the final depth reconstruction in z-direction or re-scale the depth images. It seems that the depth is too stretched/extended in the z-direction.
Develop technique to make the view static and compensate the motion
Do not delete blurred frames, because it loses the the video transition effect from frame-to-frame. Try to use a filter/technique to make these frames sharper.
Try to get out the label (from the recordings), because it is being reconstructed. Develop technique or use tool_occlusion of the EndoNeRF?
Update training parameters to a full capacity, use a lab computer for training it