guibo-tech / BIMAP-NeRF-Endoscopy-3DReconstruction

Neural Radiance Fields for 3D Reconstruction in Monoscopic Laryngeal Endoscopy. Implemented for ISBI 2024.
Apache License 2.0
0 stars 0 forks source link

Train and reconstruct data EndoNeRF - their datset #6

Open guibo-tech opened 1 year ago

guibo-tech commented 1 year ago

Training on GPU Nidia RTX 3060

GPU out of memory

Image

Solution

Image

Image

Render rays in smaller minibatches to avoid OOM

Image

guibo-tech commented 1 year ago

Parameters in the config file N_rand = 2048 N_samples = 32 N_importance = 32

In the context of rendering rays in computer graphics, N_rand, N_samples, and N_importance typically refer to the number of random samples and importance samples used for rendering and evaluating the final pixel color. Here's a brief explanation of each parameter:

N_rand: This represents the number of random samples per pixel. Random samples are often used for techniques like antialiasing or stochastic effects to reduce pixelation and create smoother images. Increasing N_rand can improve the overall image quality but may also increase rendering time.

N_samples: This refers to the number of primary samples per pixel. Primary samples are the initial rays cast from the camera into the scene. N_samples determines the number of rays per pixel and is used to calculate the color for each pixel. Higher values of N_samples can result in better sampling of the scene and reduced noise, leading to higher-quality images but with increased rendering time.

N_importance: This represents the number of importance samples per pixel. Importance sampling is a technique used to sample rays based on their contribution to the final pixel color. By biasing the samples towards more important areas of the scene, importance sampling can improve the efficiency of the rendering process. Increasing N_importance can lead to better convergence and reduced noise in the final image.

These parameters are typically adjusted based on the desired trade-off between image quality and rendering time. Higher values generally result in better quality but require more computational resources. It's common to experiment with different values to find the optimal settings for a specific scene or application.

guibo-tech commented 1 year ago

Train terminal results

_export CUDA_VISIBLEDEVICES=0 # Specify GPU id _python3 run_endonerf.py --config configs/cutting_tissuesscene.txt

Image

Image

guibo-tech commented 1 year ago

Reconstruction

The error message you encountered suggests that the version of NumPy installed in your environment is older than the required version for Matplotlib. Matplotlib requires NumPy version 1.20 or higher, but you have version 1.19.0 installed.

python3 endo_pc_reconstruction.py --config_file configs/cutting_tissues_scene.txt --n_frames 120 --depth_smoother --depth_smoother_d 28

Image

guibo-tech commented 1 year ago

cutting_training_1

Display reconstructed point clouds

_python3 vis_pc.py --pc_dir logs/cutting_training_1/reconstructed_pcds100000

I think the depth is too much in the dark area, is this because of the modified/reduced parameters to train it faster? It might get better results using the workstation.

Front Image

Side

Image

guibo-tech commented 1 year ago

cutting_training_2

training time: 20min reconstruction time: 7min

export CUDA_VISIBLE_DEVICES=0 # Specify GPU id python3 run_endonerf.py --config configs/cutting_tissues_scene.txt python3 endo_pc_reconstruction.py --config_file configs/cutting_tissues_scene.txt --n_frames 120 --depth_smoother --depth_smoother_d 28 python3 vis_pc.py --pc_dir logs/cutting_training_2/reconstructed_pcds_1000

Image

Image

Image

Image

guibo-tech commented 1 year ago

oral1v2_training_3 Only 34 images instead of 156 Without tool occlution

python3 run_endonerf.py --config configs/oral1v2.txt --no_mask_raycast

python3 endo_pc_reconstruction.py --config_file configs/oral1v2.txt --n_frames 34 --depth_smoother --depth_smoother_d 28

python3 vis_pc.py --pc_dir logs/oral1v2_training_3/reconstructed_pcds_1000

Image

Image