Open ZirongChan opened 11 months ago
this is what I get after training for 100K iterations with the same parameters I mentioned.
I think that the number of the iterations does not help much, I wonder what goes wrong in my setting.
Hi @ZirongChan
Yes, your result indeed is missing the details. Can you provide the following?
Hi @ZirongChan
Yes, your result indeed is missing the details. Can you provide the following?
* The commit you are on * A screenshot of the training metrics (both losses and variables monitored)
@mli0603 thx for your reply. 1, I'm pretty sure that the commit is "f512b278" 2, I did not manage to get the code running with wandb since some unknown issue, so I don't have the training metrics monitoring results. Is there something else that I can provide that is useful for analyzing the problem?
Could you look into wandb logging? It will be beneficial to have them for debugging purposes. I wonder if something is wrong with the setup and some metrics have gone wild.
i try meetingroom too, the result look not good. after 500k iterations:
i am using RTX 2080 Ti with 8G memory here is the parameters: Training with 1 GPUs. Using random seed 0 Make folder logs/meetingroom
and the config file : 1、projects/neuralangelo/configs/tnt.yaml
parent: projects/neuralangelo/configs/base.yaml
model: object: sdf: mlp: inside_out: True # True for Meetingroom. encoding: coarse2fine: init_active_level: 8 appear_embed: enabled: True dim: 8
data: type: projects.neuralangelo.data root: datasets/tanks_and_temples/Meetingroom num_images: 371 # The number of training images. train: image_size: [835,1500] batch_size: 1 subset: val: image_size: [300,540] batch_size: 1 subset: 1 max_viz_samples: 16
2、projects/neuralangelo/configs/base.yaml logging_iter: 9999999999999 # disable the printing logger
max_iter: 500000
wandb_scalar_iter: 100 wandb_image_iter: 10000 validation_iter: 5000 speed_benchmark: False
checkpoint: save_iter: 20000
trainer: type: projects.neuralangelo.trainer ema_config: enabled: False load_ema_checkpoint: False loss_weight: render: 1.0 eikonal: 0.1 curvature: 5e-4 init: type: none amp_config: enabled: False depth_vis_scale: 0.5
model: type: projects.neuralangelo.model object: sdf: mlp: num_layers: 1 hidden_dim: 256 skip: [] activ: softplus activ_params: beta: 100 geometric_init: True weight_norm: True out_bias: 0.5 inside_out: False encoding: type: hashgrid levels: 16 hashgrid: min_logres: 5 max_logres: 11 dict_size: 20 dim: 8 range: [-2,2] coarse2fine: enabled: True init_active_level: 4 step: 5000 gradient: mode: numerical taps: 4 rgb: mlp: num_layers: 4 hiddendim: 256 skip: [] activ: relu activ_params: {} weight_norm: True mode: idr encoding_view: type: spherical levels: 3 s_var: init_val: 3. anneal_end: 0.1 background: enabled: True white: False mlp: num_layers: 8 hidden_dim: 256 skip: [4] num_layers_rgb: 2 hidden_dim_rgb: 128 skip_rgb: [] activ: relu activ_params: {} activ_density: softplus activ_density_params: {} view_dep: True encoding: type: fourier levels: 10 encoding_view: type: spherical levels: 3 render: rand_rays: 512 num_samples: coarse: 64 fine: 16 background: 32 num_sample_hierarchy: 4 stratified: True appear_embed: enabled: False dim: 8
optim: type: AdamW params: lr: 1e-3 weight_decay: 1e-2 sched: iteration_mode: True type: two_steps_with_warmup warm_up_end: 5000 two_steps: [300000,400000] gamma: 10.0
data: type: projects.nerf.datasets.nerf_blender root: datasets/nerf-synthetic/lego use_multi_epoch_loader: True num_workers: 4 preload: True num_images: # The number of training images. train: image_size: [800,800] batch_size: 2 subset: val: image_size: [400,400] batch_size: 2 subset: 4 max_viz_samples: 16 readjust: center: [0.,0.,0.] scale: 1.
by the way ,I found only scene_type=object which video shoot for object can get good result.
Hi @qq297110281
We scale the compute up by using more GPUs. We find it is important to scale the batch size up for better results (such as 16 for TnT). This is detailed in the appenidx.
@mli0603 you mean like this ?
Training with 1 GPUs. Using random seed 0 Make folder logs/meetingroom
in the paper, for tnt dataset, the marching cubes resolution is 2048, is that mean extract mesh using --resolution=2048 ?
i try the config above, using 1 A100 80G GPU, but got strange result after 100k iterations:
extract mesh using: torchrun --nproc_per_node=1 projects/neuralangelo/scripts/extract_mesh.py --config=logs/meetingroom/config.yaml --checkpoint=logs/meetingroom/epoch_33332_iteration_000100000_checkpoint.pt --output_file=/home/nlt/meetingroom_2k_tx.ply --resolution=2048 --block_res=128 --textured
Hello, I've trained the meetingroom from TNT, using batch_size = 16, for 500000 iterations, with the bounding sphere scale = 1. the other parameters are set as default settings given in the base.yaml and the colmap outputs, the hashgrid dict_size is set to 22, and I also use the apper_embed option since I found out that you use it in the supplementary material.
The result looks good, but still, it is not as good as the one you demonstrated, mine lacks of details and sharpness. what I've got:
what you show:
![image](https://github.com/NVlabs/neuralangelo/assets/62975382/7f3a7bb8-ed66-48b7-8823-d59c8b1808a9)
I wonder what parameters else can I adjust to get a better result? should I train it for more iterations such as 1000K or set the bounding sphere more thigh to the region which contains the valid sparse points from colmap?