NVlabs / neuralangelo

Official implementation of "Neuralangelo: High-Fidelity Neural Surface Reconstruction" (CVPR 2023)
https://research.nvidia.com/labs/dir/neuralangelo/
Other
4.27k stars 383 forks source link

Bad result on custom data #158

Open zhj1013 opened 7 months ago

zhj1013 commented 7 months ago

I attempted 3D reconstruction using custom data but failed. My dataset was extracted from a video of an object on a turntable shot with a smartphone, using colmap. The results from visualize_colmap.ipynb and visualize_transforms.ipynb appear to be normal. The wandb val/vis/opacity seems off. Can someone help me figure out what the issue might be?

image

image

image

joshpwrk commented 7 months ago

Getting similar issues where the reconstructed object is VERY smoothed.

Here are the params I ran with:

  1. COLMAP:
    • 1 min 30fps video of an object
    • DOWNSAMPLE_RATE = 2
    • SCENE_TYPE = object
    • resultant images -> 1079
    • bounding box verified in blender

Below is are several sample images Screenshot from 2023-12-02 13-23-07

  1. Run neuralangelo with default settings
    CONFIG=<used exact output of the COLMAP step>
    EXPERIMENT=my_run
    GROUP=my_group
    NAME=my_name
    GPUS=1 
    torchrun --nproc_per_node=${GPUS} train.py \
    --logdir=logs/${GROUP}/${NAME} \
    --config=CONFIG \
    --show_pbar

Here are my WandB results Screenshot from 2023-12-02 13-21-20

  1. Ran isosurface extraction with exact params but mesh looks like this Screenshot from 2023-12-02 13-22-28

Some guesses

A. In the research paper I saw that bumping hash encoding level from 4 to 16 significantly improves details. Is this what could be causing this overly smooth output?

If so, at what step (COLMAP, Nueralangelo, Isosurface extraction) should this level be set and where in the config?

Is it the model.object.sdf.encoding.coarse2fine.init_active_level param?

B. Could it be related to the training never reaching the sufficient activation level? The default config set max_iter at 500k, so I wonder if this has limited the models ability to extract more exact surfaces?