Help for running ZeroNVS on custom images

[sirkrisp]

Thanks for your contribution! For evaluation purposes, it would help a lot if you could add a small example on how to run launch_inference.sh on new images. I tried to run zeronvs on the following image (resized to 256x256 PNG image):

But the output is completely wrong and the program eventually crashes because no mesh could be reconstructed.

https://github.com/kylesargent/ZeroNVS/assets/43377824/c7849a75-00a2-47bb-9d5d-13594ac4d60e

I launched launch.py in the following way (NOTE I replaced --train with --export):

python launch.py --config configs/zero123_scene.yaml --export --gpu 0 \
    system.guidance.cond_image_path=/home/user/Documents/projects/zeronvs/test_data/2008_001546_256x256.png \
    data.image_path=/home/user/Documents/projects/zeronvs/test_data/2008_001546_256x256.png \
    system.guidance.pretrained_model_name_or_path=zeronvs.ckpt \
    system.guidance.pretrained_config=zeronvs_config.yaml \
    data.view_synthesis=null \
    data.default_elevation_deg=0 \
    data.default_fovy_deg=52.55 \
    data.random_camera.fovy_range="[52.55,52.55]" \
    data.random_camera.eval_fovy_deg=52.55 \
    system.loss.lambda_opaque=0.0 \
    system.background.color='[0.5,0.5,0.5]' \
    system.background.random_aug=true \
    system.background.random_aug_prob=1.0 \
    system.guidance.guidance_scale=9.5 \
    system.renderer.near_plane=0.5 \
    system.renderer.far_plane=1000.0 \
    system.guidance.precomputed_scale=0.7 \
    system.guidance.use_anisotropic_schedule=true \
    system.guidance.anisotropic_offset=1000

Let me know if you need further information. Thanks in advance!

[zhou-pig]

Thanks for your help, but when I replace "train" with "export", I get the result shown in Figure. Have you encountered it, and how can you solve it?

[kylesargent]

Mesh export was a feature in the original threestudio library on which this repo is based, but I never tried to get it to work for ZeroNVS (and I probably wouldn't expect it to work that well anyway to be honest).

With respect to this specific bug, I'm not sure what the issue is, but I'd recommend trying to resume from a pretrained checkpoint first. You'll definitely need to fit the NeRF before exporting the mesh. The visualization above looks like it was generated at train step 0. Thanks!