search

NVlabs / nvdiffrec

Official code for the CVPR 2022 (oral) paper "Extracting Triangular 3D Models, Materials, and Lighting From Images".
Other
2.09k stars 222 forks source link

Help！Why is the resulting mesh of such poor quality #139

Open 520jz opened 12 months ago

520jz commented 12 months ago

I use my own dataset, the dataset is in llff format, and below is my dataset format image

Below is an RGB image and mask display of my dataset image image

This is my config file image

This is a bad result image

Is there a problem with my dataset or with my config file settings?

jmunkberg commented 12 months ago

Hello, Please see the discussion here: https://github.com/NVlabs/nvdiffrec/issues/84

520jz commented 12 months ago

Hello, Please see the discussion here: #84

  • Use a sufficient number of views. I would recommend 50+ views, nicely distributed around the object.
  • Verify the accuracy of the camera poses. I assume you used colmap? We sometimes check the poses by creating a NeRF using https://github.com/NVlabs/instant-ngp as a sanity check. We assume perfect poses, and do not correct for inaccuracies, so if the poses are bad, nvdiffrec will fail.
  • Accuracy of masks. Even a few pixel differences, as shown above, may make the optimization harder. If you have the possibility to create masks with higher accuracy, that will help.
  • Is the lighting constant in all images, and is the object fully static across all views?

Hi, thank you for reply! I appreciate your help. I'm going to take a look at #84 1、My number of views is 66. I feel like my number of views should be adequate. 2、This dataset was downloaded from the Internet, and the dataset itself has camera poses. But the camera pose should also be colmap generated. I thought I should Verify the accuracy of the camera poses. 3、Masks may be less accurate. Original image with white background, I used the following code to generate the mask. cv2.threshold(gray, 250, 255, cv2. THRESH_BINARY_INV) 4、yes

Some other problems： Dear author, forgive me for my ignorance. 1、Does the resolution of the input RGB image have any effect on the generated mesh? My original image is relatively large in resolution, so I lowered the resolution of the picture. 2、Do the input RGB images need an alpha channel? That is, must the background of the image be transparent?

jmunkberg commented 12 months ago

Resolution: Higher is better for reconstruction quality (better texture detail), but requires more memory. For the paper, we usually trained in 800x800 or 1024x1024 resolution with a batch size of 4-8 using GPUs with 32-48GB of memory

Alpha: If you are using the llff reader, the color images are expected to be three-channel RGB without any alpha channel or transparency. See the code here: https://github.com/NVlabs/nvdiffrec/blob/main/dataset/dataset_llff.py#L83 You can easily modify that to your needs.