search

google / nerfactor

Neural Factorization of Shape and Reflectance Under an Unknown Illumination
https://xiuming.info/projects/nerfactor/
Apache License 2.0
440 stars 56 forks source link

run geometry_from_nerf.py issues #9

Closed XiaoKangW closed 2 years ago

XiaoKangW commented 3 years ago

Hi,xiuming

Great job! i want to run this work, but i meet some problems when i run geometry_from_nerf.py , could you help me? and i didn't find the trained models in your data from project website, so do you publish the trained model for testing later?

Thanks! looking forward to your reply.

WARNING:tensorflow:10 out of the last 10 calls to <function pfor..f at 0x7f6d021b9bf8> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. W0911 12:21:18.196791 140106441705280 def_function.py:126] 10 out of the last 10 calls to <function pfor..f at 0x7f6d021b9bf8> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.

xiumingzhang commented 3 years ago

Hi, did you use the conda environment specified in environment.yml? I suspect this is a TF version issue.

Regarding the trained model, yes, I will release them soon.

XiaoKangW commented 3 years ago

Thanks for your reply! i have addressed the problem after changing TF version.

and when I run the step 'geometry_from_nerf' I need to spend very long time to run, maybe over 2 days. Is this normal?

thanks again!

cjw531 commented 3 years ago

Hi, I have similar question as what @XiaoKangW asked. So I think geometry_from_nerf takes really long as mentioned in your paper:

"Prior to the final joint optimization, computing the initial surface normals and light visibility from the trained NeRF takes 30 minutes per view on a single GPU for a 16 × 32 light probe (i.e., 512 light source locations)."

I currently have three 2080ti available with me. It has way much less GPU RAM size (11G) compared to your Titan RTX (24G). I was about to parallelize the process and use multi-GPU to speed up, but it says in 3rd step of prep that I only have to use a single GPU:

"For portability, this step runs sequentially, processing one view after another. If your infrastructure supports distributing jobs easily over multiple GPUs, you should consider having one GPU process one view to parallelize all views."

And in your paper, you mentioned that this geometry calculation step can be:

"trivially parallelized because each view is processed independently."

I am quite confused how to parallelize it. Do you have any options in your bash script to do so or do I have to modify your script? Thank you in advance for your help.

FYI: It took 6 days for me to finish train/validation/test steps to calculate the surface normals and light visibility, with a single 2080ti.

xilongzhou commented 2 years ago

Hi, xiuming,

I came across the similar issue , I try to generate geometry in parallel but current settings only allow me to generate geometry sequentially. Could you please give me some suggestion how to parallelize these?

Thank you very much!

xiumingzhang commented 2 years ago

Sorry for the delayed response, @cjw531 and @xilongzhou.

In these lines: https://github.com/google/nerfactor/blob/main/nerfactor/geometry_from_nerf.py#L85-L87 we are calling process_view() in a for-loop, sequentially.

To parallelize this step, we had one GPU running process_view() per view. Because how to parallelize all views depends highly on what your infrastructure is, we released this sequential version for portability.

LMK if you have further questions.