Visualization error in ENeRF-Outdoor dataset

[taehyung96]

Hi, thanks for the great work!

I tried to visualize ENeRF-Outdoor dataset by executing the code below.(Use pretrain model provided to you)

python run.py --type visualize --cfg_file configs/enerf/enerf_outdoor/actor1_path.yaml

However, the error below occurred, and when I looked for the solution, I guess it is because the pretrained model model was learned using multi-gpu and i loaded it on the single GPU.

Therefore, I would like to ask for an answer to the following question.

1. Please tell me how to visualize the ENeRF-Outdoor dataset using the pretrained model you provided on the single GPU.

   • The ZJU-MoCap Interactive Rendering, ZJU-MoCap / LLFF / NeRF / DTU dataasets Evaluation you provided works normally on my single GPU.

2. I would like to request the ENeRF-Outdoor / ST-NeRF dataset interactive rendering code that can be found on your project page. I think it has already been developed, so I sincerely ask you to share it.

-----------------------------------ERROR Detail--------------------------------------

EXP NAME: actor1

load model: /home/ubuntu/ENeRF/trainedmodel/enerf/actor1/latest.pth Traceback (most recent call last): File "run.py", line 106, in globals()['run' + args.type]() File "run.py", line 90, in run_visualize load_network(network, File "/home/ubuntu/ENeRF/lib/utils/net_utils.py", line 443, in load_network net.load_state_dict(pretrained_model['net'], strict=strict) File "/home/ubuntu/.conda/envs/enerf/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1406, in load_state_dict raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format( RuntimeError: Error(s) in loading state_dict for Network: Missing key(s) in state_dict: "feature_net_bg.conv0.0.conv.weight", "feature_net_bg.conv0.0.bn.weight", "feature_net_bg.conv0.0.bn.bias", "feature_net_bg.conv0.0.bn.running_mean", "feature_net_bg.conv0.0.bn.running_var", "feature_net_bg.conv0.1.conv.weight", "feature_net_bg.conv0.1.bn.weight", "feature_net_bg.conv0.1.bn.bias", "feature_net_bg.conv0.1.bn.running_mean", "feature_net_bg.conv0.1.bn.running_var", "feature_net_bg.conv1.0.conv.weight", "feature_net_bg.conv1.0.bn.weight", "feature_net_bg.conv1.0.bn.bias", "feature_net_bg.conv1.0.bn.running_mean", "feature_net_bg.conv1.0.bn.running_var", "feature_net_bg.conv1.1.conv.weight", "feature_net_bg.conv1.1.bn.weight", "feature_net_bg.conv1.1.bn.bias", "feature_net_bg.conv1.1.bn.running_mean", "feature_net_bg.conv1.1.bn.running_var", "feature_net_bg.conv2.0.conv.weight", "feature_net_bg.conv2.0.bn.weight", "feature_net_bg.conv2.0.bn.bias", "feature_net_bg.conv2.0.bn.running_mean", "feature_net_bg.conv2.0.bn.running_var", "feature_net_bg.conv2.1.conv.weight", "feature_net_bg.conv2.1.bn.weight", "feature_net_bg.conv2.1.bn.bias", "feature_net_bg.conv2.1.bn.running_mean", "feature_net_bg.conv2.1.bn.running_var", "feature_net_bg.toplayer.weight", "feature_net_bg.toplayer.bias", "feature_net_bg.lat1.weight", "feature_net_bg.lat1.bias", "feature_net_bg.lat0.weight", "feature_net_bg.lat0.bias", "feature_net_bg.smooth1.weight", "feature_net_bg.smooth1.bias", "feature_net_bg.smooth0.weight", "feature_net_bg.smooth0.bias", "cost_reg_0_layer0.conv0.conv.weight", "cost_reg_0_layer0.conv0.bn.weight", "cost_reg_0_layer0.conv0.bn.bias", "cost_reg_0_layer0.conv0.bn.running_mean", "cost_reg_0_layer0.conv0.bn.running_var", "cost_reg_0_layer0.conv1.conv.weight", "cost_reg_0_layer0.conv1.bn.weight", "cost_reg_0_layer0.conv1.bn.bias", "cost_reg_0_layer0.conv1.bn.running_mean", "cost_reg_0_layer0.conv1.bn.running_var", "cost_reg_0_layer0.conv2.conv.weight", "cost_reg_0_layer0.conv2.bn.weight", "cost_reg_0_layer0.conv2.bn.bias", "cost_reg_0_layer0.conv2.bn.running_mean", "cost_reg_0_layer0.conv2.bn.running_var", "cost_reg_0_layer0.conv3.conv.weight", "cost_reg_0_layer0.conv3.bn.weight", "cost_reg_0_layer0.conv3.bn.bias", "cost_reg_0_layer0.conv3.bn.running_mean", "cost_reg_0_layer0.conv3.bn.running_var", "cost_reg_0_layer0.conv4.conv.weight", "cost_reg_0_layer0.conv4.bn.weight", "cost_reg_0_layer0.conv4.bn.bias", "cost_reg_0_layer0.conv4.bn.running_mean", "cost_reg_0_layer0.conv4.bn.running_var", "cost_reg_0_layer0.conv9.0.weight", "cost_reg_0_layer0.conv9.1.weight", "cost_reg_0_layer0.conv9.1.bias", "cost_reg_0_layer0.conv9.1.running_mean", "cost_reg_0_layer0.conv9.1.running_var", "cost_reg_0_layer0.conv11.0.weight", "cost_reg_0_layer0.conv11.1.weight", "cost_reg_0_layer0.conv11.1.bias", "cost_reg_0_layer0.conv11.1.running_mean", "cost_reg_0_layer0.conv11.1.running_var", "cost_reg_0_layer0.depth_conv.0.weight", "cost_reg_0_layer0.feat_conv.0.weight", "nerf_0_layer0.agg.global_fc.0.weight", "nerf_0_layer0.agg.global_fc.0.bias", "nerf_0_layer0.agg.agg_w_fc.0.weight", "nerf_0_layer0.agg.agg_w_fc.0.bias", "nerf_0_layer0.agg.fc.0.weight", "nerf_0_layer0.agg.fc.0.bias", "nerf_0_layer0.lr0.0.weight", "nerf_0_layer0.lr0.0.bias", "nerf_0_layer0.sigma.0.weight", "nerf_0_layer0.sigma.0.bias", "nerf_0_layer0.color.0.weight", "nerf_0_layer0.color.0.bias", "nerf_0_layer0.color.2.weight", "nerf_0_layer0.color.2.bias", "cost_reg_0_bg.conv0.conv.weight", "cost_reg_0_bg.conv0.bn.weight", "cost_reg_0_bg.conv0.bn.bias", "cost_reg_0_bg.conv0.bn.running_mean", "cost_reg_0_bg.conv0.bn.running_var", "cost_reg_0_bg.conv1.conv.weight", "cost_reg_0_bg.conv1.bn.weight", "cost_reg_0_bg.conv1.bn.bias", "cost_reg_0_bg.conv1.bn.running_mean", "cost_reg_0_bg.conv1.bn.running_var", "cost_reg_0_bg.conv2.conv.weight", "cost_reg_0_bg.conv2.bn.weight", "cost_reg_0_bg.conv2.bn.bias", "cost_reg_0_bg.conv2.bn.running_mean", "cost_reg_0_bg.conv2.bn.running_var", "cost_reg_0_bg.conv3.conv.weight", "cost_reg_0_bg.conv3.bn.weight", "cost_reg_0_bg.conv3.bn.bias", "cost_reg_0_bg.conv3.bn.running_mean", "cost_reg_0_bg.conv3.bn.running_var", "cost_reg_0_bg.conv4.conv.weight", "cost_reg_0_bg.conv4.bn.weight", "cost_reg_0_bg.conv4.bn.bias", "cost_reg_0_bg.conv4.bn.running_mean", "cost_reg_0_bg.conv4.bn.running_var", "cost_reg_0_bg.conv9.0.weight", "cost_reg_0_bg.conv9.1.weight", "cost_reg_0_bg.conv9.1.bias", "cost_reg_0_bg.conv9.1.running_mean", "cost_reg_0_bg.conv9.1.running_var", "cost_reg_0_bg.conv11.0.weight", "cost_reg_0_bg.conv11.1.weight", "cost_reg_0_bg.conv11.1.bias", "cost_reg_0_bg.conv11.1.running_mean", "cost_reg_0_bg.conv11.1.running_var", "cost_reg_0_bg.depth_conv.0.weight", "cost_reg_0_bg.feat_conv.0.weight", "nerf_0_bg.agg.global_fc.0.weight", "nerf_0_bg.agg.global_fc.0.bias", "nerf_0_bg.agg.agg_w_fc.0.weight", "nerf_0_bg.agg.agg_w_fc.0.bias", "nerf_0_bg.agg.fc.0.weight", "nerf_0_bg.agg.fc.0.bias", "nerf_0_bg.lr0.0.weight", "nerf_0_bg.lr0.0.bias", "nerf_0_bg.sigma.0.weight", "nerf_0_bg.sigma.0.bias", "nerf_0_bg.color.0.weight", "nerf_0_bg.color.0.bias", "nerf_0_bg.color.2.weight", "nerf_0_bg.color.2.bias", "cost_reg_1_layer0.conv0.conv.weight", "cost_reg_1_layer0.conv0.bn.weight", "cost_reg_1_layer0.conv0.bn.bias", "cost_reg_1_layer0.conv0.bn.running_mean", "cost_reg_1_layer0.conv0.bn.running_var", "cost_reg_1_layer0.conv1.conv.weight", "cost_reg_1_layer0.conv1.bn.weight", "cost_reg_1_layer0.conv1.bn.bias", "cost_reg_1_layer0.conv1.bn.running_mean", "cost_reg_1_layer0.conv1.bn.running_var", "cost_reg_1_layer0.conv2.conv.weight", "cost_reg_1_layer0.conv2.bn.weight", "cost_reg_1_layer0.conv2.bn.bias", "cost_reg_1_layer0.conv2.bn.running_mean", "cost_reg_1_layer0.conv2.bn.running_var", "cost_reg_1_layer0.conv3.conv.weight", "cost_reg_1_layer0.conv3.bn.weight", "cost_reg_1_layer0.conv3.bn.bias", "cost_reg_1_layer0.conv3.bn.running_mean", "cost_reg_1_layer0.conv3.bn.running_var", "cost_reg_1_layer0.conv4.conv.weight", "cost_reg_1_layer0.conv4.bn.weight", "cost_reg_1_layer0.conv4.bn.bias", "cost_reg_1_layer0.conv4.bn.running_mean", "cost_reg_1_layer0.conv4.bn.running_var", "cost_reg_1_layer0.conv9.0.weight", "cost_reg_1_layer0.conv9.1.weight", "cost_reg_1_layer0.conv9.1.bias", "cost_reg_1_layer0.conv9.1.running_mean", "cost_reg_1_layer0.conv9.1.running_var", "cost_reg_1_layer0.conv11.0.weight", "cost_reg_1_layer0.conv11.1.weight", "cost_reg_1_layer0.conv11.1.bias", "cost_reg_1_layer0.conv11.1.running_mean", "cost_reg_1_layer0.conv11.1.running_var", "cost_reg_1_layer0.depth_conv.0.weight", "cost_reg_1_layer0.feat_conv.0.weight", "nerf_1_layer0.agg.global_fc.0.weight", "nerf_1_layer0.agg.global_fc.0.bias", "nerf_1_layer0.agg.agg_w_fc.0.weight", "nerf_1_layer0.agg.agg_w_fc.0.bias", "nerf_1_layer0.agg.fc.0.weight", "nerf_1_layer0.agg.fc.0.bias", "nerf_1_layer0.lr0.0.weight", "nerf_1_layer0.lr0.0.bias", "nerf_1_layer0.sigma.0.weight", "nerf_1_layer0.sigma.0.bias", "nerf_1_layer0.color.0.weight", "nerf_1_layer0.color.0.bias", "nerf_1_layer0.color.2.weight", "nerf_1_layer0.color.2.bias", "cost_reg_1_bg.conv0.conv.weight", "cost_reg_1_bg.conv0.bn.weight", "cost_reg_1_bg.conv0.bn.bias", "cost_reg_1_bg.conv0.bn.running_mean", "cost_reg_1_bg.conv0.bn.running_var", "cost_reg_1_bg.conv1.conv.weight", "cost_reg_1_bg.conv1.bn.weight", "cost_reg_1_bg.conv1.bn.bias", "cost_reg_1_bg.conv1.bn.running_mean", "cost_reg_1_bg.conv1.bn.running_var", "cost_reg_1_bg.conv2.conv.weight", "cost_reg_1_bg.conv2.bn.weight", "cost_reg_1_bg.conv2.bn.bias", "cost_reg_1_bg.conv2.bn.running_mean", "cost_reg_1_bg.conv2.bn.running_var", "cost_reg_1_bg.conv3.conv.weight", "cost_reg_1_bg.conv3.bn.weight", "cost_reg_1_bg.conv3.bn.bias", "cost_reg_1_bg.conv3.bn.running_mean", "cost_reg_1_bg.conv3.bn.running_var", "cost_reg_1_bg.conv4.conv.weight", "cost_reg_1_bg.conv4.bn.weight", "cost_reg_1_bg.conv4.bn.bias", "cost_reg_1_bg.conv4.bn.running_mean", "cost_reg_1_bg.conv4.bn.running_var", "cost_reg_1_bg.conv9.0.weight", "cost_reg_1_bg.conv9.1.weight", "cost_reg_1_bg.conv9.1.bias", "cost_reg_1_bg.conv9.1.running_mean", "cost_reg_1_bg.conv9.1.running_var", "cost_reg_1_bg.conv11.0.weight", "cost_reg_1_bg.conv11.1.weight", "cost_reg_1_bg.conv11.1.bias", "cost_reg_1_bg.conv11.1.running_mean", "cost_reg_1_bg.conv11.1.running_var", "cost_reg_1_bg.depth_conv.0.weight", "cost_reg_1_bg.feat_conv.0.weight", "nerf_1_bg.agg.global_fc.0.weight", "nerf_1_bg.agg.global_fc.0.bias", "nerf_1_bg.agg.agg_w_fc.0.weight", "nerf_1_bg.agg.agg_w_fc.0.bias", "nerf_1_bg.agg.fc.0.weight", "nerf_1_bg.agg.fc.0.bias", "nerf_1_bg.lr0.0.weight", "nerf_1_bg.lr0.0.bias", "nerf_1_bg.sigma.0.weight", "nerf_1_bg.sigma.0.bias", "nerf_1_bg.color.0.weight", "nerf_1_bg.color.0.bias", "nerf_1_bg.color.2.weight", "nerf_1_bg.color.2.bias". Unexpected key(s) in state_dict: "cost_reg_0.conv0.conv.weight", "cost_reg_0.conv0.bn.weight", "cost_reg_0.conv0.bn.bias", "cost_reg_0.conv0.bn.running_mean", "cost_reg_0.conv0.bn.running_var", "cost_reg_0.conv0.bn.num_batches_tracked", "cost_reg_0.conv1.conv.weight", "cost_reg_0.conv1.bn.weight", "cost_reg_0.conv1.bn.bias", "cost_reg_0.conv1.bn.running_mean", "cost_reg_0.conv1.bn.running_var", "cost_reg_0.conv1.bn.num_batches_tracked", "cost_reg_0.conv2.conv.weight", "cost_reg_0.conv2.bn.weight", "cost_reg_0.conv2.bn.bias", "cost_reg_0.conv2.bn.running_mean", "cost_reg_0.conv2.bn.running_var", "cost_reg_0.conv2.bn.num_batches_tracked", "cost_reg_0.conv3.conv.weight", "cost_reg_0.conv3.bn.weight", "cost_reg_0.conv3.bn.bias", "cost_reg_0.conv3.bn.running_mean", "cost_reg_0.conv3.bn.running_var", "cost_reg_0.conv3.bn.num_batches_tracked", "cost_reg_0.conv4.conv.weight", "cost_reg_0.conv4.bn.weight", "cost_reg_0.conv4.bn.bias", "cost_reg_0.conv4.bn.running_mean", "cost_reg_0.conv4.bn.running_var", "cost_reg_0.conv4.bn.num_batches_tracked", "cost_reg_0.conv9.0.weight", "cost_reg_0.conv9.1.weight", "cost_reg_0.conv9.1.bias", "cost_reg_0.conv9.1.running_mean", "cost_reg_0.conv9.1.running_var", "cost_reg_0.conv9.1.num_batches_tracked", "cost_reg_0.conv11.0.weight", "cost_reg_0.conv11.1.weight", "cost_reg_0.conv11.1.bias", "cost_reg_0.conv11.1.running_mean", "cost_reg_0.conv11.1.running_var", "cost_reg_0.conv11.1.num_batches_tracked", "cost_reg_0.depth_conv.0.weight", "cost_reg_0.feat_conv.0.weight", "nerf_0.agg.view_fc.0.weight", "nerf_0.agg.view_fc.0.bias", "nerf_0.agg.global_fc.0.weight", "nerf_0.agg.global_fc.0.bias", "nerf_0.agg.agg_w_fc.0.weight", "nerf_0.agg.agg_w_fc.0.bias", "nerf_0.agg.fc.0.weight", "nerf_0.agg.fc.0.bias", "nerf_0.lr0.0.weight", "nerf_0.lr0.0.bias", "nerf_0.sigma.0.weight", "nerf_0.sigma.0.bias", "nerf_0.color.0.weight", "nerf_0.color.0.bias", "nerf_0.color.2.weight", "nerf_0.color.2.bias", "cost_reg_1.conv0.conv.weight", "cost_reg_1.conv0.bn.weight", "cost_reg_1.conv0.bn.bias", "cost_reg_1.conv0.bn.running_mean", "cost_reg_1.conv0.bn.running_var", "cost_reg_1.conv0.bn.num_batches_tracked", "cost_reg_1.conv1.conv.weight", "cost_reg_1.conv1.bn.weight", "cost_reg_1.conv1.bn.bias", "cost_reg_1.conv1.bn.running_mean", "cost_reg_1.conv1.bn.running_var", "cost_reg_1.conv1.bn.num_batches_tracked", "cost_reg_1.conv2.conv.weight", "cost_reg_1.conv2.bn.weight", "cost_reg_1.conv2.bn.bias", "cost_reg_1.conv2.bn.running_mean", "cost_reg_1.conv2.bn.running_var", "cost_reg_1.conv2.bn.num_batches_tracked", "cost_reg_1.conv3.conv.weight", "cost_reg_1.conv3.bn.weight", "cost_reg_1.conv3.bn.bias", "cost_reg_1.conv3.bn.running_mean", "cost_reg_1.conv3.bn.running_var", "cost_reg_1.conv3.bn.num_batches_tracked", "cost_reg_1.conv4.conv.weight", "cost_reg_1.conv4.bn.weight", "cost_reg_1.conv4.bn.bias", "cost_reg_1.conv4.bn.running_mean", "cost_reg_1.conv4.bn.running_var", "cost_reg_1.conv4.bn.num_batches_tracked", "cost_reg_1.conv5.conv.weight", "cost_reg_1.conv5.bn.weight", "cost_reg_1.conv5.bn.bias", "cost_reg_1.conv5.bn.running_mean", "cost_reg_1.conv5.bn.running_var", "cost_reg_1.conv5.bn.num_batches_tracked", "cost_reg_1.conv6.conv.weight", "cost_reg_1.conv6.bn.weight", "cost_reg_1.conv6.bn.bias", "cost_reg_1.conv6.bn.running_mean", "cost_reg_1.conv6.bn.running_var", "cost_reg_1.conv6.bn.num_batches_tracked", "cost_reg_1.conv7.0.weight", "cost_reg_1.conv7.1.weight", "cost_reg_1.conv7.1.bias", "cost_reg_1.conv7.1.running_mean", "cost_reg_1.conv7.1.running_var", "cost_reg_1.conv7.1.num_batches_tracked", "cost_reg_1.conv9.0.weight", "cost_reg_1.conv9.1.weight", "cost_reg_1.conv9.1.bias", "cost_reg_1.conv9.1.running_mean", "cost_reg_1.conv9.1.running_var", "cost_reg_1.conv9.1.num_batches_tracked", "cost_reg_1.conv11.0.weight", "cost_reg_1.conv11.1.weight", "cost_reg_1.conv11.1.bias", "cost_reg_1.conv11.1.running_mean", "cost_reg_1.conv11.1.running_var", "cost_reg_1.conv11.1.num_batches_tracked", "cost_reg_1.depth_conv.0.weight", "cost_reg_1.feat_conv.0.weight", "nerf_1.agg.view_fc.0.weight", "nerf_1.agg.view_fc.0.bias", "nerf_1.agg.global_fc.0.weight", "nerf_1.agg.global_fc.0.bias", "nerf_1.agg.agg_w_fc.0.weight", "nerf_1.agg.agg_w_fc.0.bias", "nerf_1.agg.fc.0.weight", "nerf_1.agg.fc.0.bias", "nerf_1.lr0.0.weight", "nerf_1.lr0.0.bias", "nerf_1.sigma.0.weight", "nerf_1.sigma.0.bias", "nerf_1.color.0.weight", "nerf_1.color.0.bias", "nerf_1.color.2.weight", "nerf_1.color.2.bias".

[haotongl]

Hi, many thanks for your attention!

The pretrained model cannot works on ENeRF-Outdoor dataset since the compositional ENeRF is a little different from ENeRF trained on DTU. You can train compositional ENeRF from scratch in hours (From my experience, it converges after 20-30 epochs.).

The interactive rendering code for ST-NeRF is really messy because of rushing the conference ddl. I may release it when I am free to rewrite it.

[taehyung96]

Hello, again

Please share the interactive rendering code of ST-NeRF and Outdoor datasets.

I am currently developing to do interactive rendering with the ENeRF Outdoor dataset.

The results are printed as below, but the quality looks too low compared to your results.

I've tried to improve the quality by constantly debugging, but it's hard for me.

I developed it with the following process.

• Modify dataloader to match outdoor dataset by referring to the module(lib.datasets.zjumocap.enerf_interactive) of zjumocap dataset
• The network uses the same module(lib.networks.enerf.network_composite) as before

I'm in the middle of an HCI study on volumetric contents, and I want to see how static background and dynamic object's real time novel view synthesis can give a person a good memory of past experiences. Interactive rendering of common Scene such as ST-NeRF and Outdoor dataset is essential for this study.

In response to the above, you said that the code is messy in conference ddl, but I sincerely hope you will release the code. If it's hard to share through public, please share it through skdldlssk@gmail.com .

I admire your amazing work and wish you the best in the field of dynamic nerf.

Thanks

---

python run.py --cfg_file visualize...

python gui_human.py ...

[haotongl]

How about I send you an email with the interactive code and my processed ST-NeRF data? If that helps you, I can email you tomorrow during work time.

[taehyung96]

Thank you very much. If you could share the ST-NeRF dataset, it would be very helpful for my research. Please email to skdldlssk@gmail.com . Thank you.

[haotongl]

Hi, I have shared the code with you via email, please check it.