Siren based nerf

[bell-one]

Thanks for your wonderful implementation first.

I saw your discussion about siren based nerf in here https://github.com/bmild/nerf/issues/60 this topic was discussed long time ago but, I tried little experiments with your base implementation.

I also tried siren layer based NeRF and test results shows better performance then reference NeRF with clear edge

there is little hyper param changes for faster experiments but, for horn scene reference nerf with PE shows PSNR 29.29 for valid siren based nerf w/o PE shows PSNR 30.32 for valid

and also like leaves reference nerf with PE shows PSNR 20.99 for valid siren based nerf w/o PE shows PSNR 21.70 for valid

or flower scene reference nerf with PE shows PSNR 27.31 for valid siren based nerf w/o PE shows PSNR 29.07 for valid

I'd like to have little discussion of your implementation version for siren, can you share your results or thinks within your experiments?

[kwea123]

Hi, thanks for your fruitful experiments. Unfortunately I forgot the exact settings for my siren network, I think I used the same network as here. What is your network structure?

By the way, I plan to do some experiments about different coordinate mlps (positional encoding, siren, fourier features, and the recent gaussian activation) next week, and I will share my findings in this repo

[bell-one]

I use same network structure as you linked,

In my experiments, Missing of weights initialization for first layer make simmilar blurred error as you metioned ago.

I just change all of linear+sigmoid to linear+sine without last layer for sigma and rgb. I think I can make PR for this experiment with --siren opt later.

Also, Suggest some coordinate mlps which fits with 3D coordinate system in here maybe this approach in 3D coordinate sytem could help NeRF performance

I will look foward to your coordinate mlps repo! Thanks for your reply.

[kwea123]

It will be great if you can share your model definition with a PR! Thanks for the paper suggestion, I will try to add that to my implementation too!

[ChuhanWang10]

I use same network structure as you linked,

In my experiments, Missing of weights initialization for first layer make simmilar blurred error as you metioned ago.

I just change all of linear+sigmoid to linear+sine without last layer for sigma and rgb. I think I can make PR for this experiment with --siren opt later.

Also, Suggest some coordinate mlps which fits with 3D coordinate system in here maybe this approach in 3D coordinate sytem could help NeRF performance

I will look foward to your coordinate mlps repo! Thanks for your reply.

Hi, I also did the experiments and I got a few problems. I trained my own scene on both the PE-NeRF and SIREN-NeRF. It turns out that using SIREN layers instead of ReLU-MLP leads to a way less PSNR score (around 5dB less). My network structure is a 6-layer SIREN plus a linear+softplus in the output. I tried different w in the first layer, from 30 to 1 (I guess w is a very important hyper-parameter and it should be carefully tuned based on datasets.). For the rest of the layers, the w was kept 1.

May I ask a few questions about your experiments？

1. Can you share your choices of w in these experiments? Did they remain the same on different LLFF datasets?
2. I have tried a combination of SIREN+PE, it seems better than SIREN w/o PE. I was inspired by the paper talking about Gaussian activation, the paper says a combination of PE and Gaussian leads to better results. Do you have any experiments about this?

Thanks a lot!

[bell-one]

I use same network structure as you linked, In my experiments, Missing of weights initialization for first layer make simmilar blurred error as you metioned ago. I just change all of linear+sigmoid to linear+sine without last layer for sigma and rgb. I think I can make PR for this experiment with --siren opt later. Also, Suggest some coordinate mlps which fits with 3D coordinate system in here maybe this approach in 3D coordinate sytem could help NeRF performance I will look foward to your coordinate mlps repo! Thanks for your reply.

Hi, I also did the experiments and I got a few problems. I trained my own scene on both the PE-NeRF and SIREN-NeRF. It turns out that using SIREN layers instead of ReLU-MLP leads to a way less PSNR score (around 5dB less). My network structure is a 6-layer SIREN plus a linear+softplus in the output. I tried different w in the first layer, from 30 to 1 (I guess w is a very important hyper-parameter and it should be carefully tuned based on datasets.). For the rest of the layers, the w was kept 1.

May I ask a few questions about your experiments？

1. Can you share your choices of w in these experiments? Did they remain the same on different LLFF datasets?
2. I have tried a combination of SIREN+PE, it seems better than SIREN w/o PE. I was inspired by the paper talking about Gaussian activation, the paper says a combination of PE and Gaussian leads to better results. Do you have any experiments about this?

Thanks a lot!

Hello Chuhan Wang10

Sorry for the late reply.

1. I also conducted an experiment with the same w as you said. In SIREN's original implementation and supplement, it was said that even if w was 30 in all layers, it went well, but in nerf it was not. So I set w0=30, and the other w=1. Also, for w0=30, the input range in SIREN paper and the range of NeRF are different, so currently I try to understand the method.

ps. Did you initialize your weight params with 1/n and sqrt(c/n) correctly? In my experience, It makes much differences too.

2. Before I saw the above results, I also tried the SIREN+PE experiment. I think the performance was better at that time, but I didn't do many experiments because it seemed right to experiment after understanding both the correct understanding of SIREN and the method to apply it to NeRF.

Thanks for your interest