search

vsitzmann / siren

Official implementation of "Implicit Neural Representations with Periodic Activation Functions"
MIT License
1.72k stars 247 forks source link

Finding the proper frequency multiplier #52

Open FabricioArendTorres opened 2 years ago

FabricioArendTorres commented 2 years ago

Hi,

how do you usually initialize the frequency of the network (leaving aside the magic number 30)? As it is very much problem-dependent, I'm wondering if it is currently just a trial-and-error task or if there is a more reasonable approach, even if just for finding general scale (I saw you mentioned an value of 3000 for audio data in another issue).

Thanks already in advance, and also for this nice code-base.

wenqihuang commented 2 years ago

Hi, @vsitzmann

Thanks for the very interesting work! I also have the same question as @FabricioArendTorres. I tried to replace the ReLU in a regression network end up with nonsense outputs. It worked well with ReLU. I think the problem would relate to the selection of frequency multiplier. It would be great help if you can provide some suggestions on finding the number. Many thanks!

Best, Wenqi

FabricioArendTorres commented 2 years ago

@wenqihuang After working with it some time (for solving PDEs), I can maybe share a bit my experience.

If you get nonsense, the network usually grossly overfits, and it helps to strongly decrease the parameter. I'd suggest starting from a lower value, and then slowly increasing it. With a validation set it will be really obvious where the range of usable values lies.

Just be aware that changing the number of units and layers might also require a change in the parameter. I found that after increasing the number of units in each layer, you need to decrease the frequency parameter.

Did not find any golden solution other than basically trying it or using some hyperpameter search. I guess that's also a limitation of this work, compared to Random Fourier Feature layers where the tunable parameters a bit more understandable in their effect.

wenqihuang commented 2 years ago

@FabricioArendTorres Thanks for the experience!