search

titu1994 / tf_SIREN

Tensorflow 2.0 implementation of Sinusodial Representation networks (SIREN)
MIT License
149 stars 26 forks source link

duplicate multiplication of w0 ? #1

Closed schreon closed 4 years ago

schreon commented 4 years ago

First of all: Thanks for providing an implementation of SIREN for tensorflow so quickly after the release of the paper!

I did notice that there are 2 places where w0 multiplication is applied:

  1. directly inside the dense layer call:

https://github.com/titu1994/tf_SIREN/blob/master/tf_siren/siren.py#L124 (along with the other if-cases)

  1. after the call, within the activation layer: https://github.com/titu1994/tf_SIREN/blob/master/tf_siren/siren.py#L25

Is this intended?

titu1994 commented 4 years ago

It is intended. (1) applies w0 scaling to the kernel matrix before performing the dot product, thereby returning a preactivation matrix that is scaled to have zero mean and unit std. This is due to the SIREN initializer.

(2) is to perform long range mapping of the input to multiple cycles of the sine wave.

titu1994 commented 4 years ago

Turns out, the original paper doesnt use this form. I've therefore split apart the two implementations. The standard variant implements the paper version, and there now exists a "scaled" variant of the Dense layer as well as the MLP.