search

bioinf-jku / SNNs

Tutorials and implementations for "Self-normalizing networks"
GNU General Public License v3.0
1.58k stars 199 forks source link

cnn_graph #8

Closed maosi-chen closed 6 years ago

maosi-chen commented 6 years ago

Thank you for sharing the code. I have successfully applied them (SELU and alpha_dropout) building a pure fully connected network (7 layers) for a regression problem (the R2 between the predicted and the observed variable is greater than 0.99!).

Right now I'm trying to replace the RELU in cnn_graph with SELU. Unlike the standard cnn, cnn_graph performs the convolution on the graph Fourier transformed inputs (a recursive process involves multiple matrix multiplications between the layer-specific graph Laplacian and the layer inputs). The original normalized input is shifted to some unknown distribution by the graph Fourier transform. Therefore, I don't know how to apply the SELU even on the first cnn_graph layer. Could you give me some suggestions on this?

Besides, can I use some normalization on the output of cnn_graph before feeding them into fully connected network using SELU as activation function?

Thanks!

gklambauer commented 6 years ago

Hello Chen, Thanks for using SNNs and for your interest in our work! We really appreciate all the new work that is done on graphs now. With respect to input: SNNs are relatively robust against non-normalized inputs. If it is possible to normalize the inputs to zero mean and unit variance, this is the preferred option. Yes, sure you can use a normalization of the output of cnn_graph before feeding into FC. Probably, even a single batch-norm layer with fixed beta=0, gamma=1 could make sense if the distributions are too far off. I hope this helps!

Regards, Günter

maosi-chen commented 6 years ago

Thank you for answering my questions. They are very helpful!