search

HIPS / molecule-autoencoder

A project to enable optimization of molecules by transforming them to and from a continuous representation.
154 stars 52 forks source link

softmax activation in GRU #2

Open fgvbrt opened 7 years ago

fgvbrt commented 7 years ago

Hi, I noticed that you put softmax activation inside GRU cell, as I understand in this case you wont get sum of activations for each timestep equals to 1. Here is link for GRU cell and the same situation for terminal GRU https://github.com/HIPS/molecule-autoencoder/blob/master/autoencoder/train_autoencoder.py#L225

I also checked with you version of keras that it does not sum to 1, here is link to ghist https://gist.github.com/fgvbrt/1f2e1828c6d8c0eb88614f14c60874ad

Was it done on purpose or was it mistake? Thanks in advance.

fgvbrt commented 7 years ago

I also want to add, that this softmax in GRU would be valid if initialization of initial state would represent probability distribution (i.e. initial states sums to one), but in code there is initialization with zeros.

duvenaud commented 7 years ago

Thanks for catching this! I'll bring it up with Rafa G-B next time I talk to him.