search

openai / generating-reviews-discovering-sentiment

Code for "Learning to Generate Reviews and Discovering Sentiment"
https://arxiv.org/abs/1704.01444
MIT License
1.51k stars 380 forks source link

Weight Initialization #38

Open jonny-d opened 7 years ago

jonny-d commented 7 years ago

Hello,

Thank you very much for sharing this code. I am attempting to re-train a model like this from scratch and was wondering which weight initialization method was used for training the model?

Thanks, Jonny

raulpuric commented 7 years ago

Without any regularization I personally found that uniform sampling gave faster convergence, but was more unstable and blew up (see my issue #39). I also tried xavier initialization and that seemed to be more stable.

jonny-d commented 7 years ago

Hello, sorry for the delayed response.

I have achieved pretty good performance using a normal distribution for the initial weights. Here is a link to my Tensorflow Implementation

raulpuric commented 7 years ago

@jonnykira I found like you that they used weight norm in the paper which I initially glossed over/isn't in the code base. This turned out to be what I needed to use.

nkooli commented 7 years ago

@jonnykira hello, I trained the model on my dataset, this generates me three files : model.data, model.index, model.meta

How can i generates the 15 .npy weight files (0.npy, 1.npy, ..., 14.npy) to test the sentiment analysis code (as in in https://github.com/openai/generating-reviews-discovering-sentiment) ?

thank you !