RNN results not reproducible

melodyguan / enas

TensorFlow Code for paper "Efficient Neural Architecture Search via Parameter Sharing"

https://arxiv.org/abs/1802.03268

Apache License 2.0

1.58k stars 390 forks source link

RNN results not reproducible #50

Open Wronskia opened 6 years ago

Wronskia commented 6 years ago

Hello, Thanks for open sourcing the code. After your commit: https://github.com/melodyguan/enas/commit/2734eb2657847f090e1bc5c51c2b9cbf0be51887

I get 63.26 in ppl and not the 55.6 stated in the paper. However before this commit I get 55.6. Is there something I am missing?

Thanks

hyhieu commented 6 years ago

Hi,

Thanks for your interest. Commit 2734eb2 fixed a bug in the evaluation process. After the fix, we had to further tune the model's hyper-parameters to reach a good performance. The best number we could reach was 56.3. We have updated the paper and will soon role out a commit to fix the bug in the code. We apologize for the mistake.

liamcli commented 6 years ago

Could you update the ptb_final.sh script to use the architecture and hyperparameters you used to get to the 56.3 perplexity reported in the paper? Thanks!