search

stefbraun / rnn_benchmarks

RNN benchmarks of pytorch, tensorflow and theano
87 stars 18 forks source link

Is there any reports of the performance / learning curve comparison? #2

Closed rk2900 closed 6 years ago

rk2900 commented 6 years ago

I wonder that what performance do these compared implementations achieved w.r.t. the reported training & inference efficiency? Since in my opinion, the training / inference efficiency is just one side of the coin. Sometime the performance of each model has more importance. So I think it is necessary to report the achieved performance of each implementation. I'm curious about the training or generalization performance of the fast implementation, e.g., cuDNN and LSTM-fused implementations.

Also that is there any learning curves results?

Thanks.

stefbraun commented 6 years ago

Thanks for pointing this out. This repository has a clear focus on measuring the batch processing time (forward + backward pass) for similar network architectures. The accuracy or loss achieved in every epoch should at least be similar if not the same, when the same initialization, cost function and optimizer is used. Otherwise, there would be a bug in the training script or the deep learning framework.

If you are interested in benchmarks that measure time-to-result (hours) or cost-to-result ($), then I recommend the DAWNBench benchmark suite to you (Link).