search

magenta / magenta

Magenta: Music and Art Generation with Machine Intelligence
Apache License 2.0
19.13k stars 3.74k forks source link

sketch-rnn: Time to train #742

Closed alexis-jacq closed 7 years ago

alexis-jacq commented 7 years ago

Hello, thanks for this amazing algorithm!

I am writing a Pytorch version of sketch-rnn, and I would like to know what is the average time of training with a simple laptop's GPU. (using the same default hyperparameters, and only training to generate samples from the cat.npz dataset)

Also, what is approximatively the minimal number of epoch in order to get recognizable drawing generations?

hardmaru commented 7 years ago

Hi!

If you don't care about total convergence, training on a single class in QuickDraw for half a day on a TITAN-X is enough.

Tips for faster training at the expense of quality:

-anneal KL faster: use 0.9999 rather than 0.99995 -use learning rate at 0.0001 and not anneal below this rate to 0.00001, at the higher risk of NaNs -turn off dropout, train faster but risk overfitting too soon

Good luck with the implementation

On Mon, Jun 19, 2017 at 8:16 AM Alexis David Jacq notifications@github.com wrote:

Hello, thanks for this amazing algorithm!

I am writing a Pytorch version of sketch-rnn, and I would like to know what is the average time of training with a simple laptop's GPU. (using the same default hyperparameters, and only training to generate samples from the cat.npz dataset)

Also, what is approximatively the minimal number of epoch in order to get recognizable drawing generations?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/tensorflow/magenta/issues/742, or mute the thread https://github.com/notifications/unsubscribe-auth/AGBoHjpv3fDn0KLkDEJAvBE4Opp3ENA5ks5sFpDpgaJpZM4N-YZD .

alexis-jacq commented 7 years ago

Ok ! Thanks a lot for the tips !

falaktheoptimist commented 7 years ago

Hi @alexis-jacq @hardmaru Thanks for the tips @hardmaru. We're trying out something similar- creating similar model using keras. How many epochs did you end up training upto? Because the default number of steps 1e7 seems abnormally high. Also, at what number of steps does total convergence start?

hardmaru commented 7 years ago

You should be able to see some results (i.e. it should draw something that look like crabs if you trained on crab.npz), after 10-20K steps. We set the default number of steps close to infinity to allow us to control manually when we want to stop training, by looking at the TensorBoard graphs and the various reported losses. In practice once the validation loss curves stop decreasing we stop the training.

The convergence depends on the data, and also the regularization scheme used, and size of the network, so can vary between 50K-500K steps even.

falaktheoptimist commented 7 years ago

Ok. That helps a lot. Many thanks @hardmaru

hardmaru commented 7 years ago

The loss function for the decoder is somewhat non standard, so you may encounter some challenges if you want to implement in a high level framework such as Keras, but I think it should be possible.

If you manage to write the model in Keras in the future, please let us know so we can refer your implementation to Keras users or other interested parties.

falaktheoptimist commented 7 years ago

Ok sure.. We're making an attempt. Will share if we get some good results.

alexiskattan commented 6 years ago

@falaktheoptimist any luck getting sketch rnn to work in keras?

KKeishiro commented 5 years ago

@hardmaru @alexis-jacq @falaktheoptimist @alexiskattan I implemented Sketch-RNN in keras. The repo can be found here. Any comments are welcomed. Cheers