Does the algorithm stops training on its own?

[wolfganggillisten]

Hi there, I have been training a Wavegan model from about 6 hours now, using 11 short WAV files. I have been following the training process using TensorBoard and things look good. However, I am unclear about how long the training process will take and how would I know when the process has finished.

Thanks for your help!

[chrisdonahue]

The training process does not automatically terminate. In our paper, we terminate after some fixed number of iterations (200k IIRC). You can till the training script whenever the generated examples are to your liking.

[wolfganggillisten]

Hi Chris, thanks for getting back to me. And again, great stuff!. Yeah I suspected that after looking at the code. Where can I read about how to stop the training process and export the model to a local drive? in the paper?

Thank you so much!

On Tue, Sep 24, 2019 at 3:31 PM Chris Donahue notifications@github.com wrote:

The training process does not automatically terminate. In our paper, we terminate after some fixed number of iterations (200k IIRC). You can till the training script whenever the generated examples are to your liking.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/chrisdonahue/wavegan/issues/62?email_source=notifications&email_token=ALMKWP25NLKZ4BPSWQEXBCDQLJTI5A5CNFSM4IZSI5HKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7PRKRI#issuecomment-534713669, or mute the thread https://github.com/notifications/unsubscribe-auth/ALMKWPZG2USYGPNAVAEZ2PLQLJTI5ANCNFSM4IZSI5HA .

[chrisdonahue]

The checkpoints should be saving intermittently to the training directory specified to the training script. You can load the most recent checkpoint for generation. More info in the README, nothing about that in the paper

[wolfganggillisten]

Hi Chris, I think I was able to load a check point by using the following code.

import tensorflow as tf

tf.reset_default_graph() saver = tf.train.import_meta_graph('model.ckpt-6524.meta') <-6524 is the current step number of the training process graph = tf.get_default_graph() sess = tf.InteractiveSession() saver.restore(sess, 'model.ckpt-6524')

But when trying to get tensor z_0 by name got the following error

"The name 'z:0' refers to a Tensor which does not exist. The operation, 'z', does not exist in the graph."

I ran [n.name for n in tf.get_default_graph().as_graph_def().node] on the loaded graph and got an extensive list of operators, which I interpret as the model having data in it.

Could you please help me out loading the model properly and get to listen to the generated sounds?

Thank you so much,

Wolfgang Gil http://wolfganggil.info

On Wed, Sep 25, 2019 at 12:55 AM Chris Donahue notifications@github.com wrote:

The checkpoints should be saving intermittently to the training directory specified to the training script. You can load the most recent checkpoint for generation. More info in the README, nothing about that in the paper

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/chrisdonahue/wavegan/issues/62?email_source=notifications&email_token=ALMKWP7TLUUAGJDAU345U6TQLLVKVA5CNFSM4IZSI5HKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7QS6VY#issuecomment-534851415, or mute the thread https://github.com/notifications/unsubscribe-auth/ALMKWP4GIQFTQFEQ3YUCVNTQLLVKVANCNFSM4IZSI5HA .

[chrisdonahue]

Sorry it's taken me a while to respond. The metagraphs from the checkpoint directory don't have the proper named tensors to support the code snippet you sent. Try downloading the inference metagraph from the example notebook and loading that before loading the weights:

!wget https://s3.amazonaws.com/wavegan-v1/models/sc09_infer.meta -O infer.meta