search

bfs18 / nsynth_wavenet

parallel wavenet based on nsynth
106 stars 30 forks source link

Reloading ns_wn-gauss for teacher continue training #15

Open zhang-jian opened 6 years ago

zhang-jian commented 6 years ago

Hi when reloading ns_wn-gauss model for teacher continue training, I got

INFO:tensorflow:Restoring parameters from ns_wn-gauss/model.ckpt-200000
2018-08-09 15:04:30.388764: W tensorflow/core/framework/op_kernel.cc:1202] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key beta1_power not found in checkpoint
INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.NotFoundError'>, Key beta1_power not found in checkpoint
     [[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

Any idea?

bfs18 commented 6 years ago

Hi, the checkpoint only contains the variables which is used in evaluaiton. It does not contain the variables used only in training, e.g. adam variables. The origianl checkpoint has larger size, so I extracted the variables for evaluation. If you need one, I can upload a copy.

maozhiqiang commented 6 years ago

Hi @bfs18 ! the training process is so slowly ! Can you upload one copy?

bfs18 commented 6 years ago

https://drive.google.com/open?id=1PZQROhlo6RjtKyTmIS8faUtkDrke2VTG

maozhiqiang commented 6 years ago

@bfs18 ！ thanks!

WendongGan commented 6 years ago

@bfs18 ，Hi! Can you upload a copy of the whole “wavenet model: ns_wn-eval.tar.gz”? I also want to reloading ns_wn for teacher continue training. Thanks for your help! Look forward to your reply!