search

andabi / deep-voice-conversion

Deep neural networks for voice conversion (voice style transfer) in Tensorflow
MIT License
3.92k stars 845 forks source link

Train1 error please help me #100

Open romchina opened 5 years ago

romchina commented 5 years ago

Tried various methods The following are the log and error messages

[0317 23:46:02 @logger.py:87] Argv: train1.py test1 [0317 23:46:02 @parallel.py:195] [MultiProcessPrefetchData] Will fork a dataflow more than one times. This assumes the datapoints are i.i.d. [0317 23:46:02 @training.py:49] [DataParallel] Training a model of 2 towers. [0317 23:46:02 @training.py:51] ERR [DataParallel] TensorFlow was not built with CUDA support! [0317 23:46:02 @interface.py:43] Automatically applying StagingInput on the DataFlow. [0317 23:46:02 @develop.py:99] WRN [Deprecated] ModelDescBase._get_inputs() interface will be deprecated after 30 Mar. Use inputs() instead! Process _Worker-1: Traceback (most recent call last): File "/usr/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap self.run() File "/home/l/.local/lib/python2.7/site-packages/tensorpack/dataflow/parallel.py", line 171, in run for dp in self.ds: File "/home/l/.local/lib/python2.7/site-packages/tensorpack/dataflow/common.py", line 115, in iter for data in self.ds: File "/home/l/voice/deep-voice-conversion/data_load.py", line 35, in get_data yield get_mfccs_and_phones(wav_file=wav_file) File "/home/l/voice/deep-voice-conversion/data_load.py", line 76, in get_mfccs_and_phones hp.default.hop_length) File "/home/l/voice/deep-voice-conversion/data_load.py", line 148, in _get_mfcc_and_spec mel_basis = librosa.filters.mel(hp.default.sr, hp.default.n_fft, hp.default.n_mels) # (n_mels, 1+n_fft//2) File "/home/l/.local/lib/python2.7/site-packages/librosa/filters.py", line 247, in mel lower = -ramps[i] / fdiff[i] ValueError: operands could not be broadcast together with shapes (1,257) (0,) [0317 23:46:02 @input_source.py:220] Setting up the queue 'QueueInput/input_queue' for CPU prefetching ... WARNING:tensorflow:From /home/l/.local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled automatically by placer. Process _Worker-2: Traceback (most recent call last): File "/usr/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap self.run() File "/home/l/.local/lib/python2.7/site-packages/tensorpack/dataflow/parallel.py", line 171, in run for dp in self.ds: File "/home/l/.local/lib/python2.7/site-packages/tensorpack/dataflow/common.py", line 115, in iter for data in self.ds: File "/home/l/voice/deep-voice-conversion/data_load.py", line 35, in get_data yield get_mfccs_and_phones(wav_file=wav_file) File "/home/l/voice/deep-voice-conversion/data_load.py", line 76, in get_mfccs_and_phones hp.default.hop_length) File "/home/l/voice/deep-voice-conversion/data_load.py", line 148, in _get_mfcc_and_spec mel_basis = librosa.filters.mel(hp.default.sr, hp.default.n_fft, hp.default.n_mels) # (n_mels, 1+n_fft//2) File "/home/l/.local/lib/python2.7/site-packages/librosa/filters.py", line 247, in mel lower = -ramps[i] / fdiff[i] ValueError: operands could not be broadcast together with shapes (1,257) (0,) [0317 23:46:02 @training.py:109] Building graph for training tower 0 on device /gpu:0 ... [0317 23:46:02 @develop.py:99] WRN [Deprecated] ModelDescBase._build_graph() interface will be deprecated after 30 Mar. Use build_graph() instead! WARNING:tensorflow:From /home/l/voice/deep-voice-conversion/modules.py:272: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version. Instructions for updating: Use keras.layers.dense instead. Process _Worker-3: Traceback (most recent call last): File "/usr/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap self.run() File "/home/l/.local/lib/python2.7/site-packages/tensorpack/dataflow/parallel.py", line 171, in run for dp in self.ds: File "/home/l/.local/lib/python2.7/site-packages/tensorpack/dataflow/common.py", line 115, in iter for data in self.ds: File "/home/l/voice/deep-voice-conversion/data_load.py", line 35, in get_data yield get_mfccs_and_phones(wav_file=wav_file) File "/home/l/voice/deep-voice-conversion/data_load.py", line 76, in get_mfccs_and_phones hp.default.hop_length) File "/home/l/voice/deep-voice-conversion/data_load.py", line 148, in _get_mfcc_and_spec mel_basis = librosa.filters.mel(hp.default.sr, hp.default.n_fft, hp.default.n_mels) # (n_mels, 1+n_fft//2) File "/home/l/.local/lib/python2.7/site-packages/librosa/filters.py", line 247, in mel lower = -ramps[i] / fdiff[i] ValueError: operands could not be broadcast together with shapes (1,257) (0,) WARNING:tensorflow:From /home/l/voice/deep-voice-conversion/modules.py:273: dropout (from tensorflow.python.layers.core) is deprecated and will be removed in a future version. Instructions for updating: Use keras.layers.dropout instead. WARNING:tensorflow:From /home/l/.local/lib/python2.7/site-packages/tensorflow/python/keras/layers/core.py:143: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version. Instructions for updating: Please use rate instead of keep_prob. Rate should be set to rate = 1 - keep_prob. Process _Worker-4: Traceback (most recent call last): File "/usr/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap self.run() File "/home/l/.local/lib/python2.7/site-packages/tensorpack/dataflow/parallel.py", line 171, in run for dp in self.ds: File "/home/l/.local/lib/python2.7/site-packages/tensorpack/dataflow/common.py", line 115, in iter for data in self.ds: File "/home/l/voice/deep-voice-conversion/data_load.py", line 35, in get_data yield get_mfccs_and_phones(wav_file=wav_file) File "/home/l/voice/deep-voice-conversion/data_load.py", line 76, in get_mfccs_and_phones hp.default.hop_length) File "/home/l/voice/deep-voice-conversion/data_load.py", line 148, in _get_mfcc_and_spec mel_basis = librosa.filters.mel(hp.default.sr, hp.default.n_fft, hp.default.n_mels) # (n_mels, 1+n_fft//2) File "/home/l/.local/lib/python2.7/site-packages/librosa/filters.py", line 247, in mel lower = -ramps[i] / fdiff[i] ValueError: operands could not be broadcast together with shapes (1,257) (0,) WARNING:tensorflow:From /home/l/voice/deep-voice-conversion/modules.py:169: conv1d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version. Instructions for updating: Use keras.layers.conv1d instead. WARNING:tensorflow:From /home/l/voice/deep-voice-conversion/modules.py:312: max_pooling1d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version. Instructions for updating: Use keras.layers.max_pooling1d instead. WARNING:tensorflow:From /home/l/voice/deep-voice-conversion/modules.py:216: init (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version. Instructions for updating: This class is equivalent as tf.keras.layers.GRUCell, and will be replaced by that in Tensorflow 2.0. WARNING:tensorflow:From /home/l/voice/deep-voice-conversion/modules.py:221: bidirectional_dynamic_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version. Instructions for updating: Please use keras.layers.Bidirectional(keras.layers.RNN(cell)), which is equivalent to this API WARNING:tensorflow:From /home/l/.local/lib/python2.7/site-packages/tensorflow/python/ops/rnn.py:443: dynamic_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version. Instructions for updating: Please use keras.layers.RNN(cell), which is equivalent to this API WARNING:tensorflow:From /home/l/voice/deep-voice-conversion/models.py:64: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. WARNING:tensorflow:From /home/l/voice/deep-voice-conversion/models.py:78: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. [0317 23:46:02 @develop.py:99] WRN [Deprecated] get_cost() and self.cost will be deprecated after 30 Mar. Return the cost tensor directly in build_graph() instead! [0317 23:46:02 @develop.py:99] WRN [Deprecated] ModelDescBase._get_optimizer() interface will be deprecated after 30 Mar. Use optimizer() instead! Traceback (most recent call last): File "train1.py", line 78, in train(args, logdir=logdir_train1) File "train1.py", line 60, in train launch_train_with_config(train_conf, trainer=trainer) File "/home/l/.local/lib/python2.7/site-packages/tensorpack/train/interface.py", line 84, in launch_train_with_config model._build_graph_get_cost, model.get_optimizer) File "/home/l/.local/lib/python2.7/site-packages/tensorpack/utils/argtools.py", line 176, in wrapper return func(*args, kwargs) File "/home/l/.local/lib/python2.7/site-packages/tensorpack/train/tower.py", line 214, in setup_graph train_callbacks = self._setup_graph(input, get_cost_fn, get_opt_fn) File "/home/l/.local/lib/python2.7/site-packages/tensorpack/train/trainers.py", line 193, in _setup_graph grad_list = self._builder.call_for_each_tower(tower_fn) File "/home/l/.local/lib/python2.7/site-packages/tensorpack/graph_builder/training.py", line 225, in call_for_each_tower use_vs=[False] + [True] (len(self.towers) - 1)) File "/home/l/.local/lib/python2.7/site-packages/tensorpack/graph_builder/training.py", line 121, in build_on_towers return DataParallelBuilder.call_for_each_tower(args, kwargs) File "/home/l/.local/lib/python2.7/site-packages/tensorpack/graph_builder/training.py", line 116, in call_for_each_tower ret.append(func()) File "/home/l/.local/lib/python2.7/site-packages/tensorpack/train/tower.py", line 266, in get_grad_fn return compute_grad_from_inputs(inputs) File "/home/l/.local/lib/python2.7/site-packages/tensorpack/train/tower.py", line 261, in compute_grad_from_inputs aggregation_method=self.AGGREGATION_METHOD) File "/home/l/.local/lib/python2.7/site-packages/tensorflow/python/training/optimizer.py", line 512, in compute_gradients colocate_gradients_with_ops=colocate_gradients_with_ops) File "/home/l/.local/lib/python2.7/site-packages/tensorflow/python/ops/gradients_impl.py", line 664, in gradients unconnected_gradients) File "/home/l/.local/lib/python2.7/site-packages/tensorflow/python/ops/gradients_impl.py", line 965, in _GradientsHelper lambda: grad_fn(op, out_grads)) File "/home/l/.local/lib/python2.7/site-packages/tensorflow/python/ops/gradients_impl.py", line 420, in _MaybeCompile return grad_fn() # Exit early File "/home/l/.local/lib/python2.7/site-packages/tensorflow/python/ops/gradients_impl.py", line 965, in lambda: grad_fn(op, out_grads)) File "/home/l/.local/lib/python2.7/site-packages/tensorflow/python/ops/while_v2.py", line 245, in _WhileGrad cond_graph = _get_graph(op, "cond") File "/home/l/.local/lib/python2.7/site-packages/tensorflow/python/ops/while_v2.py", line 425, in _get_graph func_graph = function_def_to_graph.function_def_to_graph(fdef, input_shapes) File "/home/l/.local/lib/python2.7/site-packages/tensorflow/python/framework/function_def_to_graph.py", line 55, in function_def_to_graph importer.import_graph_def(graph_def, name="") File "/home/l/.local/lib/python2.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func return func(args, **kwargs) File "/home/l/.local/lib/python2.7/site-packages/tensorflow/python/framework/importer.py", line 442, in import_graph_def _ProcessNewOps(graph) File "/home/l/.local/lib/python2.7/site-packages/tensorflow/python/framework/importer.py", line 274, in _ProcessNewOps (coloc_op_name, op.name)) ValueError: Specified colocation to an op that does not exist during import: tower0/net1/cbhg/gru/bidirectional_rnn/fw/fw/while in less_1_tower0_net1_cbhg_gru_bidirectional_rnn_fw_fw_minimum l@l-pc:~/voice/deep-voice-conversion$ sudo python train1.py test1 case: test1, logdir: /data/private/vc/logdir/test1/train1 [0317 23:46:10 @logger.py:122] WRN Log directory /data/private/vc/logdir/test1/train1 exists! Use 'd' to delete it. [0317 23:46:10 @logger.py:125] WRN If you're resuming from a previous run, you can choose to keep it. Press any other key to exit. Select Action: k (keep) / d (delete) / q (quit):d [0317 23:46:11 @logger.py:87] Argv: train1.py test1 [0317 23:46:11 @parallel.py:195] [MultiProcessPrefetchData] Will fork a dataflow more than one times. This assumes the datapoints are i.i.d. [0317 23:46:11 @develop.py:99] WRN [Deprecated] ModelDescBase._get_inputs() interface will be deprecated after 30 Mar. Use inputs() instead! [0317 23:46:11 @input_source.py:220] Setting up the queue 'QueueInput/input_queue' for CPU prefetching ... WARNING:tensorflow:From /home/l/.local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled automatically by placer. Process _Worker-1: Traceback (most recent call last): File "/usr/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap self.run() File "/home/l/.local/lib/python2.7/site-packages/tensorpack/dataflow/parallel.py", line 171, in run for dp in self.ds: File "/home/l/.local/lib/python2.7/site-packages/tensorpack/dataflow/common.py", line 115, in iter for data in self.ds: File "/home/l/voice/deep-voice-conversion/data_load.py", line 35, in get_data yield get_mfccs_and_phones(wav_file=wav_file) File "/home/l/voice/deep-voice-conversion/data_load.py", line 76, in get_mfccs_and_phones hp.default.hop_length) File "/home/l/voice/deep-voice-conversion/data_load.py", line 148, in _get_mfcc_and_spec mel_basis = librosa.filters.mel(hp.default.sr, hp.default.n_fft, hp.default.n_mels) # (n_mels, 1+n_fft//2) File "/home/l/.local/lib/python2.7/site-packages/librosa/filters.py", line 247, in mel lower = -ramps[i] / fdiff[i] ValueError: operands could not be broadcast together with shapes (1,257) (0,) [0317 23:46:11 @training.py:109] Building graph for training tower 0 on device /gpu:0 ... [0317 23:46:11 @develop.py:99] WRN [Deprecated] ModelDescBase._build_graph() interface will be deprecated after 30 Mar. Use build_graph() instead! WARNING:tensorflow:From /home/l/voice/deep-voice-conversion/modules.py:272: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version. Instructions for updating: Use keras.layers.dense instead. Process _Worker-2: Traceback (most recent call last): File "/usr/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap self.run() File "/home/l/.local/lib/python2.7/site-packages/tensorpack/dataflow/parallel.py", line 171, in run for dp in self.ds: File "/home/l/.local/lib/python2.7/site-packages/tensorpack/dataflow/common.py", line 115, in iter for data in self.ds: File "/home/l/voice/deep-voice-conversion/data_load.py", line 35, in get_data yield get_mfccs_and_phones(wav_file=wav_file) File "/home/l/voice/deep-voice-conversion/data_load.py", line 76, in get_mfccs_and_phones hp.default.hop_length) File "/home/l/voice/deep-voice-conversion/data_load.py", line 148, in _get_mfcc_and_spec mel_basis = librosa.filters.mel(hp.default.sr, hp.default.n_fft, hp.default.n_mels) # (n_mels, 1+n_fft//2) File "/home/l/.local/lib/python2.7/site-packages/librosa/filters.py", line 247, in mel lower = -ramps[i] / fdiff[i] ValueError: operands could not be broadcast together with shapes (1,257) (0,) WARNING:tensorflow:From /home/l/voice/deep-voice-conversion/modules.py:273: dropout (from tensorflow.python.layers.core) is deprecated and will be removed in a future version. Instructions for updating: Use keras.layers.dropout instead. WARNING:tensorflow:From /home/l/.local/lib/python2.7/site-packages/tensorflow/python/keras/layers/core.py:143: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version. Instructions for updating: Please use rate instead of keep_prob. Rate should be set to rate = 1 - keep_prob. Process _Worker-3: Traceback (most recent call last): File "/usr/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap self.run() File "/home/l/.local/lib/python2.7/site-packages/tensorpack/dataflow/parallel.py", line 171, in run for dp in self.ds: File "/home/l/.local/lib/python2.7/site-packages/tensorpack/dataflow/common.py", line 115, in iter for data in self.ds: File "/home/l/voice/deep-voice-conversion/data_load.py", line 35, in get_data yield get_mfccs_and_phones(wav_file=wav_file) File "/home/l/voice/deep-voice-conversion/data_load.py", line 76, in get_mfccs_and_phones hp.default.hop_length) File "/home/l/voice/deep-voice-conversion/data_load.py", line 148, in _get_mfcc_and_spec mel_basis = librosa.filters.mel(hp.default.sr, hp.default.n_fft, hp.default.n_mels) # (n_mels, 1+n_fft//2) File "/home/l/.local/lib/python2.7/site-packages/librosa/filters.py", line 247, in mel lower = -ramps[i] / fdiff[i] ValueError: operands could not be broadcast together with shapes (1,257) (0,) Process _Worker-4: Traceback (most recent call last): File "/usr/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap self.run() File "/home/l/.local/lib/python2.7/site-packages/tensorpack/dataflow/parallel.py", line 171, in run for dp in self.ds: File "/home/l/.local/lib/python2.7/site-packages/tensorpack/dataflow/common.py", line 115, in iter for data in self.ds: File "/home/l/voice/deep-voice-conversion/data_load.py", line 35, in get_data yield get_mfccs_and_phones(wav_file=wav_file) File "/home/l/voice/deep-voice-conversion/data_load.py", line 76, in get_mfccs_and_phones hp.default.hop_length) File "/home/l/voice/deep-voice-conversion/data_load.py", line 148, in _get_mfcc_and_spec mel_basis = librosa.filters.mel(hp.default.sr, hp.default.n_fft, hp.default.n_mels) # (n_mels, 1+n_fft//2) File "/home/l/.local/lib/python2.7/site-packages/librosa/filters.py", line 247, in mel lower = -ramps[i] / fdiff[i] ValueError: operands could not be broadcast together with shapes (1,257) (0,) WARNING:tensorflow:From /home/l/voice/deep-voice-conversion/modules.py:169: conv1d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version. Instructions for updating: Use keras.layers.conv1d instead. WARNING:tensorflow:From /home/l/voice/deep-voice-conversion/modules.py:312: max_pooling1d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version. Instructions for updating: Use keras.layers.max_pooling1d instead. WARNING:tensorflow:From /home/l/voice/deep-voice-conversion/modules.py:216: init (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version. Instructions for updating: This class is equivalent as tf.keras.layers.GRUCell, and will be replaced by that in Tensorflow 2.0. WARNING:tensorflow:From /home/l/voice/deep-voice-conversion/modules.py:221: bidirectional_dynamic_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version. Instructions for updating: Please use keras.layers.Bidirectional(keras.layers.RNN(cell)), which is equivalent to this API WARNING:tensorflow:From /home/l/.local/lib/python2.7/site-packages/tensorflow/python/ops/rnn.py:443: dynamic_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version. Instructions for updating: Please use keras.layers.RNN(cell), which is equivalent to this API WARNING:tensorflow:From /home/l/voice/deep-voice-conversion/models.py:64: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. WARNING:tensorflow:From /home/l/voice/deep-voice-conversion/models.py:78: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. [0317 23:46:12 @develop.py:99] WRN [Deprecated] get_cost() and self.cost will be deprecated after 30 Mar. Return the cost tensor directly in build_graph() instead! [0317 23:46:12 @develop.py:99] WRN [Deprecated] ModelDescBase._get_optimizer() interface will be deprecated after 30 Mar. Use optimizer() instead! Traceback (most recent call last): File "train1.py", line 78, in train(args, logdir=logdir_train1) File "train1.py", line 60, in train launch_train_with_config(train_conf, trainer=trainer) File "/home/l/.local/lib/python2.7/site-packages/tensorpack/train/interface.py", line 84, in launch_train_with_config model._build_graph_get_cost, model.get_optimizer) File "/home/l/.local/lib/python2.7/site-packages/tensorpack/utils/argtools.py", line 176, in wrapper return func(*args, kwargs) File "/home/l/.local/lib/python2.7/site-packages/tensorpack/train/tower.py", line 214, in setup_graph train_callbacks = self._setup_graph(input, get_cost_fn, get_opt_fn) File "/home/l/.local/lib/python2.7/site-packages/tensorpack/train/trainers.py", line 193, in _setup_graph grad_list = self._builder.call_for_each_tower(tower_fn) File "/home/l/.local/lib/python2.7/site-packages/tensorpack/graph_builder/training.py", line 225, in call_for_each_tower use_vs=[False] + [True] (len(self.towers) - 1)) File "/home/l/.local/lib/python2.7/site-packages/tensorpack/graph_builder/training.py", line 121, in build_on_towers return DataParallelBuilder.call_for_each_tower(args, kwargs) File "/home/l/.local/lib/python2.7/site-packages/tensorpack/graph_builder/training.py", line 116, in call_for_each_tower ret.append(func()) File "/home/l/.local/lib/python2.7/site-packages/tensorpack/train/tower.py", line 266, in get_grad_fn return compute_grad_from_inputs(inputs) File "/home/l/.local/lib/python2.7/site-packages/tensorpack/train/tower.py", line 261, in compute_grad_from_inputs aggregation_method=self.AGGREGATION_METHOD) File "/home/l/.local/lib/python2.7/site-packages/tensorflow/python/training/optimizer.py", line 512, in compute_gradients colocate_gradients_with_ops=colocate_gradients_with_ops) File "/home/l/.local/lib/python2.7/site-packages/tensorflow/python/ops/gradients_impl.py", line 664, in gradients unconnected_gradients) File "/home/l/.local/lib/python2.7/site-packages/tensorflow/python/ops/gradients_impl.py", line 965, in _GradientsHelper lambda: grad_fn(op, out_grads)) File "/home/l/.local/lib/python2.7/site-packages/tensorflow/python/ops/gradients_impl.py", line 420, in _MaybeCompile return grad_fn() # Exit early File "/home/l/.local/lib/python2.7/site-packages/tensorflow/python/ops/gradients_impl.py", line 965, in lambda: grad_fn(op, out_grads)) File "/home/l/.local/lib/python2.7/site-packages/tensorflow/python/ops/while_v2.py", line 245, in _WhileGrad cond_graph = _get_graph(op, "cond") File "/home/l/.local/lib/python2.7/site-packages/tensorflow/python/ops/while_v2.py", line 425, in _get_graph func_graph = function_def_to_graph.function_def_to_graph(fdef, input_shapes) File "/home/l/.local/lib/python2.7/site-packages/tensorflow/python/framework/function_def_to_graph.py", line 55, in function_def_to_graph importer.import_graph_def(graph_def, name="") File "/home/l/.local/lib/python2.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func return func(args, **kwargs) File "/home/l/.local/lib/python2.7/site-packages/tensorflow/python/framework/importer.py", line 442, in import_graph_def _ProcessNewOps(graph) File "/home/l/.local/lib/python2.7/site-packages/tensorflow/python/framework/importer.py", line 274, in _ProcessNewOps (coloc_op_name, op.name)) ValueError: Specified colocation to an op that does not exist during import: tower0/net1/cbhg/gru/bidirectional_rnn/fw/fw/while in tower0_net1_cbhg_gru_bidirectional_rnn_fw_fw_grucellzerostate_zeros l@l-pc:~/voice/deep-voice-conversion$ sudo python train1.py test1 case: test1, logdir: /data/private/vc/logdir/test1/train1 [0317 23:46:22 @logger.py:122] WRN Log directory /data/private/vc/logdir/test1/train1 exists! Use 'd' to delete it. [0317 23:46:22 @logger.py:125] WRN If you're resuming from a previous run, you can choose to keep it. Press any other key to exit. Select Action: k (keep) / d (delete) / q (quit):d [0317 23:46:22 @logger.py:87] Argv: train1.py test1 [0317 23:46:22 @parallel.py:195] [MultiProcessPrefetchData] Will fork a dataflow more than one times. This assumes the datapoints are i.i.d. [0317 23:46:22 @training.py:49] [DataParallel] Training a model of 3 towers. [0317 23:46:22 @training.py:51] ERR [DataParallel] TensorFlow was not built with CUDA support! [0317 23:46:22 @interface.py:43] Automatically applying StagingInput on the DataFlow. [0317 23:46:22 @develop.py:99] WRN [Deprecated] ModelDescBase._get_inputs() interface will be deprecated after 30 Mar. Use inputs() instead! [0317 23:46:22 @input_source.py:220] Setting up the queue 'QueueInput/input_queue' for CPU prefetching ... WARNING:tensorflow:From /home/l/.local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled automatically by placer. [0317 23:46:22 @training.py:109] Building graph for training tower 0 on device /gpu:0 ... [0317 23:46:23 @develop.py:99] WRN [Deprecated] ModelDescBase._build_graph() interface will be deprecated after 30 Mar. Use build_graph() instead! WARNING:tensorflow:From /home/l/voice/deep-voice-conversion/modules.py:272: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version. Instructions for updating: Use keras.layers.dense instead. WARNING:tensorflow:From /home/l/voice/deep-voice-conversion/modules.py:273: dropout (from tensorflow.python.layers.core) is deprecated and will be removed in a future version. Instructions for updating: Use keras.layers.dropout instead. WARNING:tensorflow:From /home/l/.local/lib/python2.7/site-packages/tensorflow/python/keras/layers/core.py:143: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version. Instructions for updating: Please use rate instead of keep_prob. Rate should be set to rate = 1 - keep_prob. WARNING:tensorflow:From /home/l/voice/deep-voice-conversion/modules.py:169: conv1d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version. Instructions for updating: Use keras.layers.conv1d instead. Process _Worker-1: Traceback (most recent call last): File "/usr/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap self.run() File "/home/l/.local/lib/python2.7/site-packages/tensorpack/dataflow/parallel.py", line 171, in run for dp in self.ds: File "/home/l/.local/lib/python2.7/site-packages/tensorpack/dataflow/common.py", line 115, in iter for data in self.ds: File "/home/l/voice/deep-voice-conversion/data_load.py", line 35, in get_data yield get_mfccs_and_phones(wav_file=wav_file) File "/home/l/voice/deep-voice-conversion/data_load.py", line 76, in get_mfccs_and_phones hp.default.hop_length) File "/home/l/voice/deep-voice-conversion/data_load.py", line 148, in _get_mfcc_and_spec mel_basis = librosa.filters.mel(hp.default.sr, hp.default.n_fft, hp.default.n_mels) # (n_mels, 1+n_fft//2) File "/home/l/.local/lib/python2.7/site-packages/librosa/filters.py", line 247, in mel lower = -ramps[i] / fdiff[i] ValueError: operands could not be broadcast together with shapes (1,257) (0,) Process _Worker-2: Traceback (most recent call last): File "/usr/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap self.run() File "/home/l/.local/lib/python2.7/site-packages/tensorpack/dataflow/parallel.py", line 171, in run for dp in self.ds: File "/home/l/.local/lib/python2.7/site-packages/tensorpack/dataflow/common.py", line 115, in iter for data in self.ds: File "/home/l/voice/deep-voice-conversion/data_load.py", line 35, in get_data yield get_mfccs_and_phones(wav_file=wav_file) File "/home/l/voice/deep-voice-conversion/data_load.py", line 76, in get_mfccs_and_phones hp.default.hop_length) File "/home/l/voice/deep-voice-conversion/data_load.py", line 148, in _get_mfcc_and_spec mel_basis = librosa.filters.mel(hp.default.sr, hp.default.n_fft, hp.default.n_mels) # (n_mels, 1+n_fft//2) File "/home/l/.local/lib/python2.7/site-packages/librosa/filters.py", line 247, in mel lower = -ramps[i] / fdiff[i] ValueError: operands could not be broadcast together with shapes (1,257) (0,) Process _Worker-4: Traceback (most recent call last): File "/usr/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap self.run() File "/home/l/.local/lib/python2.7/site-packages/tensorpack/dataflow/parallel.py", line 171, in run for dp in self.ds: File "/home/l/.local/lib/python2.7/site-packages/tensorpack/dataflow/common.py", line 115, in iter for data in self.ds: File "/home/l/voice/deep-voice-conversion/data_load.py", line 35, in get_data yield get_mfccs_and_phones(wav_file=wav_file) File "/home/l/voice/deep-voice-conversion/data_load.py", line 76, in get_mfccs_and_phones hp.default.hop_length) File "/home/l/voice/deep-voice-conversion/data_load.py", line 148, in _get_mfcc_and_spec mel_basis = librosa.filters.mel(hp.default.sr, hp.default.n_fft, hp.default.n_mels) # (n_mels, 1+n_fft//2) File "/home/l/.local/lib/python2.7/site-packages/librosa/filters.py", line 247, in mel lower = -ramps[i] / fdiff[i] ValueError: operands could not be broadcast together with shapes (1,257) (0,) WARNING:tensorflow:From /home/l/voice/deep-voice-conversion/modules.py:312: max_pooling1d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version. Instructions for updating: Use keras.layers.max_pooling1d instead. Process _Worker-3: Traceback (most recent call last): File "/usr/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap self.run() File "/home/l/.local/lib/python2.7/site-packages/tensorpack/dataflow/parallel.py", line 171, in run for dp in self.ds: File "/home/l/.local/lib/python2.7/site-packages/tensorpack/dataflow/common.py", line 115, in iter for data in self.ds: File "/home/l/voice/deep-voice-conversion/data_load.py", line 35, in get_data yield get_mfccs_and_phones(wav_file=wav_file) File "/home/l/voice/deep-voice-conversion/data_load.py", line 76, in get_mfccs_and_phones hp.default.hop_length) File "/home/l/voice/deep-voice-conversion/data_load.py", line 148, in _get_mfcc_and_spec mel_basis = librosa.filters.mel(hp.default.sr, hp.default.n_fft, hp.default.n_mels) # (n_mels, 1+n_fft//2) File "/home/l/.local/lib/python2.7/site-packages/librosa/filters.py", line 247, in mel lower = -ramps[i] / fdiff[i] ValueError: operands could not be broadcast together with shapes (1,257) (0,) WARNING:tensorflow:From /home/l/voice/deep-voice-conversion/modules.py:216: init (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version. Instructions for updating: This class is equivalent as tf.keras.layers.GRUCell, and will be replaced by that in Tensorflow 2.0. WARNING:tensorflow:From /home/l/voice/deep-voice-conversion/modules.py:221: bidirectional_dynamic_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version. Instructions for updating: Please use keras.layers.Bidirectional(keras.layers.RNN(cell)), which is equivalent to this API WARNING:tensorflow:From /home/l/.local/lib/python2.7/site-packages/tensorflow/python/ops/rnn.py:443: dynamic_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version. Instructions for updating: Please use keras.layers.RNN(cell), which is equivalent to this API WARNING:tensorflow:From /home/l/voice/deep-voice-conversion/models.py:64: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. WARNING:tensorflow:From /home/l/voice/deep-voice-conversion/models.py:78: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. [0317 23:46:23 @develop.py:99] WRN [Deprecated] get_cost() and self.cost will be deprecated after 30 Mar. Return the cost tensor directly in build_graph() instead! [0317 23:46:23 @develop.py:99] WRN [Deprecated] ModelDescBase._get_optimizer() interface will be deprecated after 30 Mar. Use optimizer() instead! Traceback (most recent call last): File "train1.py", line 78, in train(args, logdir=logdir_train1) File "train1.py", line 60, in train launch_train_with_config(train_conf, trainer=trainer) File "/home/l/.local/lib/python2.7/site-packages/tensorpack/train/interface.py", line 84, in launch_train_with_config model._build_graph_get_cost, model.get_optimizer) File "/home/l/.local/lib/python2.7/site-packages/tensorpack/utils/argtools.py", line 176, in wrapper return func(*args, kwargs) File "/home/l/.local/lib/python2.7/site-packages/tensorpack/train/tower.py", line 214, in setup_graph train_callbacks = self._setup_graph(input, get_cost_fn, get_opt_fn) File "/home/l/.local/lib/python2.7/site-packages/tensorpack/train/trainers.py", line 193, in _setup_graph grad_list = self._builder.call_for_each_tower(tower_fn) File "/home/l/.local/lib/python2.7/site-packages/tensorpack/graph_builder/training.py", line 225, in call_for_each_tower use_vs=[False] + [True] (len(self.towers) - 1)) File "/home/l/.local/lib/python2.7/site-packages/tensorpack/graph_builder/training.py", line 121, in build_on_towers return DataParallelBuilder.call_for_each_tower(args, kwargs) File "/home/l/.local/lib/python2.7/site-packages/tensorpack/graph_builder/training.py", line 116, in call_for_each_tower ret.append(func()) File "/home/l/.local/lib/python2.7/site-packages/tensorpack/train/tower.py", line 266, in get_grad_fn return compute_grad_from_inputs(inputs) File "/home/l/.local/lib/python2.7/site-packages/tensorpack/train/tower.py", line 261, in compute_grad_from_inputs aggregation_method=self.AGGREGATION_METHOD) File "/home/l/.local/lib/python2.7/site-packages/tensorflow/python/training/optimizer.py", line 512, in compute_gradients colocate_gradients_with_ops=colocate_gradients_with_ops) File "/home/l/.local/lib/python2.7/site-packages/tensorflow/python/ops/gradients_impl.py", line 664, in gradients unconnected_gradients) File "/home/l/.local/lib/python2.7/site-packages/tensorflow/python/ops/gradients_impl.py", line 965, in _GradientsHelper lambda: grad_fn(op, out_grads)) File "/home/l/.local/lib/python2.7/site-packages/tensorflow/python/ops/gradients_impl.py", line 420, in _MaybeCompile return grad_fn() # Exit early File "/home/l/.local/lib/python2.7/site-packages/tensorflow/python/ops/gradients_impl.py", line 965, in lambda: grad_fn(op, out_grads)) File "/home/l/.local/lib/python2.7/site-packages/tensorflow/python/ops/while_v2.py", line 245, in _WhileGrad cond_graph = _get_graph(op, "cond") File "/home/l/.local/lib/python2.7/site-packages/tensorflow/python/ops/while_v2.py", line 425, in _get_graph func_graph = function_def_to_graph.function_def_to_graph(fdef, input_shapes) File "/home/l/.local/lib/python2.7/site-packages/tensorflow/python/framework/function_def_to_graph.py", line 55, in function_def_to_graph importer.import_graph_def(graph_def, name="") File "/home/l/.local/lib/python2.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func return func(args, **kwargs) File "/home/l/.local/lib/python2.7/site-packages/tensorflow/python/framework/importer.py", line 442, in import_graph_def _ProcessNewOps(graph) File "/home/l/.local/lib/python2.7/site-packages/tensorflow/python/framework/importer.py", line 274, in _ProcessNewOps (coloc_op_name, op.name)) ValueError: Specified colocation to an op that does not exist during import: tower0/net1/cbhg/gru/bidirectional_rnn/fw/fw/while in less_1_tower0_net1_cbhg_gru_bidirectional_rnn_fw_fw_minimum

log:

[0317 23:46:22 @logger.py:87] Argv: train1.py test1 [0317 23:46:22 @parallel.py:195] [MultiProcessPrefetchData] Will fork a dataflow more than one times. This assumes the datapoints are i.i.d. [0317 23:46:22 @training.py:49] [DataParallel] Training a model of 3 towers. [0317 23:46:22 @training.py:51] ERR [DataParallel] TensorFlow was not built with CUDA support! [0317 23:46:22 @interface.py:43] Automatically applying StagingInput on the DataFlow. [0317 23:46:22 @develop.py:99] WRN [Deprecated] ModelDescBase._get_inputs() interface will be deprecated after 30 Mar. Use inputs() instead! [0317 23:46:22 @input_source.py:220] Setting up the queue 'QueueInput/input_queue' for CPU prefetching ... [0317 23:46:22 @training.py:109] Building graph for training tower 0 on device /gpu:0 ... [0317 23:46:23 @develop.py:99] WRN [Deprecated] ModelDescBase._build_graph() interface will be deprecated after 30 Mar. Use build_graph() instead! [0317 23:46:23 @develop.py:99] WRN [Deprecated] get_cost() and self.cost will be deprecated after 30 Mar. Return the cost tensor directly in build_graph() instead! [0317 23:46:23 @develop.py:99] WRN [Deprecated] ModelDescBase._get_optimizer() interface will be deprecated after 30 Mar. Use optimizer() instead!

mindmapper15 commented 5 years ago

Looks like train1.py didn't get any input data to train. You'd better check your Dataset files' name and Dataset directory in hparams/default.yaml