Closed zhanggang11863976 closed 2 years ago
change the conf file melgan.v1.yaml "batch_max_steps_valid" from 81920 to 8192 can fix this problem
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
tensorflow-addons 0.17.1 tensorflow-estimator 2.7.0 tensorflow-gpu 2.7.0 tensorflow-io-gcs-filesystem 0.24.0
python3 examples/melgan/train_melgan.py \
2 --train-dir ./dump_baker/train/ \
3 --dev-dir ./dump_baker/valid/ \
4 --outdir ./examples/melgan/exp/train.melgan.v2/ \
5 --config ./examples/melgan/conf/melgan.v1.yaml \
6 --use-norm 1 \
7 --generator_mixed_precision 0 \
8 --resume "./examples/melgan/exp/train.melgan.v2/checkpoints"
File "/root/paddlejob/workspace/TensorFlowTTS-master/tensorflow_tts/trainers/base_trainer.py", line 685, in fit self.run() File "/root/paddlejob/workspace/TensorFlowTTS-master/tensorflow_tts/trainers/base_trainer.py", line 104, in run self._train_epoch() File "/root/paddlejob/workspace/TensorFlowTTS-master/tensorflow_tts/trainers/base_trainer.py", line 130, in _train_epoch self._check_eval_interval() File "/root/paddlejob/workspace/TensorFlowTTS-master/tensorflow_tts/trainers/base_trainer.py", line 167, in _check_eval_interval self._eval_epoch() File "/root/paddlejob/workspace/TensorFlowTTS-master/tensorflow_tts/trainers/base_trainer.py", line 510, in _eval_epoch tqdm(self.eval_data_loader, desc="[eval]"), 1 File "/usr/local/lib/python3.7/dist-packages/tqdm/std.py", line 1167, in iter for obj in iterable: File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/data/ops/iterator_ops.py", line 800, in next return self._next_internal() File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/data/ops/iterator_ops.py", line 786, in _next_internal output_shapes=self._flat_output_shapes) File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/gen_dataset_ops.py", line 2845, in iterator_get_next _ops.raise_from_not_ok_status(e, name) File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/ops.py", line 7107, in raise_from_not_ok_status raise core._status_to_exception(e) from None # pylint: disable=protected-access tensorflow.python.framework.errors_impl.InvalidArgumentError: Paddings must be non-negative: 0 -13480 [[{{function_node cond_4_false_817}}{{node cond_4/Pad}}]] [Op:IteratorGetNext]