search

NVIDIA / tacotron2

Tacotron 2 - PyTorch implementation with faster-than-realtime inference
BSD 3-Clause "New" or "Revised" License
4.98k stars 1.37k forks source link

ValueError: The histogram is empty, please file a bug report. #549

Open Dyongh613 opened 2 years ago

Dyongh613 commented 2 years ago

Traceback (most recent call last): File "train.py", line 290, in args.warm_start, args.n_gpus, args.rank, args.group_name, hparams) File "train.py", line 248, in train hparams.distributed_run, rank) File "train.py", line 146, in validate logger.log_validation(val_loss, model, y, y_pred, iteration) File "D:\项目\tacotron2\logger.py", line 27, in log_validation self.add_histogram(tag, value.data.cpu().numpy(), iteration) File "C:\Users\12604\Anaconda3\envs\pytorch\lib\site-packages\torch\utils\tensorboard\writer.py", line 430, in add_histogram histogram(tag, values, bins, max_bins=max_bins), global_step, walltime) File "C:\Users\12604\Anaconda3\envs\pytorch\lib\site-packages\torch\utils\tensorboard\summary.py", line 300, in histogram hist = make_histogram(values.astype(float), bins, max_bins) File "C:\Users\12604\Anaconda3\envs\pytorch\lib\site-packages\torch\utils\tensorboard\summary.py", line 338, in make_histogram raise ValueError('The histogram is empty, please file a bug report.') ValueError: The histogram is empty, please file a bug report.

Wemmons831 commented 1 year ago

@qw1260497397 did you ever figure this out

bryant0918 commented 1 year ago

I had this problem, it's because I was getting nan or inf values in my grad_norm, training_loss, or validation_loss when plotting.