Closed VladC12 closed 4 years ago
adrianastan
commented
4 years ago I would recommend sticking to the 22k sampling frequency so that you can use the pretrained Waveglow model at inference.
Regarding your error it might be that because of the higher sampling rate some samples might not fit into the GPU memory. So you could at first try to remove some of the longer ones from the training list.
VladC12
commented
4 years ago I would recommend sticking to the 22k sampling frequency so that you can use the pretrained Waveglow model at inference.
Regarding your error it might be that because of the higher sampling rate some samples might not fit into the GPU memory. So you could at first try to remove some of the longer ones from the training list.
I get this error instead now: `python train.py -c config.json -p train_config.output_directory=outdir train_config.output_directory=outdir output_directory=outdir {'train_config': {'output_directory': 'outdir', 'epochs': 10000000, 'learning_rate': 0.0001, 'weight_decay': 1e-06, 'sigma': 1.0, 'iters_per_checkpoint': 5000, 'batch_size': 1, 'seed': 1234, 'checkpoint_path': '', 'ignore_layers': [], 'include_layers': ['speaker', 'encoder', 'embedding'], 'warmstart_checkpoint_path': '', 'with_tensorboard': True, 'fp16_run': False}, 'data_config': {'training_files': 'filelists/libritts_train_clean_100_audiopath_text_sid_shorterthan10s_atleast5min_train_filelist.txt', 'validation_files': 'filelists/libritts_train_clean_100_audiopath_text_sid_atleast5min_val_filelist.txt', 'text_cleaners': ['flowtron_cleaners'], 'p_arpabet': 0.5, 'cmudict_path': 'data/cmudict_dictionary', 'sampling_rate': 22000, 'filter_length': 1024, 'hop_length': 256, 'win_length': 1024, 'mel_fmin': 0.0, 'mel_fmax': 8000.0, 'max_wav_value': 32768.0}, 'dist_config': {'dist_backend': 'nccl', 'dist_url': 'tcp://localhost:54321'}, 'model_config': {'n_speakers': 1, 'n_speaker_dim': 128, 'n_text': 185, 'n_text_dim': 512, 'n_flows': 2, 'n_mel_channels': 80, 'n_attn_channels': 640, 'n_hidden': 1024, 'n_lstm_layers': 2, 'mel_encoder_n_hidden': 512, 'n_components': 0, 'mean_scale': 0.0, 'fixed_gaussian': True, 'dummy_speaker_embedding': False, 'use_gate_layer': True}}
got rank 0 and world size 1 ... Flowtron( (speaker_embedding): Embedding(1, 128) (embedding): Embedding(185, 512) (flows): ModuleList( (0): AR_Step( (conv): Conv1d(1024, 160, kernel_size=(1,), stride=(1,)) (lstm): LSTM(1664, 1024, num_layers=2) (attention_lstm): LSTM(80, 1024) (attention_layer): Attention( (softmax): Softmax(dim=2) (query): LinearNorm( (linear_layer): Linear(in_features=1024, out_features=640, bias=False) ) (key): LinearNorm( (linear_layer): Linear(in_features=640, out_features=640, bias=False) ) (value): LinearNorm( (linear_layer): Linear(in_features=640, out_features=640, bias=False) ) (v): LinearNorm( (linear_layer): Linear(in_features=640, out_features=1, bias=False) ) ) (dense_layer): DenseLayer( (layers): ModuleList( (0): LinearNorm( (linear_layer): Linear(in_features=1024, out_features=1024, bias=True) ) (1): LinearNorm( (linear_layer): Linear(in_features=1024, out_features=1024, bias=True) ) ) ) ) (1): AR_Back_Step( (ar_step): AR_Step( (conv): Conv1d(1024, 160, kernel_size=(1,), stride=(1,)) (lstm): LSTM(1664, 1024, num_layers=2) (attention_lstm): LSTM(80, 1024) (attention_layer): Attention( (softmax): Softmax(dim=2) (query): LinearNorm( (linear_layer): Linear(in_features=1024, out_features=640, bias=False) ) (key): LinearNorm( (linear_layer): Linear(in_features=640, out_features=640, bias=False) ) (value): LinearNorm( (linear_layer): Linear(in_features=640, out_features=640, bias=False) ) (v): LinearNorm( (linear_layer): Linear(in_features=640, out_features=1, bias=False) ) ) (dense_layer): DenseLayer( (layers): ModuleList( (0): LinearNorm( (linear_layer): Linear(in_features=1024, out_features=1024, bias=True) ) (1): LinearNorm( (linear_layer): Linear(in_features=1024, out_features=1024, bias=True) ) ) ) (gate_layer): LinearNorm( (linear_layer): Linear(in_features=1664, out_features=1, bias=True) ) ) ) ) (encoder): Encoder( (convolutions): ModuleList( (0): Sequential( (0): ConvNorm( (conv): Conv1d(512, 512, kernel_size=(5,), stride=(1,), padding=(2,)) ) (1): InstanceNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) ) (1): Sequential( (0): ConvNorm( (conv): Conv1d(512, 512, kernel_size=(5,), stride=(1,), padding=(2,)) ) (1): InstanceNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) ) (2): Sequential( (0): ConvNorm( (conv): Conv1d(512, 512, kernel_size=(5,), stride=(1,), padding=(2,)) ) (1): InstanceNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) ) ) (lstm): LSTM(512, 256, batch_first=True, bidirectional=True) ) ) Number of speakers : 123 output directory outdir Epoch: 0 C:\AI_Research_Project\flowtron\data.py:40: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at ..\torch\csrc\utils\tensor_numpy.cpp:141.) return torch.from_numpy(data).float(), sampling_rate Traceback (most recent call last): File "train.py", line 300, in
train(n_gpus, rank, **train_config) File "train.py", line 217, in train for batch in train_loader: File "C:\Users\vladc\anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 363, in next data = self._next_data() File "C:\Users\vladc\anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 989, in _next_data return self._process_data(data) File "C:\Users\vladc\anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 1014, in _process_data data.reraise() File "C:\Users\vladc\anaconda3\lib\site-packages\torch_utils.py", line 395, in reraise raise self.exc_type(msg) ValueError: Caught ValueError in DataLoader worker process 0. Original Traceback (most recent call last): File "C:\Users\vladc\anaconda3\lib\site-packages\torch\utils\data_utils\worker.py", line 185, in _worker_loop data = fetcher.fetch(index) File "C:\Users\vladc\anaconda3\lib\site-packages\torch\utils\data_utils\fetch.py", line 44, in fetch data = [self.dataset[idx] for idx in possibly_batched_index] File "C:\Users\vladc\anaconda3\lib\site-packages\torch\utils\data_utils\fetch.py", line 44, in data = [self.dataset[idx] for idx in possibly_batched_index] File "C:\AI_Research_Project\flowtron\data.py", line 100, in getitem sampling_rate, self.sampling_rate)) ValueError: 24000 SR doesn't match target 22000 SR`
I should mention memory should not be an issue. I have 1060 with 6gb
adrianastan
commented
4 years ago It's not enough to change the sampling frequency in the config file, you need to resample the LibriTTS data.
1 flow with a batch size of 1 takes around 9GB of vRAM depending on the max length of the input files, hence the padding.
VladC12
commented
4 years ago Thanks you. I switched back to ljs and changed the sampling rate and now is well different but I will open a different issue since this is something different.
WARNING:root:NaN or Inf found in input tensor.
I'm using "train-clean-100" from LibriTTS to train it. I also changed the sampling rate to 24000
` ❯ python train.py -c config.json -p train_config.output_directory=outdir train_config.output_directory=outdir output_directory=outdir {'train_config': {'output_directory': 'outdir', 'epochs': 10000000, 'learning_rate': 0.0001, 'weight_decay': 1e-06, 'sigma': 1.0, 'iters_per_checkpoint': 5000, 'batch_size': 1, 'seed': 1234, 'checkpoint_path': '', 'ignore_layers': [], 'include_layers': ['speaker', 'encoder', 'embedding'], 'warmstart_checkpoint_path': '', 'with_tensorboard': True, 'fp16_run': False}, 'data_config': {'training_files': 'filelists/libritts_train_clean_100_audiopath_text_sid_shorterthan10s_atleast5min_train_filelist.txt', 'validation_files': 'filelists/libritts_train_clean_100_audiopath_text_sid_atleast5min_val_filelist.txt', 'text_cleaners': ['flowtron_cleaners'], 'p_arpabet': 0.5, 'cmudict_path': 'data/cmudict_dictionary', 'sampling_rate': 24000, 'filter_length': 1024, 'hop_length': 256, 'win_length': 1024, 'mel_fmin': 0.0, 'mel_fmax': 8000.0, 'max_wav_value': 32768.0}, 'dist_config': {'dist_backend': 'nccl', 'dist_url': 'tcp://localhost:54321'}, 'model_config': {'n_speakers': 1, 'n_speaker_dim': 128, 'n_text': 185, 'n_text_dim': 512, 'n_flows': 2, 'n_mel_channels': 80, 'n_attn_channels': 640, 'n_hidden': 1024, 'n_lstm_layers': 2, 'mel_encoder_n_hidden': 512, 'n_components': 0, 'mean_scale': 0.0, 'fixed_gaussian': True, 'dummy_speaker_embedding': False, 'use_gate_layer': True}}