Error setting up training

rikabi89 commented 1 year ago

I would like to check this is the correct format for the path to wavs/transcripts :

datasets:
  train:
    name: harry
    n_workers: 8 # idk what this does
    batch_size: 32 # This leads to ~16GB of vram usage on my 3090.
    mode: paired_voice_audio
    path: H:/DL-Art-School/experiments/harry.zip
    fetcher_mode: ['lj'] # CHANGEME if your dataset isn't in LJSpeech format
    phase: train
    max_wav_length: 255995
    max_text_length: 200
    sample_rate: 22050
    load_conditioning: True
    num_conditioning_candidates: 2
    conditioning_length: 44000
    use_bpe_tokenizer: True
    load_aligned_codes: False
  val:
    name: harry
    n_workers: 1
    batch_size: 32 # this could be higher probably
    mode: paired_voice_audio
    path: H:/DL-Art-School/experiments/harry.txt

Should it point towards the zip file and text file directly like above?

I check getting this error after running python3 train.py -opt ../experiments/EXAMPLE_gpt.yml :

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ H:\DL-Art-School\codes\train.py:369 in <module>                                                  │
│                                                                                                  │
│   366 │   │   trainer.rank = torch.distributed.get_rank()                                        │
│   367 │   │   torch.cuda.set_device(torch.distributed.get_rank())                                │
│   368 │                                                                                          │
│ ❱ 369 │   trainer.init(args.opt, opt, args.launcher)                                             │
│   370 │   trainer.do_training()                                                                  │
│   371                                                                                            │
│                                                                                                  │
│ H:\DL-Art-School\codes\train.py:130 in init                                                      │
│                                                                                                  │
│   127 │   │   │   │   │   self.logger.info('Total epochs needed: {:d} for iters {:,d}'.format(   │
│   128 │   │   │   │   │   │   self.total_epochs, total_iters))                                   │
│   129 │   │   │   elif phase == 'val':                                                           │
│ ❱ 130 │   │   │   │   self.val_set, collate_fn = create_dataset(dataset_opt, return_collate=Tr   │
│   131 │   │   │   │   self.val_loader = create_dataloader(self.val_set, dataset_opt, opt, None   │
│   132 │   │   │   │   if self.rank <= 0:                                                         │
│   133 │   │   │   │   │   self.logger.info('Number of val images in [{:s}]: {:d}'.format(        │
│                                                                                                  │
│ H:\DL-Art-School\codes\data\__init__.py:107 in create_dataset                                    │
│                                                                                                  │
│   104 │   │   │   collate = C()                                                                  │
│   105 │   else:                                                                                  │
│   106 │   │   raise NotImplementedError('Dataset [{:s}] is not recognized.'.format(mode))        │
│ ❱ 107 │   dataset = D(dataset_opt)                                                               │
│   108 │                                                                                          │
│   109 │   if return_collate:                                                                     │
│   110 │   │   return dataset, collate                                                            │
│                                                                                                  │
│ H:\DL-Art-School\codes\data\audio\paired_voice_audio_dataset.py:156 in __init__                  │
│                                                                                                  │
│   153 │   │   │   │   fetcher_fn = load_voxpopuli                                                │
│   154 │   │   │   else:                                                                          │
│   155 │   │   │   │   raise NotImplementedError()                                                │
│ ❱ 156 │   │   │   self.audiopaths_and_text.extend(fetcher_fn(p, type))                           │
│   157 │   │   self.text_cleaners = hparams.text_cleaners                                         │
│   158 │   │   self.sample_rate = hparams.sample_rate                                             │
│   159 │   │   random.seed(hparams.seed)                                                          │
│                                                                                                  │
│ H:\DL-Art-School\codes\models\audio\tts\tacotron2\taco_utils.py:31 in                            │
│ load_filepaths_and_text_type                                                                     │
│                                                                                                  │
│   28                                                                                             │
│   29 def load_filepaths_and_text_type(filename, type, split="|"):                                │
│   30 │   with open(filename, encoding='utf-8') as f:                                             │
│ ❱ 31 │   │   filepaths_and_text = [list(line.strip().split(split)) + [type] for line in f]       │
│   32 │   │   base = os.path.dirname(filename)                                                    │
│   33 │   │   for j in range(len(filepaths_and_text)):                                            │
│   34 │   │   │   filepaths_and_text[j][0] = os.path.join(base, filepaths_and_text[j][0])         │
│                                                                                                  │
│ H:\DL-Art-School\codes\models\audio\tts\tacotron2\taco_utils.py:31 in <listcomp>                 │
│                                                                                                  │
│   28                                                                                             │
│   29 def load_filepaths_and_text_type(filename, type, split="|"):                                │
│   30 │   with open(filename, encoding='utf-8') as f:                                             │
│ ❱ 31 │   │   filepaths_and_text = [list(line.strip().split(split)) + [type] for line in f]       │
│   32 │   │   base = os.path.dirname(filename)                                                    │
│   33 │   │   for j in range(len(filepaths_and_text)):                                            │
│   34 │   │   │   filepaths_and_text[j][0] = os.path.join(base, filepaths_and_text[j][0])         │
│                                                                                                  │
│ C:\Users\Ali\anaconda3\envs\tts-fast\lib\codecs.py:322 in decode                                 │
│                                                                                                  │
│    319 │   def decode(self, input, final=False):                                                 │
│    320 │   │   # decode input (taking the buffer into account)                                   │
│    321 │   │   data = self.buffer + input                                                        │
│ ❱  322 │   │   (result, consumed) = self._buffer_decode(data, self.errors, final)                │
│    323 │   │   # keep undecoded input until the next call                                        │
│    324 │   │   self.buffer = data[consumed:]                                                     │
│    325 │   │   return result                                                                     │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 10: invalid start byte

The dataset is from a former TalkNet dataset which is setup as LJ speech and so I can't understand the issue here the text file is in UTF8 - Any ideas?

152334H commented 1 year ago

path: H:/DL-Art-School/experiments/harry.zip
hi, the path is supposed to point to an LJSpeech formatted txt file. the dataset folder should look something like this:

/dataset
- train.txt
- val.txt
- wavs/
  - a.wav
  - b.wav
  ...

and the path: entries in the yml file should point to /dataset/train.txt and /dataset/val.txt respectively.

152334H commented 1 year ago

if you only have a single text file listing all the wavs with their transcriptions (instead of a train/val split), you can split the text files very easily:

def split_file(src, train, test, per=0.85):
    import random
    with open(src) as f:
        lines = [l+'\n' for l in f.read().split('\n')] # make sure newline exists universally
    random.shuffle(lines)
    idx = int(len(lines)*per)
    with open(train, 'w') as f: f.write(''.join(lines[:idx]))
    with open(test, 'w') as f: f.write(''.join(lines[idx:]))
split_file("metadata.txt", "train.txt", "val.txt")

rikabi89 commented 1 year ago

Thank you, so I made more progress but now it's saying the .wav files are not found even though they are in the /wavs folder from where the validtion file is

`(tts-fast) H:\DL-Art-School\codes>python train.py -opt ../experiments/EXAMPLE_gpt.yml Disabled distributed training. Path already exists. Rename it to [H:\DL-Art-School\experiments\harry_archived_230217-083402] 23-02-17 08:34:02.108 - INFO: name: harry model: extensibletrainer scale: 1 gpu_ids: [0] start_step: -1 checkpointing_enabled: True fp16: False wandb: False use_tb_logger: True datasets:[ train:[ name: harry n_workers: 8 batch_size: 32 mode: paired_voice_audio path: H:/data/datasets/harry/1.txt fetcher_mode: ['lj'] phase: train max_wav_length: 255995 max_text_length: 200 sample_rate: 22050 load_conditioning: True num_conditioning_candidates: 2 conditioning_length: 44000 use_bpe_tokenizer: True load_aligned_codes: False data_type: img ] val:[ name: harry n_workers: 1 batch_size: 32 mode: paired_voice_audio path: H:/data/datasets/harry/1.txt fetcher_mode: ['lj'] phase: val max_wav_length: 255995 max_text_length: 200 sample_rate: 22050 load_conditioning: True num_conditioning_candidates: 2 conditioning_length: 44000 use_bpe_tokenizer: True load_aligned_codes: False data_type: img ] ] steps:[ gpt_train:[ training: gpt loss_log_buffer: 500 optimizer: adamw optimizer_params:[ lr: 1e-05 weight_decay: 0.01 beta1: 0.9 beta2: 0.96 ] clip_grad_eps: 4 injectors:[ paired_to_mel:[ type: torch_mel_spectrogram mel_norm_file: ../experiments/clips_mel_norms.pth in: wav out: paired_mel ] paired_cond_to_mel:[ type: for_each subtype: torch_mel_spectrogram mel_norm_file: ../experiments/clips_mel_norms.pth in: conditioning out: paired_conditioning_mel ] to_codes:[ type: discrete_token in: paired_mel out: paired_mel_codes dvae_config: ../experiments/train_diffusion_vocoder_22k_level.yml ] paired_fwd_text:[ type: generator generator: gpt in: ['paired_conditioning_mel', 'padded_text', 'text_lengths', 'paired_mel_codes', 'wav_lengths'] out: ['loss_text_ce', 'loss_mel_ce', 'logits'] ] ] losses:[ text_ce:[ type: direct weight: 0.01 key: loss_text_ce ] mel_ce:[ type: direct weight: 1 key: loss_mel_ce ] ] ] ] networks:[ gpt:[ type: generator which_model_G: unified_voice2 kwargs:[ layers: 30 model_dim: 1024 heads: 16 max_text_tokens: 402 max_mel_tokens: 604 max_conditioning_inputs: 2 mel_length_compression: 1024 number_text_tokens: 256 number_mel_codes: 8194 start_mel_token: 8192 stop_mel_token: 8193 start_text_token: 255 train_solo_embeddings: False use_mel_codes_as_input: True checkpointing: True ] ] ] path:[ pretrain_model_gpt: ../experiments/autoregressive.pth strict_load: True root: H:\DL-Art-School experiments_root: H:\DL-Art-School\experiments\harry models: H:\DL-Art-School\experiments\harry\models training_state: H:\DL-Art-School\experiments\harry\training_state log: H:\DL-Art-School\experiments\harry val_images: H:\DL-Art-School\experiments\harry\val_images ] train:[ niter: 50000 warmup_iter: -1 mega_batch_factor: 4 val_freq: 500 default_lr_scheme: MultiStepLR gen_lr_steps: [500, 1000, 1400, 1800] lr_gamma: 0.5 ] eval:[ output_state: gen injectors:[ gen_inj_eval:[ type: generator generator: generator in: hq out: ['gen', 'codebook_commitment_loss'] ] ] ] logger:[ print_freq: 100 save_checkpoint_freq: 500 visuals: ['gen', 'mel'] visual_debug_rate: 500 is_mel_spectrogram: True ] is_train: True dist: False

23-02-17 08:34:02.112 - INFO: Random seed: 9087 23-02-17 08:34:02.978 - INFO: Number of training data elements: 1,810, iters: 57 23-02-17 08:34:02.978 - INFO: Total epochs needed: 878 for iters 50,000 23-02-17 08:34:02.988 - INFO: Number of val images in [harry]: 1810 C:\Users\Ali\anaconda3\envs\tts-fast\lib\site-packages\transformers\configuration_utils.py:375: UserWarning: Passing gradient_checkpointing to a config initialization is deprecated and will be removed in v5 Transformers. Using model.gradient_checkpointing_enable() instead, or if you are using the Trainer API, pass gradient_checkpointing=True in your TrainingArguments. warnings.warn( Loading from ../experiments/dvae.pth 23-02-17 08:34:10.852 - INFO: Network gpt structure: DataParallel, with parameters: 421,526,786 23-02-17 08:34:10.852 - INFO: UnifiedVoice( (conditioning_encoder): ConditioningEncoder( (init): Conv1d(80, 1024, kernel_size=(1,), stride=(1,)) (attn): Sequential( (0): AttentionBlock( (norm): GroupNorm32(32, 1024, eps=1e-05, affine=True) (qkv): Conv1d(1024, 3072, kernel_size=(1,), stride=(1,)) (attention): QKVAttentionLegacy() (x_proj): Identity() (proj_out): Conv1d(1024, 1024, kernel_size=(1,), stride=(1,)) ) (1): AttentionBlock( (norm): GroupNorm32(32, 1024, eps=1e-05, affine=True) (qkv): Conv1d(1024, 3072, kernel_size=(1,), stride=(1,)) (attention): QKVAttentionLegacy() (x_proj): Identity() (proj_out): Conv1d(1024, 1024, kernel_size=(1,), stride=(1,)) ) (2): AttentionBlock( (norm): GroupNorm32(32, 1024, eps=1e-05, affine=True) (qkv): Conv1d(1024, 3072, kernel_size=(1,), stride=(1,)) (attention): QKVAttentionLegacy() (x_proj): Identity() (proj_out): Conv1d(1024, 1024, kernel_size=(1,), stride=(1,)) ) (3): AttentionBlock( (norm): GroupNorm32(32, 1024, eps=1e-05, affine=True) (qkv): Conv1d(1024, 3072, kernel_size=(1,), stride=(1,)) (attention): QKVAttentionLegacy() (x_proj): Identity() (proj_out): Conv1d(1024, 1024, kernel_size=(1,), stride=(1,)) ) (4): AttentionBlock( (norm): GroupNorm32(32, 1024, eps=1e-05, affine=True) (qkv): Conv1d(1024, 3072, kernel_size=(1,), stride=(1,)) (attention): QKVAttentionLegacy() (x_proj): Identity() (proj_out): Conv1d(1024, 1024, kernel_size=(1,), stride=(1,)) ) (5): AttentionBlock( (norm): GroupNorm32(32, 1024, eps=1e-05, affine=True) (qkv): Conv1d(1024, 3072, kernel_size=(1,), stride=(1,)) (attention): QKVAttentionLegacy() (x_proj): Identity() (proj_out): Conv1d(1024, 1024, kernel_size=(1,), stride=(1,)) ) ) ) (text_embedding): Embedding(256, 1024) (mel_embedding): Embedding(8194, 1024) (gpt): GPT2Model( (drop): Dropout(p=0.1, inplace=False) (h): ModuleList( (0): GPT2Block( (ln_1): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (attn): GPT2Attention( (c_attn): Conv1D() (c_proj): Conv1D() (attn_dropout): Dropout(p=0.1, inplace=False) (resid_dropout): Dropout(p=0.1, inplace=False) ) (ln_2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): GPT2MLP( (c_fc): Conv1D() (c_proj): Conv1D() (act): NewGELUActivation() (dropout): Dropout(p=0.1, inplace=False) ) ) (1): GPT2Block( (ln_1): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (attn): GPT2Attention( (c_attn): Conv1D() (c_proj): Conv1D() (attn_dropout): Dropout(p=0.1, inplace=False) (resid_dropout): Dropout(p=0.1, inplace=False) ) (ln_2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): GPT2MLP( (c_fc): Conv1D() (c_proj): Conv1D() (act): NewGELUActivation() (dropout): Dropout(p=0.1, inplace=False) ) ) (2): GPT2Block( (ln_1): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (attn): GPT2Attention( (c_attn): Conv1D() (c_proj): Conv1D() (attn_dropout): Dropout(p=0.1, inplace=False) (resid_dropout): Dropout(p=0.1, inplace=False) ) (ln_2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): GPT2MLP( (c_fc): Conv1D() (c_proj): Conv1D() (act): NewGELUActivation() (dropout): Dropout(p=0.1, inplace=False) ) ) (3): GPT2Block( (ln_1): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (attn): GPT2Attention( (c_attn): Conv1D() (c_proj): Conv1D() (attn_dropout): Dropout(p=0.1, inplace=False) (resid_dropout): Dropout(p=0.1, inplace=False) ) (ln_2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): GPT2MLP( (c_fc): Conv1D() (c_proj): Conv1D() (act): NewGELUActivation() (dropout): Dropout(p=0.1, inplace=False) ) ) (4): GPT2Block( (ln_1): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (attn): GPT2Attention( (c_attn): Conv1D() (c_proj): Conv1D() (attn_dropout): Dropout(p=0.1, inplace=False) (resid_dropout): Dropout(p=0.1, inplace=False) ) (ln_2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): GPT2MLP( (c_fc): Conv1D() (c_proj): Conv1D() (act): NewGELUActivation() (dropout): Dropout(p=0.1, inplace=False) ) ) (5): GPT2Block( (ln_1): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (attn): GPT2Attention( (c_attn): Conv1D() (c_proj): Conv1D() (attn_dropout): Dropout(p=0.1, inplace=False) (resid_dropout): Dropout(p=0.1, inplace=False) ) (ln_2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): GPT2MLP( (c_fc): Conv1D() (c_proj): Conv1D() (act): NewGELUActivation() (dropout): Dropout(p=0.1, inplace=False) ) ) (6): GPT2Block( (ln_1): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (attn): GPT2Attention( (c_attn): Conv1D() (c_proj): Conv1D() (attn_dropout): Dropout(p=0.1, inplace=False) (resid_dropout): Dropout(p=0.1, inplace=False) ) (ln_2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): GPT2MLP( (c_fc): Conv1D() (c_proj): Conv1D() (act): NewGELUActivation() (dropout): Dropout(p=0.1, inplace=False) ) ) (7): GPT2Block( (ln_1): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (attn): GPT2Attention( (c_attn): Conv1D() (c_proj): Conv1D() (attn_dropout): Dropout(p=0.1, inplace=False) (resid_dropout): Dropout(p=0.1, inplace=False) ) (ln_2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): GPT2MLP( (c_fc): Conv1D() (c_proj): Conv1D() (act): NewGELUActivation() (dropout): Dropout(p=0.1, inplace=False) ) ) (8): GPT2Block( (ln_1): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (attn): GPT2Attention( (c_attn): Conv1D() (c_proj): Conv1D() (attn_dropout): Dropout(p=0.1, inplace=False) (resid_dropout): Dropout(p=0.1, inplace=False) ) (ln_2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): GPT2MLP( (c_fc): Conv1D() (c_proj): Conv1D() (act): NewGELUActivation() (dropout): Dropout(p=0.1, inplace=False) ) ) (9): GPT2Block( (ln_1): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (attn): GPT2Attention( (c_attn): Conv1D() (c_proj): Conv1D() (attn_dropout): Dropout(p=0.1, inplace=False) (resid_dropout): Dropout(p=0.1, inplace=False) ) (ln_2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): GPT2MLP( (c_fc): Conv1D() (c_proj): Conv1D() (act): NewGELUActivation() (dropout): Dropout(p=0.1, inplace=False) ) ) (10): GPT2Block( (ln_1): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (attn): GPT2Attention( (c_attn): Conv1D() (c_proj): Conv1D() (attn_dropout): Dropout(p=0.1, inplace=False) (resid_dropout): Dropout(p=0.1, inplace=False) ) (ln_2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): GPT2MLP( (c_fc): Conv1D() (c_proj): Conv1D() (act): NewGELUActivation() (dropout): Dropout(p=0.1, inplace=False) ) ) (11): GPT2Block( (ln_1): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (attn): GPT2Attention( (c_attn): Conv1D() (c_proj): Conv1D() (attn_dropout): Dropout(p=0.1, inplace=False) (resid_dropout): Dropout(p=0.1, inplace=False) ) (ln_2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): GPT2MLP( (c_fc): Conv1D() (c_proj): Conv1D() (act): NewGELUActivation() (dropout): Dropout(p=0.1, inplace=False) ) ) (12): GPT2Block( (ln_1): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (attn): GPT2Attention( (c_attn): Conv1D() (c_proj): Conv1D() (attn_dropout): Dropout(p=0.1, inplace=False) (resid_dropout): Dropout(p=0.1, inplace=False) ) (ln_2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): GPT2MLP( (c_fc): Conv1D() (c_proj): Conv1D() (act): NewGELUActivation() (dropout): Dropout(p=0.1, inplace=False) ) ) (13): GPT2Block( (ln_1): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (attn): GPT2Attention( (c_attn): Conv1D() (c_proj): Conv1D() (attn_dropout): Dropout(p=0.1, inplace=False) (resid_dropout): Dropout(p=0.1, inplace=False) ) (ln_2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): GPT2MLP( (c_fc): Conv1D() (c_proj): Conv1D() (act): NewGELUActivation() (dropout): Dropout(p=0.1, inplace=False) ) ) (14): GPT2Block( (ln_1): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (attn): GPT2Attention( (c_attn): Conv1D() (c_proj): Conv1D() (attn_dropout): Dropout(p=0.1, inplace=False) (resid_dropout): Dropout(p=0.1, inplace=False) ) (ln_2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): GPT2MLP( (c_fc): Conv1D() (c_proj): Conv1D() (act): NewGELUActivation() (dropout): Dropout(p=0.1, inplace=False) ) ) (15): GPT2Block( (ln_1): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (attn): GPT2Attention( (c_attn): Conv1D() (c_proj): Conv1D() (attn_dropout): Dropout(p=0.1, inplace=False) (resid_dropout): Dropout(p=0.1, inplace=False) ) (ln_2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): GPT2MLP( (c_fc): Conv1D() (c_proj): Conv1D() (act): NewGELUActivation() (dropout): Dropout(p=0.1, inplace=False) ) ) (16): GPT2Block( (ln_1): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (attn): GPT2Attention( (c_attn): Conv1D() (c_proj): Conv1D() (attn_dropout): Dropout(p=0.1, inplace=False) (resid_dropout): Dropout(p=0.1, inplace=False) ) (ln_2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): GPT2MLP( (c_fc): Conv1D() (c_proj): Conv1D() (act): NewGELUActivation() (dropout): Dropout(p=0.1, inplace=False) ) ) (17): GPT2Block( (ln_1): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (attn): GPT2Attention( (c_attn): Conv1D() (c_proj): Conv1D() (attn_dropout): Dropout(p=0.1, inplace=False) (resid_dropout): Dropout(p=0.1, inplace=False) ) (ln_2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): GPT2MLP( (c_fc): Conv1D() (c_proj): Conv1D() (act): NewGELUActivation() (dropout): Dropout(p=0.1, inplace=False) ) ) (18): GPT2Block( (ln_1): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (attn): GPT2Attention( (c_attn): Conv1D() (c_proj): Conv1D() (attn_dropout): Dropout(p=0.1, inplace=False) (resid_dropout): Dropout(p=0.1, inplace=False) ) (ln_2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): GPT2MLP( (c_fc): Conv1D() (c_proj): Conv1D() (act): NewGELUActivation() (dropout): Dropout(p=0.1, inplace=False) ) ) (19): GPT2Block( (ln_1): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (attn): GPT2Attention( (c_attn): Conv1D() (c_proj): Conv1D() (attn_dropout): Dropout(p=0.1, inplace=False) (resid_dropout): Dropout(p=0.1, inplace=False) ) (ln_2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): GPT2MLP( (c_fc): Conv1D() (c_proj): Conv1D() (act): NewGELUActivation() (dropout): Dropout(p=0.1, inplace=False) ) ) (20): GPT2Block( (ln_1): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (attn): GPT2Attention( (c_attn): Conv1D() (c_proj): Conv1D() (attn_dropout): Dropout(p=0.1, inplace=False) (resid_dropout): Dropout(p=0.1, inplace=False) ) (ln_2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): GPT2MLP( (c_fc): Conv1D() (c_proj): Conv1D() (act): NewGELUActivation() (dropout): Dropout(p=0.1, inplace=False) ) ) (21): GPT2Block( (ln_1): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (attn): GPT2Attention( (c_attn): Conv1D() (c_proj): Conv1D() (attn_dropout): Dropout(p=0.1, inplace=False) (resid_dropout): Dropout(p=0.1, inplace=False) ) (ln_2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): GPT2MLP( (c_fc): Conv1D() (c_proj): Conv1D() (act): NewGELUActivation() (dropout): Dropout(p=0.1, inplace=False) ) ) (22): GPT2Block( (ln_1): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (attn): GPT2Attention( (c_attn): Conv1D() (c_proj): Conv1D() (attn_dropout): Dropout(p=0.1, inplace=False) (resid_dropout): Dropout(p=0.1, inplace=False) ) (ln_2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): GPT2MLP( (c_fc): Conv1D() (c_proj): Conv1D() (act): NewGELUActivation() (dropout): Dropout(p=0.1, inplace=False) ) ) (23): GPT2Block( (ln_1): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (attn): GPT2Attention( (c_attn): Conv1D() (c_proj): Conv1D() (attn_dropout): Dropout(p=0.1, inplace=False) (resid_dropout): Dropout(p=0.1, inplace=False) ) (ln_2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): GPT2MLP( (c_fc): Conv1D() (c_proj): Conv1D() (act): NewGELUActivation() (dropout): Dropout(p=0.1, inplace=False) ) ) (24): GPT2Block( (ln_1): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (attn): GPT2Attention( (c_attn): Conv1D() (c_proj): Conv1D() (attn_dropout): Dropout(p=0.1, inplace=False) (resid_dropout): Dropout(p=0.1, inplace=False) ) (ln_2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): GPT2MLP( (c_fc): Conv1D() (c_proj): Conv1D() (act): NewGELUActivation() (dropout): Dropout(p=0.1, inplace=False) ) ) (25): GPT2Block( (ln_1): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (attn): GPT2Attention( (c_attn): Conv1D() (c_proj): Conv1D() (attn_dropout): Dropout(p=0.1, inplace=False) (resid_dropout): Dropout(p=0.1, inplace=False) ) (ln_2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): GPT2MLP( (c_fc): Conv1D() (c_proj): Conv1D() (act): NewGELUActivation() (dropout): Dropout(p=0.1, inplace=False) ) ) (26): GPT2Block( (ln_1): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (attn): GPT2Attention( (c_attn): Conv1D() (c_proj): Conv1D() (attn_dropout): Dropout(p=0.1, inplace=False) (resid_dropout): Dropout(p=0.1, inplace=False) ) (ln_2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): GPT2MLP( (c_fc): Conv1D() (c_proj): Conv1D() (act): NewGELUActivation() (dropout): Dropout(p=0.1, inplace=False) ) ) (27): GPT2Block( (ln_1): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (attn): GPT2Attention( (c_attn): Conv1D() (c_proj): Conv1D() (attn_dropout): Dropout(p=0.1, inplace=False) (resid_dropout): Dropout(p=0.1, inplace=False) ) (ln_2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): GPT2MLP( (c_fc): Conv1D() (c_proj): Conv1D() (act): NewGELUActivation() (dropout): Dropout(p=0.1, inplace=False) ) ) (28): GPT2Block( (ln_1): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (attn): GPT2Attention( (c_attn): Conv1D() (c_proj): Conv1D() (attn_dropout): Dropout(p=0.1, inplace=False) (resid_dropout): Dropout(p=0.1, inplace=False) ) (ln_2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): GPT2MLP( (c_fc): Conv1D() (c_proj): Conv1D() (act): NewGELUActivation() (dropout): Dropout(p=0.1, inplace=False) ) ) (29): GPT2Block( (ln_1): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (attn): GPT2Attention( (c_attn): Conv1D() (c_proj): Conv1D() (attn_dropout): Dropout(p=0.1, inplace=False) (resid_dropout): Dropout(p=0.1, inplace=False) ) (ln_2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): GPT2MLP( (c_fc): Conv1D() (c_proj): Conv1D() (act): NewGELUActivation() (dropout): Dropout(p=0.1, inplace=False) ) ) ) (ln_f): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) ) (mel_pos_embedding): LearnedPositionEmbeddings( (emb): Embedding(608, 1024) ) (text_pos_embedding): LearnedPositionEmbeddings( (emb): Embedding(404, 1024) ) (final_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (text_head): Linear(in_features=1024, out_features=256, bias=True) (mel_head): Linear(in_features=1024, out_features=8194, bias=True) ) 23-02-17 08:34:10.858 - INFO: Loading model for [../experiments/autoregressive.pth] WARNING! Unable to find EMA network! Starting a new EMA from given model parameters. 23-02-17 08:34:11.931 - INFO: Start training from epoch: 0, iter: -1 0%| | 0/56 [00:39<?, ?it/s] ╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮ │ H:\DL-Art-School\codes\train.py:370 in │ │ │ │ 367 │ │ torch.cuda.set_device(torch.distributed.get_rank()) │ │ 368 │ │ │ 369 │ trainer.init(args.opt, opt, args.launcher) │ │ ❱ 370 │ trainer.do_training() │ │ 371 │ │ │ │ H:\DL-Art-School\codes\train.py:324 in do_training │ │ │ │ 321 │ │ │ tq_ldr = tqdm(self.train_loader) if self.rank <= 0 else self.train_loader │ │ 322 │ │ │ │ │ 323 │ │ │ _t = time() │ │ ❱ 324 │ │ │ for train_data in tq_ldr: │ │ 325 │ │ │ │ self.do_step(train_data) │ │ 326 │ │ │ 327 │ def create_training_generator(self, index): │ │ │ │ C:\Users\Ali\anaconda3\envs\tts-fast\lib\site-packages\tqdm\std.py:1195 in iter │ │ │ │ 1192 │ │ time = self._time │ │ 1193 │ │ │ │ 1194 │ │ try: │ │ ❱ 1195 │ │ │ for obj in iterable: │ │ 1196 │ │ │ │ yield obj │ │ 1197 │ │ │ │ # Update and possibly print the progressbar. │ │ 1198 │ │ │ │ # Note: does not call self.update(1) for speed optimisation. │ │ │ │ C:\Users\Ali\anaconda3\envs\tts-fast\lib\site-packages\torch\utils\data\dataloader.py:628 in │ │ next │ │ │ │ 625 │ │ │ if self._sampler_iter is None: │ │ 626 │ │ │ │ # TODO(https://github.com/pytorch/pytorch/issues/76750) │ │ 627 │ │ │ │ self._reset() # type: ignore[call-arg] │ │ ❱ 628 │ │ │ data = self._next_data() │ │ 629 │ │ │ self._num_yielded += 1 │ │ 630 │ │ │ if self._dataset_kind == _DatasetKind.Iterable and \ │ │ 631 │ │ │ │ │ self._IterableDataset_len_called is not None and \ │ │ │ │ C:\Users\Ali\anaconda3\envs\tts-fast\lib\site-packages\torch\utils\data\dataloader.py:1333 in │ │ _next_data │ │ │ │ 1330 │ │ │ │ self._task_info[idx] += (data,) │ │ 1331 │ │ │ else: │ │ 1332 │ │ │ │ del self._task_info[idx] │ │ ❱ 1333 │ │ │ │ return self._process_data(data) │ │ 1334 │ │ │ 1335 │ def _try_put_index(self): │ │ 1336 │ │ assert self._tasks_outstanding < self._prefetch_factor * self._num_workers │ │ │ │ C:\Users\Ali\anaconda3\envs\tts-fast\lib\site-packages\torch\utils\data\dataloader.py:1359 in │ │ _process_data │ │ │ │ 1356 │ │ self._rcvd_idx += 1 │ │ 1357 │ │ self._try_put_index() │ │ 1358 │ │ if isinstance(data, ExceptionWrapper): │ │ ❱ 1359 │ │ │ data.reraise() │ │ 1360 │ │ return data │ │ 1361 │ │ │ 1362 │ def _mark_worker_as_unavailable(self, worker_id, shutdown=False): │ │ │ │ C:\Users\Ali\anaconda3\envs\tts-fast\lib\site-packages\torch_utils.py:543 in reraise │ │ │ │ 540 │ │ │ # If the exception takes multiple arguments, don't try to │ │ 541 │ │ │ # instantiate since we don't know how to │ │ 542 │ │ │ raise RuntimeError(msg) from None │ │ ❱ 543 │ │ raise exception │ │ 544 │ │ 545 │ │ 546 def _get_available_device_type(): │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯ FileNotFoundError: Caught FileNotFoundError in DataLoader worker process 0. Original Traceback (most recent call last): File "H:\DL-Art-School\codes\data\audio\paired_voice_audio_dataset.py", line 194, in getitem tseq, wav, text, path, type = self.get_wav_text_pair(self.audiopaths_and_text[index]) File "H:\DL-Art-School\codes\data\audio\paired_voice_audio_dataset.py", line 178, in get_wav_text_pair wav = load_audio(audiopath, self.sample_rate) File "H:\DL-Art-School\codes\data\audio\unsupervised_audio_dataset.py", line 19, in load_audio audio, lsr = load_wav_to_torch(audiopath) File "H:\DL-Art-School\codes\models\audio\tts\tacotron2\taco_utils.py", line 17, in load_wav_to_torch sampling_rate, data = read(full_path) File "C:\Users\Ali\anaconda3\envs\tts-fast\lib\site-packages\scipy\io\wavfile.py", line 647, in read fid = open(filename, 'rb') FileNotFoundError: [Errno 2] No such file or directory: 'H:/data/datasets/harry\025848270_025857120.wav'