A scalable generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI (Automatic Speech Recognition and Text-to-Speech)
!bash start_train_ctc.sh
[NeMo W 2024-04-05 16:16:10 nemo_logging:349] /usr/local/lib/python3.10/dist-packages/hydra/_internal/hydra.py:119: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default.
See https://hydra.cc/docs/1.2/upgrades/1.1_to_1.2/changes_to_job_working_dir/ for more information.
ret = run_job(
Error executing job with overrides: ['model.train_ds.manifest_filepath=/content/drive/MyDrive/drive-download-20240405T032246Z-001/train.json', 'model.validation_ds.manifest_filepath=/content/drive/MyDrive/drive-download-20240405T032246Z-001/validated.json', 'model.test_ds.manifest_filepath=/content/drive/MyDrive/drive-download-20240405T032246Z-001/test.json', 'model.tokenizer.dir=/content/drive/MyDrive/1T', 'model.tokenizer.type=bpe', 'trainer.devices=-1', 'trainer.accelerator=gpu', 'trainer.strategy=ddp', 'exp_manager.create_wandb_logger=True', 'exp_manager.wandb_logger_kwargs.name=conformer_large_filtered_english', 'exp_manager.wandb_logger_kwargs.project=nemo_conformer', '+init_from_pretrained_model=/root/ekstep/nemo_exp/vakyansh-nemo-experimentation/scripts/finetuning/nemo_experiments/Conformer-CTC-BPE-Large/2022-06-01_19-16-48/checkpoints/Conformer-CTC-BPE-Large.nemo']
Traceback (most recent call last):
File "/content/vakyansh-nemo-experimentation/scripts/finetuning/../../src/finetuning/speech_to_text_ctc_bpe.py", line 101, in main
trainer = pl.Trainer(cfg.trainer)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/utilities/argparse.py", line 70, in insert_env_defaults
return fn(self, kwargs)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 418, in init
self.fit_loop.epoch_loop = _TrainingEpochLoop(self, min_steps=min_steps, max_steps=max_steps)
File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/training_epoch_loop.py", line 59, in init
if max_steps < -1:
TypeError: '<' not supported between instances of 'NoneType' and 'int'
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
!bash start_train_ctc.sh [NeMo W 2024-04-05 16:16:10 nemo_logging:349] /usr/local/lib/python3.10/dist-packages/hydra/_internal/hydra.py:119: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default. See https://hydra.cc/docs/1.2/upgrades/1.1_to_1.2/changes_to_job_working_dir/ for more information. ret = run_job(
[NeMo I 2024-04-05 16:16:10 speech_to_text_ctc_bpe:99] Hydra config: name: Conformer-CTC-BPE-Large model: sample_rate: 16000 log_prediction: true ctc_reduction: mean_batch skip_nan_grad: false train_ds: manifest_filepath: /content/drive/MyDrive/drive-download-20240405T032246Z-001/train.json sample_rate: ${model.sample_rate} batch_size: 24 shuffle: true num_workers: 16 pin_memory: true use_start_end_token: false trim_silence: false max_duration: 30 min_duration: 0.1 is_tarred: false tarred_audio_filepaths: null shuffle_n: 2048 bucketing_strategy: synced_randomized bucketing_batch_size: 8 validation_ds: manifest_filepath: /content/drive/MyDrive/drive-download-20240405T032246Z-001/validated.json sample_rate: ${model.sample_rate} batch_size: 1 max_duration: 30 shuffle: false num_workers: 8 pin_memory: true use_start_end_token: false test_ds: manifest_filepath: /content/drive/MyDrive/drive-download-20240405T032246Z-001/test.json sample_rate: ${model.sample_rate} batch_size: 4 shuffle: false num_workers: 8 pin_memory: true use_start_end_token: false tokenizer: dir: /content/drive/MyDrive/1T type: bpe preprocessor: target: nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor sample_rate: ${model.sample_rate} normalize: per_feature window_size: 0.025 window_stride: 0.01 window: hann features: 80 n_fft: 512 log: true frame_splicing: 1 dither: 1.0e-05 pad_to: 0 pad_value: 0.0 spec_augment: target: nemo.collections.asr.modules.SpectrogramAugmentation freq_masks: 2 time_masks: 5 freq_width: 27 time_width: 0.05 encoder: target: nemo.collections.asr.modules.ConformerEncoder feat_in: ${model.preprocessor.features} feat_out: -1 n_layers: 17 d_model: 512 subsampling: striding subsampling_factor: 4 subsampling_conv_channels: -1 ff_expansion_factor: 4 self_attention_model: rel_pos n_heads: 8 att_context_size:
Error executing job with overrides: ['model.train_ds.manifest_filepath=/content/drive/MyDrive/drive-download-20240405T032246Z-001/train.json', 'model.validation_ds.manifest_filepath=/content/drive/MyDrive/drive-download-20240405T032246Z-001/validated.json', 'model.test_ds.manifest_filepath=/content/drive/MyDrive/drive-download-20240405T032246Z-001/test.json', 'model.tokenizer.dir=/content/drive/MyDrive/1T', 'model.tokenizer.type=bpe', 'trainer.devices=-1', 'trainer.accelerator=gpu', 'trainer.strategy=ddp', 'exp_manager.create_wandb_logger=True', 'exp_manager.wandb_logger_kwargs.name=conformer_large_filtered_english', 'exp_manager.wandb_logger_kwargs.project=nemo_conformer', '+init_from_pretrained_model=/root/ekstep/nemo_exp/vakyansh-nemo-experimentation/scripts/finetuning/nemo_experiments/Conformer-CTC-BPE-Large/2022-06-01_19-16-48/checkpoints/Conformer-CTC-BPE-Large.nemo'] Traceback (most recent call last): File "/content/vakyansh-nemo-experimentation/scripts/finetuning/../../src/finetuning/speech_to_text_ctc_bpe.py", line 101, in main trainer = pl.Trainer(cfg.trainer) File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/utilities/argparse.py", line 70, in insert_env_defaults return fn(self, kwargs) File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 418, in init self.fit_loop.epoch_loop = _TrainingEpochLoop(self, min_steps=min_steps, max_steps=max_steps) File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/training_epoch_loop.py", line 59, in init if max_steps < -1: TypeError: '<' not supported between instances of 'NoneType' and 'int'
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.