Ksponspeech dataset으로 Train 을 진행하려하는데, 아래와 같이 Error 가 나오면서 진행이 더 되지 않습니다.
이 Error를 어떻게 해결해야할지 모르겠어서, 문의 드립니다. Full Error 로그는 하기에 첨부하겠습니다.
File "/home/test/.local/lib/python3.8/site-packages/pytorch_lightning/utilities/data.py", line 33, in has_len
raise ValueError('Dataloader returned 0 length. Please make sure that it returns at least 1 batch')
ValueError: Dataloader returned 0 length. Please make sure that it returns at least 1 batch
제가 사용한 CLI 커멘드는 아래와 같습니다.
python3 ./openspeech_cli/hydra_train.py dataset=ksponspeech dataset.dataset_path=$DATASET_PATH dataset.manifest_file_path=$MANIFEST_FILE_PATH dataset.test_dataset_path=$TEST_DATASET_PATH dataset.test_manifest_dir=$TEST_MANIFEST_DIR tokenizer=kspon_character model=listen_attend_spell audio=melspectrogram lr_scheduler=warmup_reduce_lr_on_plateau trainer=gpu criterion=cross_entropy
DATASET_PATH 아래엔 다음과 같이 폴더로 구성되어있습니다.
(kospeech) ~/openspeech$ ls $DATASET_PATH
KsponSpeech_01 KsponSpeech_02 KsponSpeech_03 KsponSpeech_04 KsponSpeech_05
Full Error log
python3 ./openspeech_cli/hydra_train.py dataset=ksponspeech dataset.dataset_path=$DATASET_PATH dataset.manifest_file_path=$MANIFEST_FILE_PATH dataset.test_dataset_path=$TEST_DATASET_PATH dataset.test_manifest_dir=$TEST_MANIFEST_DIR tokenizer=kspon_character model=listen_attend_spell audio=melspectrogram lr_scheduler=warmup_reduce_lr_on_plateau trainer=gpu criterion=cross_entropy
./openspeech_cli/hydra_train.py:38: UserWarning:
The version_base parameter is not specified.
Please specify a compatability version level, or None.
Will assume defaults for version 1.1
@hydra.main(config_path=os.path.join("..", "openspeech", "configs"), config_name="train")
['./openspeech_cli/hydra_train.py', 'dataset=ksponspeech', 'dataset.dataset_path=/home/test/data/ksponSpeech/train_set/KsponSpeech/', 'dataset.manifest_file_path=/home/test/data/ksponSpeech/manifest/train.trn', 'dataset.test_dataset_path=/home/test/data/ksponSpeech/validation_set/eval_clean', 'dataset.test_manifest_dir=/home/test/data/ksponSpeech/manifest/eval_clean.trn', 'tokenizer=kspon_character', 'model=listen_attend_spell', 'audio=melspectrogram', 'lr_scheduler=warmup_reduce_lr_on_plateau', 'trainer=gpu', 'criterion=cross_entropy']
/home/test/anaconda3/envs/kospeech/lib/python3.8/site-packages/hydra/core/default_element.py:124: UserWarning: In 'train': Usage of deprecated keyword in package header '# @package group'.
See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/changes_to_package_header for more information
deprecation_warning(
/home/test/anaconda3/envs/kospeech/lib/python3.8/site-packages/hydra/_internal/hydra.py:119: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default.
See https://hydra.cc/docs/1.2/upgrades/1.1_to_1.2/changes_to_job_working_dir/ for more information.
ret = run_job(
audio:
name: melspectrogram
sample_rate: 16000
frame_length: 20.0
frame_shift: 10.0
del_silence: false
num_mels: 80
apply_spec_augment: true
apply_noise_augment: false
apply_time_stretch_augment: false
apply_joining_augment: false
augment:
apply_spec_augment: false
apply_noise_augment: false
apply_joining_augment: false
apply_time_stretch_augment: false
freq_mask_para: 27
freq_mask_num: 2
time_mask_num: 4
noise_dataset_dir: None
noise_level: 0.7
time_stretch_min_rate: 0.7
time_stretch_max_rate: 1.4
dataset:
dataset: ksponspeech
dataset_path: /home/test/data/ksponSpeech/train_set/KsponSpeech/
test_dataset_path: /home/test/data/ksponSpeech/validation_set/eval_clean
manifest_file_path: /home/test/data/ksponSpeech/manifest/train.trn
test_manifest_dir: /home/test/data/ksponSpeech/manifest/eval_clean.trn
preprocess_mode: phonetic
criterion:
criterion_name: cross_entropy
reduction: mean
lr_scheduler:
lr: 0.0001
scheduler_name: warmup_reduce_lr_on_plateau
lr_patience: 1
lr_factor: 0.3
peak_lr: 0.0001
init_lr: 1.0e-10
warmup_steps: 4000
model:
model_name: listen_attend_spell
num_encoder_layers: 3
num_decoder_layers: 2
hidden_state_dim: 512
encoder_dropout_p: 0.3
encoder_bidirectional: true
rnn_type: lstm
joint_ctc_attention: false
max_length: 128
num_attention_heads: 1
decoder_dropout_p: 0.2
decoder_attn_mechanism: dot
teacher_forcing_ratio: 1.0
optimizer: adam
trainer:
seed: 1
accelerator: dp
accumulate_grad_batches: 1
num_workers: 4
batch_size: 32
check_val_every_n_epoch: 1
gradient_clip_val: 5.0
logger: wandb
max_epochs: 20
save_checkpoint_n_steps: 10000
auto_scale_batch_size: binsearch
sampler: else
name: gpu
device: gpu
use_cuda: true
auto_select_gpus: true
tokenizer:
sos_token:
eos_token:
pad_token:
blank_token:
encoding: utf-8
unit: kspon_character
vocab_path: ../../../aihub_labels.csv
[2023-05-22 11:30:50,008][openspeech.utils][INFO] - Operating System : Linux 5.15.0-69-generic
[2023-05-22 11:30:50,008][openspeech.utils][INFO] - Processor : x86_64
[2023-05-22 11:30:50,008][openspeech.utils][INFO] - device : NVIDIA GeForce RTX 3080
[2023-05-22 11:30:50,008][openspeech.utils][INFO] - CUDA is available : True
[2023-05-22 11:30:50,008][openspeech.utils][INFO] - CUDA version : 11.0
[2023-05-22 11:30:50,008][openspeech.utils][INFO] - PyTorch version : 1.7.1+cu110
GPU available: True, used: True
[2023-05-22 11:30:52,788][lightning][INFO] - GPU available: True, used: True
TPU available: None, using: 0 TPU cores
[2023-05-22 11:30:52,788][lightning][INFO] - TPU available: None, using: 0 TPU cores
[2023-05-22 11:30:52,789][pytorch_lightning.accelerators.gpu][INFO] - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
wandb: (1) Create a W&B account
wandb: (2) Use an existing W&B account
wandb: (3) Don't visualize my results
wandb: Enter your choice: 3
wandb: You chose "Don't visualize my results"
wandb: WARNING resume will be ignored since W&B syncing is set to offline. Starting a new run with run id az94if5r.
wandb: Tracking run with wandb version 0.15.3
wandb: W&B syncing is set to offline in this directory.
wandb: Run wandb online or set WANDB_MODE=online to enable cloud syncing.
38.0 M Trainable params
0 Non-trainable params
38.0 M Total params
152.088 Total estimated model params size (MB)
[2023-05-22 11:30:55,709][lightning][INFO] -
| Name | Type | Params
38.0 M Trainable params
0 Non-trainable params
38.0 M Total params
152.088 Total estimated model params size (MB)
Error executing job with overrides: ['dataset=ksponspeech', 'dataset.dataset_path=/home/test/data/ksponSpeech/train_set/KsponSpeech/', 'dataset.manifest_file_path=/home/test/data/ksponSpeech/manifest/train.trn', 'dataset.test_dataset_path=/home/test/data/ksponSpeech/validation_set/eval_clean', 'dataset.test_manifest_dir=/home/test/data/ksponSpeech/manifest/eval_clean.trn', 'tokenizer=kspon_character', 'model=listen_attend_spell', 'audio=melspectrogram', 'lr_scheduler=warmup_reduce_lr_on_plateau', 'trainer=gpu', 'criterion=cross_entropy']
Traceback (most recent call last):
File "./openspeech_cli/hydra_train.py", line 54, in hydra_main
trainer.fit(model, data_module)
File "/home/test/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 513, in fit
self.dispatch()
File "/home/test/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 553, in dispatch
self.accelerator.start_training(self)
File "/home/test/.local/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 74, in start_training
self.training_type_plugin.start_training(trainer)
File "/home/test/.local/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 111, in start_training
self._results = trainer.run_train()
File "/home/test/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 614, in run_train
self.run_sanity_check(self.lightning_module)
File "/home/test/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 853, in run_sanity_check
self.reset_val_dataloader(ref_model)
File "/home/test/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/data_loading.py", line 364, in reset_val_dataloader
self.num_val_batches, self.val_dataloaders = self._reset_eval_dataloader(model, 'val')
File "/home/test/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/data_loading.py", line 325, in _reset_eval_dataloader
num_batches = len(dataloader) if has_len(dataloader) else float('inf')
File "/home/test/.local/lib/python3.8/site-packages/pytorch_lightning/utilities/data.py", line 33, in has_len
raise ValueError('Dataloader returned 0 length. Please make sure that it returns at least 1 batch')
ValueError: Dataloader returned 0 length. Please make sure that it returns at least 1 batch
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
wandb: Waiting for W&B process to finish... (failed 1).
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /home/test/openspeech/outputs/2023-05-22/11-30-49/wandb/offline-run-20230522_113054-az94if5r
wandb: Find logs at: ./wandb/offline-run-20230522_113054-az94if5r/logs
❓ Questions & Help
Ksponspeech dataset으로 Train 을 진행하려하는데, 아래와 같이 Error 가 나오면서 진행이 더 되지 않습니다. 이 Error를 어떻게 해결해야할지 모르겠어서, 문의 드립니다. Full Error 로그는 하기에 첨부하겠습니다.
File "/home/test/.local/lib/python3.8/site-packages/pytorch_lightning/utilities/data.py", line 33, in has_len raise ValueError('
Dataloaderreturned 0 length. Please make sure that it returns at least 1 batch') ValueError:Dataloaderreturned 0 length. Please make sure that it returns at least 1 batch제가 사용한 CLI 커멘드는 아래와 같습니다. python3 ./openspeech_cli/hydra_train.py dataset=ksponspeech dataset.dataset_path=$DATASET_PATH dataset.manifest_file_path=$MANIFEST_FILE_PATH dataset.test_dataset_path=$TEST_DATASET_PATH dataset.test_manifest_dir=$TEST_MANIFEST_DIR tokenizer=kspon_character model=listen_attend_spell audio=melspectrogram lr_scheduler=warmup_reduce_lr_on_plateau trainer=gpu criterion=cross_entropy
DATASET_PATH 아래엔 다음과 같이 폴더로 구성되어있습니다. (kospeech) ~/openspeech$ ls $DATASET_PATH KsponSpeech_01 KsponSpeech_02 KsponSpeech_03 KsponSpeech_04 KsponSpeech_05
Full Error log
python3 ./openspeech_cli/hydra_train.py dataset=ksponspeech dataset.dataset_path=$DATASET_PATH dataset.manifest_file_path=$MANIFEST_FILE_PATH dataset.test_dataset_path=$TEST_DATASET_PATH dataset.test_manifest_dir=$TEST_MANIFEST_DIR tokenizer=kspon_character model=listen_attend_spell audio=melspectrogram lr_scheduler=warmup_reduce_lr_on_plateau trainer=gpu criterion=cross_entropy ./openspeech_cli/hydra_train.py:38: UserWarning: The version_base parameter is not specified. Please specify a compatability version level, or None. Will assume defaults for version 1.1 @hydra.main(config_path=os.path.join("..", "openspeech", "configs"), config_name="train") ['./openspeech_cli/hydra_train.py', 'dataset=ksponspeech', 'dataset.dataset_path=/home/test/data/ksponSpeech/train_set/KsponSpeech/', 'dataset.manifest_file_path=/home/test/data/ksponSpeech/manifest/train.trn', 'dataset.test_dataset_path=/home/test/data/ksponSpeech/validation_set/eval_clean', 'dataset.test_manifest_dir=/home/test/data/ksponSpeech/manifest/eval_clean.trn', 'tokenizer=kspon_character', 'model=listen_attend_spell', 'audio=melspectrogram', 'lr_scheduler=warmup_reduce_lr_on_plateau', 'trainer=gpu', 'criterion=cross_entropy'] /home/test/anaconda3/envs/kospeech/lib/python3.8/site-packages/hydra/core/default_element.py:124: UserWarning: In 'train': Usage of deprecated keyword in package header '# @package group'. See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/changes_to_package_header for more information deprecation_warning( /home/test/anaconda3/envs/kospeech/lib/python3.8/site-packages/hydra/_internal/hydra.py:119: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default. See https://hydra.cc/docs/1.2/upgrades/1.1_to_1.2/changes_to_job_working_dir/ for more information. ret = run_job( audio: name: melspectrogram sample_rate: 16000 frame_length: 20.0 frame_shift: 10.0 del_silence: false num_mels: 80 apply_spec_augment: true apply_noise_augment: false apply_time_stretch_augment: false apply_joining_augment: false augment: apply_spec_augment: false apply_noise_augment: false apply_joining_augment: false apply_time_stretch_augment: false freq_mask_para: 27 freq_mask_num: 2 time_mask_num: 4 noise_dataset_dir: None noise_level: 0.7 time_stretch_min_rate: 0.7 time_stretch_max_rate: 1.4 dataset: dataset: ksponspeech dataset_path: /home/test/data/ksponSpeech/train_set/KsponSpeech/ test_dataset_path: /home/test/data/ksponSpeech/validation_set/eval_clean manifest_file_path: /home/test/data/ksponSpeech/manifest/train.trn test_manifest_dir: /home/test/data/ksponSpeech/manifest/eval_clean.trn preprocess_mode: phonetic criterion: criterion_name: cross_entropy reduction: mean lr_scheduler: lr: 0.0001 scheduler_name: warmup_reduce_lr_on_plateau lr_patience: 1 lr_factor: 0.3 peak_lr: 0.0001 init_lr: 1.0e-10 warmup_steps: 4000 model: model_name: listen_attend_spell num_encoder_layers: 3 num_decoder_layers: 2 hidden_state_dim: 512 encoder_dropout_p: 0.3 encoder_bidirectional: true rnn_type: lstm joint_ctc_attention: false max_length: 128 num_attention_heads: 1 decoder_dropout_p: 0.2 decoder_attn_mechanism: dot teacher_forcing_ratio: 1.0 optimizer: adam trainer: seed: 1 accelerator: dp accumulate_grad_batches: 1 num_workers: 4 batch_size: 32 check_val_every_n_epoch: 1 gradient_clip_val: 5.0 logger: wandb max_epochs: 20 save_checkpoint_n_steps: 10000 auto_scale_batch_size: binsearch sampler: else name: gpu device: gpu use_cuda: true auto_select_gpus: true tokenizer: sos_token:
eos_token:
pad_token:
blank_token:
encoding: utf-8
unit: kspon_character
vocab_path: ../../../aihub_labels.csv
[2023-05-22 11:30:49,991][lightning][INFO] - audio: name: melspectrogram sample_rate: 16000 frame_length: 20.0 frame_shift: 10.0 del_silence: false num_mels: 80 apply_spec_augment: true apply_noise_augment: false apply_time_stretch_augment: false apply_joining_augment: false augment: apply_spec_augment: false apply_noise_augment: false apply_joining_augment: false apply_time_stretch_augment: false freq_mask_para: 27 freq_mask_num: 2 time_mask_num: 4 noise_dataset_dir: None noise_level: 0.7 time_stretch_min_rate: 0.7 time_stretch_max_rate: 1.4 dataset: dataset: ksponspeech dataset_path: /home/test/data/ksponSpeech/train_set/KsponSpeech/ test_dataset_path: /home/test/data/ksponSpeech/validation_set/eval_clean manifest_file_path: /home/test/data/ksponSpeech/manifest/train.trn test_manifest_dir: /home/test/data/ksponSpeech/manifest/eval_clean.trn preprocess_mode: phonetic criterion: criterion_name: cross_entropy reduction: mean lr_scheduler: lr: 0.0001 scheduler_name: warmup_reduce_lr_on_plateau lr_patience: 1 lr_factor: 0.3 peak_lr: 0.0001 init_lr: 1.0e-10 warmup_steps: 4000 model: model_name: listen_attend_spell num_encoder_layers: 3 num_decoder_layers: 2 hidden_state_dim: 512 encoder_dropout_p: 0.3 encoder_bidirectional: true rnn_type: lstm joint_ctc_attention: false max_length: 128 num_attention_heads: 1 decoder_dropout_p: 0.2 decoder_attn_mechanism: dot teacher_forcing_ratio: 1.0 optimizer: adam trainer: seed: 1 accelerator: dp accumulate_grad_batches: 1 num_workers: 4 batch_size: 32 check_val_every_n_epoch: 1 gradient_clip_val: 5.0 logger: wandb max_epochs: 20 save_checkpoint_n_steps: 10000 auto_scale_batch_size: binsearch sampler: else name: gpu device: gpu use_cuda: true auto_select_gpus: true tokenizer: sos_token:
eos_token:
pad_token:
blank_token:
encoding: utf-8
unit: kspon_character
vocab_path: ../../../aihub_labels.csv
Global seed set to 1 [2023-05-22 11:30:49,991][lightning][INFO] - Global seed set to 1 [2023-05-22 11:30:49,994][openspeech.utils][INFO] - audio: name: melspectrogram sample_rate: 16000 frame_length: 20.0 frame_shift: 10.0 del_silence: false num_mels: 80 apply_spec_augment: true apply_noise_augment: false apply_time_stretch_augment: false apply_joining_augment: false augment: apply_spec_augment: false apply_noise_augment: false apply_joining_augment: false apply_time_stretch_augment: false freq_mask_para: 27 freq_mask_num: 2 time_mask_num: 4 noise_dataset_dir: None noise_level: 0.7 time_stretch_min_rate: 0.7 time_stretch_max_rate: 1.4 dataset: dataset: ksponspeech dataset_path: /home/test/data/ksponSpeech/train_set/KsponSpeech/ test_dataset_path: /home/test/data/ksponSpeech/validation_set/eval_clean manifest_file_path: /home/test/data/ksponSpeech/manifest/train.trn test_manifest_dir: /home/test/data/ksponSpeech/manifest/eval_clean.trn preprocess_mode: phonetic criterion: criterion_name: cross_entropy reduction: mean lr_scheduler: lr: 0.0001 scheduler_name: warmup_reduce_lr_on_plateau lr_patience: 1 lr_factor: 0.3 peak_lr: 0.0001 init_lr: 1.0e-10 warmup_steps: 4000 model: model_name: listen_attend_spell num_encoder_layers: 3 num_decoder_layers: 2 hidden_state_dim: 512 encoder_dropout_p: 0.3 encoder_bidirectional: true rnn_type: lstm joint_ctc_attention: false max_length: 128 num_attention_heads: 1 decoder_dropout_p: 0.2 decoder_attn_mechanism: dot teacher_forcing_ratio: 1.0 optimizer: adam trainer: seed: 1 accelerator: dp accumulate_grad_batches: 1 num_workers: 4 batch_size: 32 check_val_every_n_epoch: 1 gradient_clip_val: 5.0 logger: wandb max_epochs: 20 save_checkpoint_n_steps: 10000 auto_scale_batch_size: binsearch sampler: else name: gpu device: gpu use_cuda: true auto_select_gpus: true tokenizer: sos_token:
eos_token:
pad_token:
blank_token:
encoding: utf-8
unit: kspon_character
vocab_path: ../../../aihub_labels.csv
[2023-05-22 11:30:50,008][openspeech.utils][INFO] - Operating System : Linux 5.15.0-69-generic [2023-05-22 11:30:50,008][openspeech.utils][INFO] - Processor : x86_64 [2023-05-22 11:30:50,008][openspeech.utils][INFO] - device : NVIDIA GeForce RTX 3080 [2023-05-22 11:30:50,008][openspeech.utils][INFO] - CUDA is available : True [2023-05-22 11:30:50,008][openspeech.utils][INFO] - CUDA version : 11.0 [2023-05-22 11:30:50,008][openspeech.utils][INFO] - PyTorch version : 1.7.1+cu110 GPU available: True, used: True [2023-05-22 11:30:52,788][lightning][INFO] - GPU available: True, used: True TPU available: None, using: 0 TPU cores [2023-05-22 11:30:52,788][lightning][INFO] - TPU available: None, using: 0 TPU cores [2023-05-22 11:30:52,789][pytorch_lightning.accelerators.gpu][INFO] - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] wandb: (1) Create a W&B account wandb: (2) Use an existing W&B account wandb: (3) Don't visualize my results wandb: Enter your choice: 3 wandb: You chose "Don't visualize my results" wandb: WARNING
resumewill be ignored since W&B syncing is set tooffline. Starting a new run with run id az94if5r. wandb: Tracking run with wandb version 0.15.3 wandb: W&B syncing is set toofflinein this directory.wandb: Run
wandb onlineor set WANDB_MODE=online to enable cloud syncing.| Name | Type | Params
0 | criterion | CrossEntropyLoss | 0
1 | encoder | LSTMEncoder | 15.0 M 2 | decoder | LSTMAttentionDecoder | 23.0 M
38.0 M Trainable params 0 Non-trainable params 38.0 M Total params 152.088 Total estimated model params size (MB) [2023-05-22 11:30:55,709][lightning][INFO] - | Name | Type | Params
0 | criterion | CrossEntropyLoss | 0
1 | encoder | LSTMEncoder | 15.0 M 2 | decoder | LSTMAttentionDecoder | 23.0 M
38.0 M Trainable params 0 Non-trainable params 38.0 M Total params 152.088 Total estimated model params size (MB) Error executing job with overrides: ['dataset=ksponspeech', 'dataset.dataset_path=/home/test/data/ksponSpeech/train_set/KsponSpeech/', 'dataset.manifest_file_path=/home/test/data/ksponSpeech/manifest/train.trn', 'dataset.test_dataset_path=/home/test/data/ksponSpeech/validation_set/eval_clean', 'dataset.test_manifest_dir=/home/test/data/ksponSpeech/manifest/eval_clean.trn', 'tokenizer=kspon_character', 'model=listen_attend_spell', 'audio=melspectrogram', 'lr_scheduler=warmup_reduce_lr_on_plateau', 'trainer=gpu', 'criterion=cross_entropy'] Traceback (most recent call last): File "./openspeech_cli/hydra_train.py", line 54, in hydra_main trainer.fit(model, data_module) File "/home/test/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 513, in fit self.dispatch() File "/home/test/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 553, in dispatch self.accelerator.start_training(self) File "/home/test/.local/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 74, in start_training self.training_type_plugin.start_training(trainer) File "/home/test/.local/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 111, in start_training self._results = trainer.run_train() File "/home/test/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 614, in run_train self.run_sanity_check(self.lightning_module) File "/home/test/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 853, in run_sanity_check self.reset_val_dataloader(ref_model) File "/home/test/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/data_loading.py", line 364, in reset_val_dataloader self.num_val_batches, self.val_dataloaders = self._reset_eval_dataloader(model, 'val') File "/home/test/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/data_loading.py", line 325, in _reset_eval_dataloader num_batches = len(dataloader) if has_len(dataloader) else float('inf') File "/home/test/.local/lib/python3.8/site-packages/pytorch_lightning/utilities/data.py", line 33, in has_len raise ValueError('
Dataloaderreturned 0 length. Please make sure that it returns at least 1 batch') ValueError:Dataloaderreturned 0 length. Please make sure that it returns at least 1 batchSet the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace. wandb: Waiting for W&B process to finish... (failed 1). wandb: You can sync this run to the cloud by running: wandb: wandb sync /home/test/openspeech/outputs/2023-05-22/11-30-49/wandb/offline-run-20230522_113054-az94if5r wandb: Find logs at: ./wandb/offline-run-20230522_113054-az94if5r/logs