terminal hangs when train starts

[tae0y]

❓ Questions & Help

I want to know whether it needs more configuration,
or just patience .... I'm beginner for torch and speech to text, please share your warm advice.

• environments : mac m1
• command I used

  HYDRA_FULL_ERROR=1 python3 ./openspeech_cli/hydra_train.py \
  dataset=foreignkorean \
  dataset.dataset_path={datasetpath} \
  dataset.manifest_file_path={manifestpath} \
  dataset.test_dataset_path={test_datasetpath} \
  dataset.test_manifest_dir={manifestdir} \
  tokenizer=foreignkorean_character \
  tokenizer.vocab_path={vocabpath} \
  model=listen_attend_spell \
  audio=melspectrogram \
  lr_scheduler=warmup_reduce_lr_on_plateau \
  trainer=cpu \
  criterion=cross_entropy

Details

• in my terminal

  

• in wnadb

  

[tae0y]

terminal is printing like below.... maybe it's running

Epoch 0:   0%| | 2/1080 [39:58<239:22:18, 799.39s/it, loss=7.04, v_num=8rcc, train_loss=7

[upskyy]

@tae0y I think it's very slow because you're training with cpu. I recommend training in a GPU environment.

[tae0y]

@upskyy 감사합니다아아아!! 혹시나 구글코랩 TPU에서 돌릴분들을 위해 환경정보를 공유드립니다. 여전히 느리긴 하지만.. 이제 진행은 되네요!

thankssssss!!! I share environment for those who run this project on colab tpu. It's still slow but it is ongoing.

#프로젝트 경로로 이동해서 의존성을 설치한다
%cd /content/drive/MyDrive/openspeech-clone
!pip install -e .

#다음 이슈를 참고해서 tpu 관련 패키지를 설치한다 https://github.com/googlecolab/colabtools/issues/3646
!pip install pytorch-lightning==1.8
!pip install cloud-tpu-client==0.10 https://storage.googleapis.com/tpu-pytorch/wheels/colab/torch_xla-2.0-cp310-cp310-linux_x86_64.whl
!pip uninstall -y torch torchvision torchaudio torchdata torchtext
!pip install torch torchvision torchaudio torchdata torchtext --extra-index-url https://download.pytorch.org/whl/cu117