search

SeanNaren / deepspeech.torch

Speech Recognition using DeepSpeech2 network and the CTC activation function.
MIT License
259 stars 73 forks source link

hang in data preparation (Librispeech) #52

Closed ghost closed 7 years ago

ghost commented 8 years ago

I faced a problem in the following line (with Ubuntu 16.04, 4.4.0-38-generic, Lua 5.1.5,):

$ th FormatLibriSpeech.lua -rootPath LibriSpeech -newPath libri_dataset -threads 8
....................................... 34/2703 .....................................]  ETA: 14s840ms | Step: 5ms
^C

It stuck there. Below are the debugger lines:

(gdb) info threads
  Id   Target Id         Frame
* 1    Thread 0x7f180fc29740 (LWP 168437) "luajit" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  2    Thread 0x7f176b510700 (LWP 168443) "luajit" __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95
  3    Thread 0x7f176ad0f700 (LWP 168444) "luajit" __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95
  4    Thread 0x7f176a50e700 (LWP 168445) "luajit" __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95
  5    Thread 0x7f1769d0d700 (LWP 168446) "luajit" __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95
  6    Thread 0x7f176950c700 (LWP 168447) "luajit" __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95
  7    Thread 0x7f1768d0b700 (LWP 168448) "luajit" __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95
  8    Thread 0x7f1763fff700 (LWP 168449) "luajit" __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95
  9    Thread 0x7f17637fe700 (LWP 168450) "luajit" __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95
(gdb) quit

With "-threads 1", I can get through it. Looks like a race condition/deadlock.

SeanNaren commented 8 years ago

Sorry for the late response, here was seeing if I could create the dataset locally on my machine. I'm running into the same issue as yourself with it taking much longer towards the end of the processing. I'll investigate further and try fix this issue.

SeanNaren commented 8 years ago

I'm not able to replicate this and everything went fine (did take some time since I was using a hdd). Could you use the libri_changes branch and see if this works? Also updating the lua---threads library via luarocks install threads won't hurt!