search

llSourcell / tensorflow_speech_recognition_demo

This is the code for 'How to Make a Simple Tensorflow Speech Recognizer' by @Sirajology on Youtube
383 stars 250 forks source link

Training not using GPU #17

Open julitosm opened 7 years ago

julitosm commented 7 years ago

Hey everyone!

I'm trying to train my model with Sirajs Code. Unfortunately it takes ages to train (a few days). I have installed Tensorflow with GPU support, but the Code is not using any of the GPUs capacity. What am I getting wrong? Any suggestions? In Sirajs Video he says, that it takes some ours to train fully...

Thanks! Cheers julitos

vshmyhlo commented 7 years ago

Hey @julitosm, it will train forever because Siraj left an infinite loop in the code:

while 1:
  model.fit(trainX, trainY, n_epoch=10, validation_set=(testX, testY), show_metric=True, batch_size=batch_size)
  _y = model.predict(X)

replace while 1: with something like for _ in range(100):

julitosm commented 7 years ago

Hey! Thanks for your answer. I forgot to mention that I already fixed this loop. Tough, it takes more than a second to make one training iteration. For 300.000 iters it would take about 4 days to train. I also checked my GPU workload; it's between 0-5%.

laceywang1993 commented 6 years ago

Hello! Have you fixed your problem? I have the same problem as yours.

julitosm commented 6 years ago

Hey @laceywang1993! No, I have not solved it. I'm now using another project for my speech recognition. To solve this issue you could try to preprocess all your data before training the model and load the data into queues (https://www.tensorflow.org/programmers_guide/threading_and_queues). This could bypass the bottleneck between cpu and gpu.