search

baidu-research / ba-dls-deepspeech

Apache License 2.0
486 stars 174 forks source link

Floating point exception (core dumped) #10

Closed dylanbfox closed 6 years ago

dylanbfox commented 7 years ago

I'm seeing a Floating point exception (core dumped) with no other traceback or errors unfortunately. It looks like it has something to do with the mini batch size, because it goes away whenever I set the mini batch size to (seemingly) >= 24. I added some logs to see where this is happening and it looks like it's at this line:

 _, ctc_cost = train_fn([inputs, output_lengths, labels, label_lengths, True])

I can't run the training routine with a mini batch size of 32 though because I always get memory errors.

Any ideas?

HulkSun commented 7 years ago

Hi @dylanbfox , if you use Theano as your backend,you could enlarge the memory size by changing the value 'cmmem',and you could find it in ~/.theanorc. hope that is useful!