search

idiap / pkwrap

A pytorch wrapper for LF-MMI training and parallel training in Kaldi
Other
73 stars 12 forks source link

RuntimeError: CUDA out of memory. Tried to allocate… but memory is empty #11

Closed pchampio closed 3 years ago

pchampio commented 3 years ago

I was constantly getting a out of memory error even through I reduced my minibatch size.

I've replace https://github.com/idiap/pkwrap/blob/e055fbc32f72bf7d3925a2c146cbe82557c5ab26/pkwrap/chain.py#L687 and https://github.com/idiap/pkwrap/blob/e055fbc32f72bf7d3925a2c146cbe82557c5ab26/pkwrap/chain.py#L519 to

minibatch_size=f"1:{chain_opts.minibatch_size.split(",")[-1]}",

Is this correct ? Thanks for this toolkit!

mrsrikanth commented 3 years ago

Right, that would work too. But before that I would test with a small minibatch size like 8 or 16. That should resolve your error, imo.

pchampio commented 3 years ago

thanks!