Closed bliunlpr closed 4 years ago
Thanks for pointing out this. I have not fully tested the chain function yet, as I'm working on a few other aspects currently. Currently, it only works with batchsize as 1. I think @glynpu made a good point. It is possible that I have missed this, and hence why it is slow now. To fully utilize the chain objective, I need to implement another dataloader that can prepare minibatches like Kaldi does, e.g., 128 sequences of 1.5 second of audio with supervisions. Currently I do not have time to do that yet, but will work on that as long as I get my hands free.
Hi @bliunlpr
I double checked by my setup, and it takes around 1s to compute the loss for each utterance. Is the 8.2s that you mentioned corresponds to the computation of one utterance?
Thanks for your reply. @glynpu made a good point. I have added the lines to enable CUDA. Now the loss computation time is about 0.5s. Thanks again! @glynpu jzlianglu
@bliunlpr , great, would you like to push your changes to the main branch?
I have added it, its in pull requests. @jzlianglu
@bliunlpr , thanks, will test it.
Added CuDevice activation for LFMMI loss computation for significant speed improvement.
When I trained the model by ChainObjtiveFunction, I found the chain loss computation is very slow. For example, the data load time is 0.2s, the model forward time is 0.2s, but the loss computation time is 8.2s. I wonder why the chain loss computation is so slow and how to accelerate it? Thanks!