I notice that doing inference with language model on large amount of texts can be quite slow. In particular, it took me 11 minutes to decode around 4600 lines of text. I wonder if pyctcdecode supports using GPU for acceleration ? At the moment, I see that the model's using CPU only, making it hard for batch decoding and such.
thangld201
commented
2 years ago
I notice that doing inference with language model on large amount of texts can be quite slow. In particular, it took me 11 minutes to decode around 4600 lines of text. I wonder if pyctcdecode supports using GPU for acceleration ? At the moment, I see that the model's using CPU only, making it hard for batch decoding and such.