Add batch prediction - Githubissues

Shivanandroy / simpleT5

simpleT5 is built on top of PyTorch-lightning⚡️ and Transformers🤗 that lets you quickly train your T5 models.

MIT License

386 stars 62 forks source link

Open entropy2333 opened 2 years ago

entropy2333 commented 2 years ago

add batch encoding and decoding to speed up prediction

EdGaere commented 1 year ago

Great contribution :-) I tested it on a GPU (NVIDIA TITAN V), batched predictions provide up to 4x speedup.

ke-lara commented 1 year ago

This will be a big productivity booster. Please merge it, in case not done yet.