The current inference (inference = prediction on live user data in production) is unnecessary slow.
For each prediction of a sentence the implementation does currently require approx. 3,5 seconds.
Possible solutions
Instantiate tokenizer only once with the startup of the backend -> DONE
Speed up iteration on the data loader or substitute by another solution.
Current situation
The current inference (inference = prediction on live user data in production) is unnecessary slow. For each prediction of a sentence the implementation does currently require approx. 3,5 seconds.
Possible solutions