Closed whoislimshady closed 2 years ago
The first generation always takes time, and then after that its quicker. See table below where I tested the time it takes to generate tokens of different length. This was tested on a v2-8 TPU instance with slim weights.
thanks @safeeazeem for the information really appreciate the effort
i am trying to deploy the gpt jax on gcp tpu v2-8 the generation takes around 35 sec which is way to slow i found few blogs which states that with 2-8 we can achieve latency of 5 sec can someone suggest me how to speed up the process and yes i am using the slim weights