How can we optimize the Model Inference time. Single NLQ taking more than a 45seconds.

[vaib26]

Hello everyone,

I hope you're doing well. I encountered an issue while using Fine-Tuned RESDSQL on my dataset(spider-like) for predicting SQL .The inference time goes around one minute for it. While profiling the steps I found that schema_item_calssifier.py and text2sql.py are taking majority of the time. I would greatly appreciate any suggestions or insights on optimizing/minimizing the prediction time. Thank you in advance for your assistance!

[lihaoyang-ruc]

Hi, here are some suggestions:

1. Keep the model in GPU memory instead of releasing it after each use, as releasing it each time will result in a lot of I/O time.
2. Using GPU to inference instead of CPU.
3. Try to reduce the beam size during decoding.
4. Try using tools that can accelerate model inference, such as TensorRT and FasterTransformer.

However, my suggestions are quite general, so they may not necessarily address your specific issue. If you want to achieve deeper acceleration, I recommend that you try to identify the code blocks that consume the most time.