How can we prune the INSTRUCTOR

[kishalxd]

How can we prune teh INSTRUCTOR model

[Harry-hash]

Could you provide more contexts about when do you want to prune the INSTRUCTOR model? Thanks!

[kishalxd]

Yes. I am trying to use the INSTRUCTOR in prod env. to reduce latency i want to prune the model. maybe using ONNX. Can u reference some ways to prune the vanilla instructor and prune it.

[Harry-hash]

Hi, Thanks a lot for the information!

To prune the INSTRUCTOR model and reduce the latency, apart from ONNX, here are some other suggested tools you may try:

• PyTorch Pruning: PyTorch has built-in support for weight and neuron pruning using various strategies, such as magnitude-based and structured pruning.
• NVIDIA TensorRT: TensorRT is a high-performance deep learning inference library developed by NVIDIA. It optimizes, prunes, and quantizes trained models for deployment on NVIDIA GPUs, delivering low-latency and high-throughput inference.
• Distiller: Distiller also provides a comprehensive set of tools for model pruning, quantization, and knowledge distillation.

Hope this helps! Feel free to add any further questions or comments!

[Harry-hash]

Feel free to re-open the issue if your have any further question or comment!