Open devpramod opened 3 months ago
Hi @gkumbhat
Enabling graph mode is most applicable for models that need high performance and efficiency in production. A wide range of PyTorch operations are covered in TorchScript's graph mode. It is not recommended in cases where there might be complex post processing involving custom python libraries as part of the model code.
The speedups are dependent on the workload at hand. More information can be found here - https://pytorch.org/blog/optimizing-production-pytorch-performance-with-graph-transformations/ On Intel hardware, using torchscript along with ipex optimize uses AMX an ISA with instructions to speed up AI workloads.
working on addressing 1 & 2
Hi @gkumbhat I have resolved the formatting, linting and DCO check issues
Graph mode (torchscript) is an additional step that can further accelerate a workload that has been optimized with ipex and bfloat16 along with mixed precision. Graph mode utilizes AMX on SPR for additional speedups.
This PR modifies config file to have
graphmode
as an option to end users The graph is compiled in the constructor of the embedding module such that it is ready for execution when requests arrive.