search

OpenNMT / CTranslate2

Fast inference engine for Transformer models
https://opennmt.net/CTranslate2
MIT License
3.07k stars 273 forks source link

[Feature Request] Expose Profiler to Python #1736

Open yehudaorel opened 2 weeks ago

yehudaorel commented 2 weeks ago

Problem

We am working on analyzing the performance of different layers of a workload and comparing the performance with different backends.

The current issue is that we are running whisper via python and were not able to use the built in profiler

I came across this comment which surprised me it still has not been implemented.

Solution

As a workaround, I exposed the profiler to python with some small code modifications and it seemed to function smoothly on my linux env.

see changes here: https://github.com/OpenNMT/CTranslate2/compare/master...yehudaorel:CTranslate2:python_profiler

I am aware I hard coded threads and device but this can easily be fixed.

Usage:

import ctranslate2

ctranslate2.init_profiler()
 # Translate/Transcribe
ctranslate2.dump_profiler()

I understand this is a very niche request but would be extremely beneficial to have integrated and properly documented. Let me know if you would like me to refine the implementation and open a PR for the fix. Either way I thought to leave this hear in case someone else would like to use it in the future.

minhthuc2502 commented 2 weeks ago

Hello, Could you open a PR for this feature? Thank you a lot