Closed aodhan-domhnaill closed 3 months ago
I am looking to record input and output to the vLLM. I could put an HTTP proxy in front and capture the traffic, or modify your handler.
Rather than make changes to the code, I was wondering if you might have a better way to do this?
We currently do not have a way to capture token usage across all workers, but it is a future item we will implement
I am looking to record input and output to the vLLM. I could put an HTTP proxy in front and capture the traffic, or modify your handler.
Rather than make changes to the code, I was wondering if you might have a better way to do this?