Closed horheynm closed 4 months ago
Great job!
- Lets test with an openai example to make sure the integration works
- Add testing, specifically around benchmark_pipeline and server integration
- Refactor benchmark_pipeline function to prevent repeat code
I addressed tests and refactor, but not testing with openai - talked to Ben, dont need it for now
server running pipeline for benchmark
Note: Cannot have continuous batching with timer middleware.
Configs
Server side
client side
Outputs
server
client