search

aws-samples / foundation-model-benchmarking-tool

Foundation model benchmarking tool. Run any model on any AWS platform and benchmark for performance across instance type and serving stack options.
MIT No Attribution
151 stars 21 forks source link

Add Time to first token (TTFT), time per output token (TPOT) and time to last token (TTLT) #131

Open madhurprash opened 1 week ago

madhurprash commented 1 week ago

To add time to first token (TTFT), time per output token (TPOT) and time to last token (TTLT). This would be done using the streaming API that is now supported for SageMaker and Bedrock, and maybe EC2 as well over time.