search

aws-samples / foundation-model-benchmarking-tool

Foundation model benchmarking tool. Run any model on any AWS platform and benchmark for performance across instance type and serving stack options.
MIT No Attribution
154 stars 22 forks source link

FMBench to support REST predictors #64

Closed madhurprash closed 2 months ago

madhurprash commented 2 months ago

Issue #, if available:

Description of changes:

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

this example contains a Llama-2-13b chat deployed on an EKS cluster through Ray Serve. The rest predictor utilizes the rest ep url to generate responses.