search

aws-samples / foundation-model-benchmarking-tool

Foundation model benchmarking tool. Run any model on any AWS platform and benchmark for performance across instance type and serving stack options.
https://aws-samples.github.io/foundation-model-benchmarking-tool/
MIT No Attribution
196 stars 31 forks source link

Is it possible to run FMBench on an EC2 instance on endpoints currently not supported on Sagemaker. #120

Open prasad-nair-amd opened 5 months ago

prasad-nair-amd commented 5 months ago

For example i want to run an endpoint which is running on intel/amd/nvidia instances which is generally available. ie, instances which are not available on Sagemaker but available on AWS.

aarora79 commented 5 months ago

We are working on a new FMBench release which supports hosting both the LLMs and FMBench on the same EC2 instance (so no SageMaker, no S3 dependancy) and this should enable you to test on AMD instances. Note that you can already do this today in a "bring your own endpoint" kind of way, see instructions here and config file examples here. What the next release would enable doing is to deploy the model on EC2 also through FMBench (although we have not yet to tried this on AMD instances so there would be some trial and error involved, in the meantime please take a look at the links above).

prasadwrites commented 4 months ago

@aarora79 Is there an update on the FMBench release you had pointed out which works on EC2 instance ?

aarora79 commented 4 months ago

Yes, the latest version of fmbench supports deploying models on ec2. Please let us connect offline so that we can get this moving faster. Cc @madhurprash @4mp

herothedog commented 3 months ago

Hello @aarora79 I can help with @prasadwrites to eval FMBench on EC2 AMD instances. Could we connect offline?

aarora79 commented 3 months ago

@herothedog absolutely, please send me an email at aroraai@amazon.com and I would setup a meeting. Looking forward.