search

aws-samples / foundation-model-benchmarking-tool

Foundation model benchmarking tool. Run any model on any AWS platform and benchmark for performance across instance type and serving stack options.
https://aws-samples.github.io/foundation-model-benchmarking-tool/
MIT No Attribution
193 stars 31 forks source link

Add support for different payload formats for bring your own datasets for that might be needed for different inference containers #5

Open madhurprash opened 9 months ago

madhurprash commented 9 months ago

This tool currently supports the HF TGI container, and DJL Deep Speed container on SageMaker and both use the same format but in future other containers might need a different payload format.

Goal: To give user full flexibility to bring their payloads or contain code that generalizes payload generation irrespective of the container type that the user uses. Two options for solution to this issue here:

1/ Have the user bring in their own payload 2/ Have a generic function defined to convert the payload in support for the container type the user is using to deploy their model and generate inference from.

aarora79 commented 8 months ago

Do this in the same way we have bring your own deployment script in that there is a inference function which is called from the run inference notebook.