search

microsoft / vidur

A large-scale simulation framework for LLM inference
MIT License
273 stars 42 forks source link

Support simulation of 4090 cards #8

Open Enoch2090 opened 6 months ago

Enoch2090 commented 6 months ago

Since 4090 really has an edge on inferencing with smaller models - is it possible to add data for 4090 cards? Thanks!

AgrawalAmey commented 6 months ago

Hi @Enoch2090, thanks a lot of your interest in the project. We definitely want to extend support to as many devices as possible. Unfortunately, I don't have access to 4090 cards, but if you are willing to contribute we could share all the profiling instructions. Thanks again!

teds-lin commented 6 months ago

Hi @Enoch2090, thanks a lot of your interest in the project. We definitely want to extend support to as many devices as possible. Unfortunately, I don't have access to 4090 cards, but if you are willing to contribute we could share all the profiling instructions. Thanks again!

Based on my current understanding, one of the key points for adding new GPU card support is collecting profiling data. Will you publish the relevant code and implementation steps? Thank!

AgrawalAmey commented 6 months ago

@teds-lin yes, the profiling code is already in the repo. We will add appropriate documentation soon. Thanks!

Enoch2090 commented 6 months ago

@AgrawalAmey Thanks for sharing! Would also be great if the document can point out approximately how much data profiled (or equivalently, how many GPU hours) is needed to produce accurate results.

AgrawalAmey commented 6 months ago

@teds-lin @Enoch2090 We have added the documentation to add profiling data for new models/devices. Adding a new model should take less than 15ish mins. Adding a new device across all 6 models should take less than a couple of hours. Please have a look https://github.com/microsoft/vidur/blob/main/vidur/profiling/README.md and let us know if you have any questions.

lhpp1314 commented 4 weeks ago

@AgrawalAmey The profiling link https://github.com/microsoft/vidur/blob/main/vidur/profiling/README.md is not available now. Can you give a new one?

nitinkedia7 commented 4 weeks ago

Hi @lhpp1314, the Supported Models section in the project README has the updated link for profiling. Please use the central readme to find links to other documentation.