tenstorrent / tt-metal

:metal: TT-NN operator library, and TT-Metalium low level kernel programming model.
Apache License 2.0
396 stars 49 forks source link

Request for a T3K device perf pipeline #11032

Open skhorasganiTT opened 1 month ago

skhorasganiTT commented 1 month ago

There is currently no appropriate pipeline for testing model device perf on T3K. This is required for all models that run on T3K, particularly the ones that are model-parallel. The frequency of the pipeline could be once a day or once every other day depending on the availability of machines.

cc @uaydonat @djordje-tt @mtairum @cglagovich

skhorasganiTT commented 1 month ago

@TT-billteng @tt-rkim @ttmchiou If this request sounds alright, would you be able to help out with this?

djordje-tt commented 1 month ago

@tt-rkim @TT-billteng @ttmchiou, is there any update on this?

tt-rkim commented 1 month ago

Hey this is on our to-do list but we have other pipeline requests as well. We'll go through and update as we get to them.

Sorry for the delay. If you would like to try your hand at it and put up a PR, we can help guide.

tt-rkim commented 1 month ago

Can we get:

without all 3, we're not adding anything.

mtairum commented 1 month ago

The specific tests should be the same as the ones we have in the t3k perf pipeline:

https://github.com/tenstorrent/tt-metal/blob/main/tests/scripts/t3000/run_t3000_model_perf_tests.sh

Same owners as the ones associated for those models in that pipeline

tt-rkim commented 1 month ago

Can the models team please re-format the tests so that they can be called with the filename per #7586 ?