Falcon 7b prefill CI coverage

tenstorrent / tt-metal

:metal: TT-NN operator library, and TT-Metalium low level kernel programming model.

Apache License 2.0

303 stars 26 forks source link

Falcon 7b prefill CI coverage #7823

Open s-jovic opened 2 months ago

s-jovic commented 2 months ago

Current CI coverage for Falcon 7b prefill:

test_perf_falcon.py is ran as a part of model perf regression pipeline; PCC is checked here but the threshold is 0.89 so it's not very useful
some unit tests as a part of post commit fd/sd tests - very low coverage, some matmuls are not even used in the model

test_perf_falcon.py seems like a good test to discover hangs and perf regression, but we need to add one for 2k seq len as well, and proper e2e PCC tests for both 128 and 2k seq len.

s-jovic commented 2 months ago

Added e2e test for 128 seq len and proper PCC check in https://github.com/tenstorrent/tt-metal/pull/8218.

To add (after optimized prefill version is enabled):

[ ] 2k seq len e2e PCC test
[ ] 2k seq len perf test

s-jovic commented 1 month ago

Added e2e and perf tests here: https://github.com/tenstorrent/tt-metal/commit/4fc1bb496fad50b1ee8c4bbdb4ab0be631d19dc7.

uaydonat commented 5 days ago

can we close this?

pavlejosipovic commented 4 days ago

I'll leave this for @s-jovic to check when she is back from vacation, I don't think we have proper 2k e2e tests yet.