Llama3 test_model_prefill running out of DRAM and segfault

tenstorrent / tt-metal

:metal: TT-NN operator library, and TT-Metalium low level kernel programming model.

Apache License 2.0

486 stars 80 forks source link

Open mtairum opened 3 weeks ago

mtairum commented 3 weeks ago

Describe the bug Issue present in pipeline here: https://github.com/tenstorrent/tt-metal/actions/runs/11718877474/job/32641073113

Also able to reproduce locally when running pytest models/demos/llama3/tests/test_llama_model_prefill.py

For now I've disabled this test from CI. Do not forget to re-add the test in tests/scripts/t3000/run_t3000_frequent_tests.sh - PR https://github.com/tenstorrent/tt-metal/pull/14844

mtairum commented 1 week ago

I have a fix in branch mtairum/debug_llama_post_commit.

When that one is merged this should be closed.