search

tenstorrent / tt-metal

:metal: TT-NN operator library, and TT-Metalium low level kernel programming model.
Apache License 2.0
429 stars 57 forks source link

L1 wrap watcher error in stress test pipeline on WH #13496

Open TT-billteng opened 5 days ago

TT-billteng commented 5 days ago

Describe the bug

See logs in:

https://github.com/tenstorrent/tt-metal/actions/runs/11155702572/job/31007307926

To Reproduce

Using RelWithDebInfo build, run tests/tt_eager/python_api_testing/unit_testing/test_prod_all.py::test_prod[shapes0] with watcher enabled.

tt-asaigal commented 1 day ago

This seems to be related to code size overflow with watcher enabled. On the run that failed, the violated code size constraint was not detected. https://github.com/tenstorrent/tt-metal/commit/b64b445f1c17de277ce2299c13ee975fdeef25d5 makes sure that the prefetcher will not exceed code size with watcher enabled.

Will keep an eye on this test to make sure it doesn't show up with latest changes.