search

tenstorrent / tt-metal

:metal: TT-NN operator library, and TT-Metalium low level kernel programming model.
Apache License 2.0
469 stars 73 forks source link

ttnn.Conv1d fails with L1 OOM issue for SqueezeBERT [WH] #13093

Closed kkeerthana0573 closed 1 month ago

kkeerthana0573 commented 1 month ago

Describe the bug ttnn.Conv1d fails with L1 OOM issue for SqueezeBERT.

On WH - n300:

  E       RuntimeError: TT_THROW @ ../tt_metal/impl/allocator/allocator.cpp:143: tt::exception
  E       info:
  E       Out of Memory: Not enough space to allocate 192 B L1_SMALL buffer across 48 banks, where each bank needs to store 32 B

To Reproduce Steps to reproduce the behavior:

  1. Pick the test from the commit.
  2. Run pytest tests/ttnn/unit_tests/operations/test_squeezebert_conv1d.py::test_squeezebert_attention.

Expected behavior Expected to pass the testcase.

Please complete the following environment information: OS: [e.g. Ubuntu 20.04] Devices: WH(n300)

kkeerthana0573 commented 1 month ago

cc: @boris-drazic

mywoodstock commented 1 month ago

@kkeerthana0573 The test is missing device params for L1 small space. Please add this to the test:

@pytest.mark.parametrize("device_params", [{"l1_small_size": 16384}], indirect=True)
kkeerthana0573 commented 1 month ago

The issue is solved. Thank you.