tenstorrent / tt-metal

:metal: TT-NN operator library, and TT-Metalium low level kernel programming model.
Apache License 2.0
303 stars 26 forks source link

Shallow unet 512x512 resolution fails for batch>1 #8484

Open keerthana-r-mcw opened 1 month ago

keerthana-r-mcw commented 1 month ago

Shallow unet with input resolution 512x512 is working for the dimension (1, 4, 512, 512), but fails with circular buffer issue (L1) for the dimension (2, 4, 512, 512)

To Reproduce Steps to reproduce the behavior:

  1. Checkout the branch saichand/shallow_unet_512x512_and_1024x1024.
  2. In /tt-metal/tests/ttnn/integration_tests/unet/test_ttnn_shallow_unet.py, change the batch = 2 in the function test_unet_512x512 (currently in line number 636) .
  3. Run the following command, pytest tests/ttnn/integration_tests/unet/test_ttnn_shallow_unet.py::test_unet_512x512
  4. The folowing error will occur, E RuntimeError: TT_THROW @ tt_metal/impl/program/program.cpp:496: tt::exception E info: E Statically allocated circular buffers in program 9 clash with L1 buffers on core range [(x=0,y=0) - (x=7,y=7)]. L1 buffer allocated at 595712 and static circular buffer region ends at 631680
nsmithtt commented 1 month ago

This is because this it too large to fit in L1.

Anyway, the current dims of 1056160 only just fit w/ batch 2, 512512 is 1.5x bigger and 1024*1024 is 6x bigger, so unfortunately I think this is expected.

dvartaniansTT commented 1 month ago

@nsmithtt thanks for the prompt response on this. Do we have a plan to support higher resolutions? I heard partial sharding is meant to enable us high res support.