tenstorrent / tt-metal

:metal: TT-NN operator library, and TT-Metalium low level kernel programming model.
Apache License 2.0
463 stars 70 forks source link

Pytorch Sweeps tracing - ttnn.add - Low PCC and other fails #14730

Open KalaivaniMCW opened 16 hours ago

KalaivaniMCW commented 16 hours ago

Following parameters failed during Pytorch Sweeps tracing tests: for ttnn.add Test file: tests/sweep_framework/sweeps/eltwise/binary/add/add_all_pytorch2.py Pytorch sweeps: tracker

**Parameter** |   | **Error** -- | -- | -- {'input_shape': {'self': [1, 71, 7, 7], 'other': [7, 7]}, |   | low pcc fail {'input_shape': {'self': [920, 1, 256], 'other': [256]}, |   | low pcc fail {'input_shape': {'self': [0, 1], 'other': [0, 1]}, |   | round_up: multiple must not be 0 {'input_shape': {'self': [0], 'other': [0]}, |   | round_up: multiple must not be 0 {'input_shape': {'self': [3234], 'other': [3234]}, |   | message list(expected_pytorch_result.shape)=[3234] vs list(actual_pytorch_result.shape)=[1, 3234] {'input_shape': {'self': [8732], 'other': [8732]}, |   | message list(expected_pytorch_result.shape)=[8732] vs list(actual_pytorch_result.shape)=[1, 8732] {'input_shape': {'self': [], 'other': []}, |   | incompatible function arguments. The following argument types are supported: {'input_shape': {'self': [], 'other': 1}, |   | incompatible function arguments. The following argument types are supported: **Below are fails that occur on L1 memory: 4 cases (both inputs on L1)** {'input_shape': {'self': [13600, 1, 4], 'other': [1, 9, 4]}, |   | Out of Memory: Not enough space to allocate 27852800 B L1 buffer across 64 banks, where each bank needs to store 436224 B {'input_shape': {'self': [1, 64, 480, 640], 'other': [1, 64, 480, 640]}, |   | Out of Memory: Not enough space to allocate 39321600 B L1 buffer across 64 banks, where each bank needs to store 614400 B {'input_shape': {'self': [1, 256, 200, 272], 'other': [1, 256, 200, 272]}, |   | Out of Memory: Not enough space to allocate 33030144 B L1 buffer across 64 banks, where each bank needs to store 516096 B {'input_shape': {'self': [1, 256, 180, 320], 'other': [1, 256, 180, 320]}, |   | Out of Memory: Not enough space to allocate 31457280 B L1 buffer across 64 banks, where each bank needs to store 491520 B
ayerofieiev-tt commented 13 hours ago

@KalaivaniMCW , can you share some info on the PCC issue? Do we have any ideas on why this might happen?