tenstorrent / tt-metal

:metal: TT-NN operator library, and TT-Metalium low level kernel programming model.
Apache License 2.0
312 stars 27 forks source link

[N300] TTNN Unit Test Failures: out_subblock_h * out_subblock_w <= 4 && "Total number of tiles in a subblock must be less than 4 when in fp32_dest_acc mode" #6990

Open cfjchu opened 3 months ago

cfjchu commented 3 months ago

Failure:

RuntimeError: TT_FATAL @ tt_eager/tt_dnn/op_library/bmm/multi_core_reuse_mcast_2d_optimized/bmm_op_multi_core_reuse_mcast_2d_optimized.cpp:956

Repro:

pytest -svv "tests/ttnn/unit_tests/operations/test_conv2d.py::test_sd_conv_wh[enable_auto_formatting=False-math_fidelity=MathFidelity.LoFi-fp32_accum=True-activations_dtype=DataType.BFLOAT16-weights_dtype=DataType.BFLOAT16-batch_size=2-output_channels=320-input_channels=960-input_height=64-input_width=64-filter_height=1-filter_width=1-stride_h=1-stride_w=1-pad_h=0-pad_w=0-use_1d_systolic_array=False-config_override=None]"

Will be disabling this test while I enable N300 unit test suite. fyi @jliangTT @xanderchin

tt-nshanker commented 3 months ago

Ill fix this issue.

nsmithtt commented 1 month ago

@sankarmanoj-tt, can you see if this is still an issue?

sankarmanoj-tt commented 1 month ago

This test case still fails on N300, with the error: Statically allocated circular buffers in program 4 clash with L1 buffers on core range [(x=0,y=0) - (x=4,y=6)]. L1 buffer allocated at 876544 and static circular buffer region ends at 1069952

nsmithtt commented 1 month ago

@sankarmanoj-tt , can you try repro with env WH_ARCH_YAML=wormhole_b0_80_arch_eth_dispatch.yaml?

sankarmanoj-tt commented 1 month ago

It still fails even with that env variable set.