search

tenstorrent / tt-metal

:metal: TT-NN operator library, and TT-Metalium low level kernel programming model.
Apache License 2.0
430 stars 58 forks source link

Investigate hang with batch 6 for post-softmax BMM #1744

Closed TT-BrianLiu closed 6 months ago

TT-BrianLiu commented 1 year ago

Post-softmax BMM with batch 6 hangs with this setup for some reason:

pytest tests/python_api_testing/models/bert_large_performant/unit_tests/test_bert_large_matmuls_and_bmms_with_mixed_precision.py::test_bert_large_bmm[post_softmax_bmm-batch_6-in0_BFLOAT8_B-in1_BFLOAT8_B-out_BFLOAT8_B-L1

For now, only support batches 7 - 9. It should work for 2 - 9 in general with multicasting in both directions.

TT-BrianLiu commented 1 year ago

Seems to hang for batch_size <= 6 for post_softmax BMM.

davorchap commented 1 year ago

@TT-BrianLiu this has been fixed? if yes, please close.