tenstorrent / tt-metal

:metal: TT-NN operator library, and TT-Metalium low level kernel programming model.
Apache License 2.0
471 stars 75 forks source link

Global_avg_pool2d TTNN op fails with low PCC error [Grayskull and Wormhole] #6237

Open banekg opened 8 months ago

banekg commented 8 months ago

Describe the bug

Global_avg_pool2d TTNN operation breaks with low PCC error in some test cases, for both, Grayskull and Wormhole chips.

To Reproduce Steps to reproduce the behavior for both chips, GS and WH:

  1. Checkout main branch
  2. Run unit test test_global_avg_pool2d.py using this command: pytest tests/ttnn/python_api_testing/non_working_unit_tests/grayskull/test_global_avg_pool2d.py

Expected behavior There are test cases presented in the unit tests pytest tests/ttnn/python_api_testing/non_working_unit_tests/grayskull/test_global_avg_pool2d.py and they all are expected to fail with low PCC error.

Getting Additional info for the operation under test and its behavior To get additional information and results for different combinations of input shapes, types, layouts and memory configs for which this operation was tested you can also run locally sweeps for _ttnn.global_avgpool2d and check the results. To do this you should:

  1. Follow the Getting Started page to setup the repo, environment variables and python-env
  2. Activate source build/python_env/bin/activate
  3. Run sweeps by using python tests/tt_eager/python_api_testing/sweep_tests/run_pytorch_test.py -i tests/ttnn/python_api_testing/sweep_tests/test_configs/ci_sweep_tests_broken/grayskull/ttnn_global_avg_pool2d_test.yaml -o /home/ubuntu/tt-metal/ttnn-test-sweeps/global_avg_pool_2d
  4. After the run is completed all test sweeps results should be available inside specified output directory (in this case ./result-sweeps). There you will find global_avg_pool2d_sweep.csv which holds all executed sweeps, among which you can also find the ones that failed and were recreated by the unit test, which you can get by searching unique data_seed field.
jliangTT commented 8 months ago

i set as p2 for now, and assign to Nick to triage.