tenstorrent / pytorch2.0_ttnn

⭐️ TTNN Compiler for PyTorch 2.0 ⭐️ It enables running PyTorch2.0 models on Tenstorrent hardware
https://tenstorrent.github.io/tt-metal/latest/ttnn/
25 stars 6 forks source link

`test_speecht5_tts.py::test_speecht5_tts` is failing on main branch #375

Open jerrysky3 opened 3 days ago

jerrysky3 commented 3 days ago

The tests/models/speecht5_tts/test_speecht5_tts.py::test_speecht5_tts[eval] seems to fail with compilation error on main branch.

This issue is currently blocking PR merges (https://github.com/tenstorrent/pytorch2.0_ttnn/actions/runs/11624113518/job/32372283812)

To reproduce

On main branch run:

pytest --capture=no "tests/models/speecht5_tts/test_speecht5_tts.py::test_speecht5_tts[eval]"

It shows the error:

                except Exception as e2:
                    err_msg = f"{model_name} - Torch run with bypass compilation failed. "
                    err_msg += "Please check whether `model` or `model.generate` is passed to `record_property`."
                    raise TypeError(err_msg) from e2
                else:
                    if request.node.get_closest_marker("compilation_xfail"):
                        pytest.xfail()
                    else:
>                       raise TypeError(f"{model_name} compiled failed to run.") from e
E                       TypeError: speecht5-tts compiled failed to run.

tests/conftest.py:189: TypeError
=============================================================================================== warnings summary ================================================================================================
models/speecht5_tts/test_speecht5_tts.py::test_speecht5_tts[eval]
  /home/jerry/pytorch2.0_ttnn/venv/lib/python3.8/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
    warnings.warn(

models/speecht5_tts/test_speecht5_tts.py::test_speecht5_tts[eval]
  /home/jerry/pytorch2.0_ttnn/tests/utils.py:63: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working
    if isinstance(inputs, collections.Mapping):

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================================================================================ short test summary info ============================================================================================
ERROR tests/models/speecht5_tts/test_speecht5_tts.py::test_speecht5_tts[eval] - TypeError: speecht5-tts compiled failed to run.
============================================================================== 1 passed, 2 warnings, 1 error in 104.85s (0:01:44) ===============================================================================
                 Device | INFO     | Closing user mode device drivers
jerrysky3 commented 3 days ago

The test seems to be flaky. Sometimes it failed with result shape mismatch (e.g. torch.Size([23552]) vs. torch.Size([25600])). sometimes it matched and passed