search

pytorch / test-infra

This repository hosts code that supports the testing infrastructure for the main PyTorch repo. For example, this repo hosts the logic to track disabled tests and slow tests, as well as our continuation integration jobs HUD/dashboard.
https://hud.pytorch.org/
Other
77 stars 73 forks source link

Codemod service to disable and mark slow tests directly in PyTorch code base #4981

Open huydhn opened 6 months ago

huydhn commented 6 months ago

The mechanism to disable a test is sometimes "abused" to keep a test disable for a long time without any owner to work on fixing it. Although, this helps mitigate an flaky test well, it has several drawback:

Ideally, we want to keep the current mechanism but limit it only to recently flaky tests, i.e. within 2 weeks. Anything older than that and has not yet been addressed should be disabled directly in the code. Doing this manually doesn't scale, so we need a better tool for this like a codemod service of sort.

cc @clee2000 @malfet @kit1980

huydhn commented 5 months ago

AI: It's better to be able to keep the test as XFAIL if possible or a special decorator like skipifFlaky