Add a wrapper to skip flaky tests and un-skip some MLP unit tests

ROCm / apex

A PyTorch Extension: Tools for easy mixed precision and distributed training in Pytorch

BSD 3-Clause "New" or "Revised" License

17 stars 14 forks source link

Add a wrapper to skip flaky tests and un-skip some MLP unit tests #86

Closed hubertlu-tw closed 2 years ago

hubertlu-tw commented 2 years ago

Un-skipped tests:

tests/L0/run_mlp/test_mlp.py::TestMLP::test_numeric
tests/L0/run_mlp/test_mlp.py::TestMLP::test_no_bias
tests/L0/run_mlp/test_mlp.py::TestMLP::test_no_grad
tests/L0/run_mlp/test_mlp.py::TestMLP::test_performance_half

Skipped flaky tests:

tests/L0/run_amp/test_checkpointing.py::test_loss_scale_decrease
tests/L0/run_fused_layer_norm/test_fused_layer_norm.py::TestFusedLayerNormElemWiseBFloat16
tests/L0/run_mlp/test_mlp.py::TestMLP::test_with_bias

hubertlu-tw commented 2 years ago

rocm-pytorch-master failed synced_batchnorm/test_batchnorm1d.py in run_rocm_distributed.sh. The error message is as follows:

Running syncbn batchnorm1d tests
Traceback (most recent call last):
  File "synced_batchnorm/test_batchnorm1d.py", line 16, in <module>
    assert(output.allclose(output_ref))
AssertionError
Build step 'Execute shell' marked build as failure

This test may be marked as flaky test in the future.