[Pytorch] There are 8 Recurrently Failing Jobs on pytorch/pytorch main - Githubissues

pytorch / test-infra

This repository hosts code that supports the testing infrastructure for the PyTorch organization. For example, this repo hosts the logic to track disabled tests and slow tests, as well as our continuation integration jobs HUD/dashboard.

https://hud.pytorch.org/

Other

83 stars 80 forks source link

[Pytorch] There are 8 Recurrently Failing Jobs on pytorch/pytorch main #4219

Closed github-actions[bot] closed 4 months ago

github-actions[bot] commented 1 year ago

Within the last 50 commits, there are the following failures on the main branch of pytorch:

periodic / win-vs2019-cuda11.8-py3 / test (default, 1, 4, windows.g5.4xlarge.nvidia.gpu) failed consecutively starting with commit 39427288f4ea36d4218bd509769e5cd800861420
periodic / win-vs2019-cuda11.8-py3 / test (default, 2, 4, windows.g5.4xlarge.nvidia.gpu) failed consecutively starting with commit c07a799ed552a9944e9a8d887e4cf0d9337af810
periodic / win-vs2019-cuda11.8-py3 / test (default, 4, 4, windows.g5.4xlarge.nvidia.gpu) failed consecutively starting with commit c07a799ed552a9944e9a8d887e4cf0d9337af810
pull / before-test / llm-retrieval failed consecutively starting with commit 67416a2996349a2339328cac0f7e54c7d3b3c1d9
pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 1, 5, linux.g5.4xlarge.nvidia.gpu) failed consecutively starting with commit b9d3cedd648d4ed9d0bf5b918893341e5f95289a
slow / linux-focal-py3.8-clang10 / test (slow, 2, 2, linux.2xlarge) failed consecutively starting with commit ad607b91f4cde64ccb01ffb50a344caf22070732
trunk / before-test / llm-retrieval failed consecutively starting with commit 67416a2996349a2339328cac0f7e54c7d3b3c1d9
trunk / linux-focal-cuda12.4-py3.10-gcc9-sm86 / test (default, 1, 5, linux.g5.4xlarge.nvidia.gpu) failed consecutively starting with commit b9d3cedd648d4ed9d0bf5b918893341e5f95289a

Please review the errors and revert if needed.

clee2000 commented 8 months ago

These jobs started failing:

inductor / rocm6.0-py3.8-inductor / test (inductor, 1, 1, linux.rocm.gpu.2)
rocm / linux-focal-rocm6.0-py3.8 / test (default, 1, 6, linux.rocm.gpu.2)
rocm / linux-focal-rocm6.0-py3.8 / test (default, 2, 6, linux.rocm.gpu.2)
rocm / linux-focal-rocm6.0-py3.8 / test (default, 3, 6, linux.rocm.gpu.2)
rocm / linux-focal-rocm6.0-py3.8 / test (default, 4, 6, linux.rocm.gpu.2)
rocm / linux-focal-rocm6.0-py3.8 / test (default, 5, 6, linux.rocm.gpu.2)
rocm / linux-focal-rocm6.0-py3.8 / test (default, 6, 6, linux.rocm.gpu.2)

github-actions[bot] commented 8 months ago

These jobs started failing:

trunk / macos-12-py3-arm64 / test (default, 3, 3, macos-m1-12)

github-actions[bot] commented 8 months ago

These jobs started failing:

pull / linux-focal-py3.12-clang10 / test (default, 1, 3, linux.2xlarge)
trunk / linux-focal-cuda12.1-py3.10-gcc9 / test (nogpu_AVX512, 1, 1, linux.2xlarge)
trunk / linux-focal-cuda12.1-py3.10-gcc9 / test (nogpu_NO_AVX2, 1, 1, linux.2xlarge)

github-actions[bot] commented 8 months ago

These jobs stopped failing:

trunk / macos-12-py3-arm64 / test (default, 3, 3, macos-m1-12)

These jobs started failing:

pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 2, 5, linux.4xlarge.nvidia.gpu)
pull / linux-focal-py3.11-clang10 / test (default, 3, 3, linux.2xlarge)

github-actions[bot] commented 8 months ago

These jobs stopped failing:

pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 2, 5, linux.4xlarge.nvidia.gpu)
pull / linux-focal-py3.12-clang10 / test (default, 1, 3, linux.2xlarge)

github-actions[bot] commented 8 months ago

These jobs stopped failing:

pull / linux-focal-py3.11-clang10 / test (default, 3, 3, linux.2xlarge)

github-actions[bot] commented 8 months ago

These jobs started failing:

slow / linux-focal-cuda12.1-py3-gcc9-slow-gradcheck / test (default, 2, 4, linux.g5.4xlarge.nvidia.gpu)

github-actions[bot] commented 8 months ago

These jobs started failing:

trunk / macos-12-py3-arm64 / test (default, 3, 3, macos-m1-12)
trunk / win-vs2019-cpu-py3 / test (default, 1, 3, windows.4xlarge.nonephemeral)

github-actions[bot] commented 8 months ago

These jobs started failing:

pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 3, 5, linux.4xlarge.nvidia.gpu)
pull / linux-focal-py3.11-clang10 / test (default, 3, 3, linux.2xlarge)

github-actions[bot] commented 8 months ago

These jobs stopped failing:

trunk / win-vs2019-cpu-py3 / test (default, 1, 3, windows.4xlarge.nonephemeral)

These jobs started failing:

pull / linux-focal-py3.12-clang10 / test (default, 1, 3, linux.2xlarge)

github-actions[bot] commented 8 months ago

These jobs stopped failing:

pull / linux-focal-py3.11-clang10 / test (default, 3, 3, linux.2xlarge)
trunk / macos-12-py3-arm64 / test (default, 3, 3, macos-m1-12)

github-actions[bot] commented 8 months ago

These jobs stopped failing:

pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 3, 5, linux.4xlarge.nvidia.gpu)

These jobs started failing:

trunk / win-vs2019-cpu-py3 / test (default, 1, 3, windows.4xlarge.nonephemeral)

github-actions[bot] commented 8 months ago

These jobs started failing:

pull / linux-focal-py3.8-clang10 / test (default, 3, 3, linux.2xlarge)

github-actions[bot] commented 8 months ago

These jobs stopped failing:

trunk / linux-focal-cuda12.1-py3.10-gcc9 / test (nogpu_AVX512, 1, 1, linux.2xlarge)
trunk / linux-focal-cuda12.1-py3.10-gcc9 / test (nogpu_NO_AVX2, 1, 1, linux.2xlarge)

github-actions[bot] commented 8 months ago

These jobs stopped failing:

pull / linux-focal-py3.8-clang10 / test (default, 3, 3, linux.2xlarge)

github-actions[bot] commented 8 months ago

These jobs started failing:

trunk / win-vs2019-cpu-py3 / test (default, 2, 3, windows.4xlarge.nonephemeral)

github-actions[bot] commented 8 months ago

These jobs stopped failing:

trunk / win-vs2019-cpu-py3 / test (default, 2, 3, windows.4xlarge.nonephemeral)

github-actions[bot] commented 8 months ago

These jobs stopped failing:

pull / linux-focal-py3.12-clang10 / test (default, 1, 3, linux.2xlarge)

github-actions[bot] commented 8 months ago

These jobs stopped failing:

trunk / win-vs2019-cpu-py3 / test (default, 1, 3, windows.4xlarge.nonephemeral)

github-actions[bot] commented 8 months ago

These jobs stopped failing:

inductor / rocm6.0-py3.8-inductor / test (inductor, 1, 1, linux.rocm.gpu.2)
rocm / linux-focal-rocm6.0-py3.8 / test (default, 1, 6, linux.rocm.gpu.2)
rocm / linux-focal-rocm6.0-py3.8 / test (default, 2, 6, linux.rocm.gpu.2)
rocm / linux-focal-rocm6.0-py3.8 / test (default, 3, 6, linux.rocm.gpu.2)
rocm / linux-focal-rocm6.0-py3.8 / test (default, 4, 6, linux.rocm.gpu.2)
rocm / linux-focal-rocm6.0-py3.8 / test (default, 5, 6, linux.rocm.gpu.2)
rocm / linux-focal-rocm6.0-py3.8 / test (default, 6, 6, linux.rocm.gpu.2)

github-actions[bot] commented 8 months ago

These jobs started failing:

pull / linux-focal-cuda12.1-py3.10-gcc9-bazel-test / build-and-test (default, 1, 1, linux.4xlarge.nvidia.gpu, mem_leak_check)

github-actions[bot] commented 8 months ago

These jobs stopped failing:

slow / linux-focal-cuda12.1-py3-gcc9-slow-gradcheck / test (default, 2, 4, linux.g5.4xlarge.nvidia.gpu)

github-actions[bot] commented 8 months ago

These jobs started failing:

slow / linux-focal-cuda12.1-py3-gcc9-slow-gradcheck / test (default, 2, 4, linux.g5.4xlarge.nvidia.gpu)

github-actions[bot] commented 8 months ago

These jobs started failing:

trunk / linux-focal-cuda12.1-py3.10-gcc9 / test (nogpu_AVX512, 1, 1, linux.2xlarge)
trunk / linux-focal-cuda12.1-py3.10-gcc9 / test (nogpu_NO_AVX2, 1, 1, linux.2xlarge)

github-actions[bot] commented 8 months ago

These jobs started failing:

pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 5, 5, linux.g5.4xlarge.nvidia.gpu)

github-actions[bot] commented 8 months ago

These jobs stopped failing:

pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 5, 5, linux.g5.4xlarge.nvidia.gpu)
trunk / linux-focal-cuda12.1-py3.10-gcc9 / test (nogpu_AVX512, 1, 1, linux.2xlarge)

github-actions[bot] commented 8 months ago

These jobs stopped failing:

trunk / linux-focal-cuda12.1-py3.10-gcc9 / test (nogpu_NO_AVX2, 1, 1, linux.2xlarge)

github-actions[bot] commented 8 months ago

These jobs stopped failing:

pull / linux-focal-cuda12.1-py3.10-gcc9-bazel-test / build-and-test (default, 1, 1, linux.4xlarge.nvidia.gpu, mem_leak_check)

github-actions[bot] commented 8 months ago

These jobs started failing:

pull / linux-focal-cuda12.1-py3.10-gcc9-bazel-test / build-and-test (default, 1, 1, linux.4xlarge.nvidia.gpu)

github-actions[bot] commented 8 months ago

These jobs stopped failing:

pull / linux-focal-cuda12.1-py3.10-gcc9-bazel-test / build-and-test (default, 1, 1, linux.4xlarge.nvidia.gpu)

github-actions[bot] commented 8 months ago

These jobs started failing:

Lint / lintrunner-noclang / linux-job

github-actions[bot] commented 8 months ago

These jobs started failing:

pull / linux-focal-cuda12.1-py3.10-gcc9-bazel-test / build-and-test (default, 1, 1, linux.4xlarge.nvidia.gpu)

github-actions[bot] commented 8 months ago

These jobs started failing:

periodic / win-vs2019-cuda11.8-py3 / test (default, 4, 4, windows.g5.4xlarge.nvidia.gpu)

github-actions[bot] commented 8 months ago

These jobs started failing:

slow / linux-focal-cuda12.1-py3-gcc9-slow-gradcheck / test (default, 4, 4, linux.g5.4xlarge.nvidia.gpu)

github-actions[bot] commented 8 months ago

These jobs stopped failing:

pull / linux-focal-cuda12.1-py3.10-gcc9-bazel-test / build-and-test (default, 1, 1, linux.4xlarge.nvidia.gpu)

These jobs started failing:

trunk / macos-12-py3-arm64 / test (default, 3, 3, macos-m1-12)

github-actions[bot] commented 8 months ago

These jobs stopped failing:

trunk / macos-12-py3-arm64 / test (default, 3, 3, macos-m1-12)

github-actions[bot] commented 8 months ago

These jobs started failing:

trunk / macos-12-py3-arm64 / test (default, 1, 3, macos-m1-12)

github-actions[bot] commented 8 months ago

These jobs started failing:

trunk / linux-focal-cuda12.1-py3.10-gcc9 / test (nogpu_NO_AVX2, 1, 1, linux.2xlarge)

github-actions[bot] commented 8 months ago

These jobs started failing:

trunk / linux-focal-cuda12.1-py3.10-gcc9 / test (nogpu_AVX512, 1, 1, linux.2xlarge)

github-actions[bot] commented 8 months ago

These jobs started failing:

pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 2, 5, linux.4xlarge.nvidia.gpu)

github-actions[bot] commented 8 months ago

These jobs stopped failing:

trunk / macos-12-py3-arm64 / test (default, 1, 3, macos-m1-12)

These jobs started failing:

pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 5, 5, linux.g5.4xlarge.nvidia.gpu)

github-actions[bot] commented 8 months ago

These jobs stopped failing:

pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 5, 5, linux.g5.4xlarge.nvidia.gpu)

github-actions[bot] commented 8 months ago

These jobs stopped failing:

pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 2, 5, linux.4xlarge.nvidia.gpu)

github-actions[bot] commented 8 months ago

These jobs started failing:

pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 2, 5, linux.4xlarge.nvidia.gpu)
pull / linux-focal-py3.8-clang10 / test (default, 2, 3, linux.2xlarge)
pull / linux-jammy-py3.8-gcc11 / test (default, 1, 3, linux.2xlarge)

github-actions[bot] commented 8 months ago

These jobs started failing:

pull / linux-focal-py3.11-clang10 / test (default, 2, 3, linux.2xlarge)

github-actions[bot] commented 8 months ago

These jobs stopped failing:

pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 2, 5, linux.4xlarge.nvidia.gpu)

github-actions[bot] commented 8 months ago

These jobs stopped failing:

pull / linux-jammy-py3.8-gcc11 / test (default, 1, 3, linux.2xlarge)

github-actions[bot] commented 8 months ago

These jobs stopped failing:

pull / linux-focal-py3.11-clang10 / test (default, 2, 3, linux.2xlarge)

github-actions[bot] commented 8 months ago

These jobs started failing:

pull / linux-jammy-py3.10-clang15-asan / test (default, 1, 6, linux.4xlarge)

github-actions[bot] commented 8 months ago

These jobs started failing:

pull / linux-jammy-py3.8-gcc11 / test (default, 1, 3, linux.2xlarge)

Next