[Pytorch] There are 6 Recurrently Failing Jobs on pytorch/pytorch main - Githubissues

pytorch / test-infra

This repository hosts code that supports the testing infrastructure for the PyTorch organization. For example, this repo hosts the logic to track disabled tests and slow tests, as well as our continuation integration jobs HUD/dashboard.

https://hud.pytorch.org/

Other

82 stars 79 forks source link

[Pytorch] There are 6 Recurrently Failing Jobs on pytorch/pytorch main #4216

Open github-actions[bot] opened 1 year ago

github-actions[bot] commented 1 year ago

Within the last 50 commits, there are the following failures on the main branch of pytorch:

docker-builds / docker-build (linux.12xlarge, pytorch-linux-focal-rocm-n-py3) failed consecutively starting with commit 1e738420296a84406cd0a1626074ea6447a6603a
inductor / unit-test / cuda12.1-py3.10-gcc9-sm86 / test (inductor, 1, 2, linux.g5.4xlarge.nvidia.gpu) failed consecutively starting with commit e31136d07bbfb10735df101df953c73d22dde24b
inductor / unit-test / cuda12.1-py3.12-gcc9-sm86 / test (inductor, 1, 2, linux.g5.4xlarge.nvidia.gpu) failed consecutively starting with commit e31136d07bbfb10735df101df953c73d22dde24b
periodic / linux-focal-rocm6.2-py3.10 / test (distributed, 1, 3, linux.rocm.gpu) failed consecutively starting with commit 8d1eaa3da663c2bdd626ff1aafa1c5d0ff6e1bf3
periodic / linux-focal-rocm6.2-py3.10 / test (distributed, 3, 3, linux.rocm.gpu) failed consecutively starting with commit 0b13bdd877f7b612ab2990e327ab3b40242945bf
trunk / linux-focal-rocm6.2-py3.10 / test (distributed, 1, 1, linux.rocm.gpu) failed consecutively starting with commit b8b60e0bc5d061bcca22e456f58c804773e3c5e8

Please review the errors and revert if needed.

github-actions[bot] commented 1 year ago

These jobs started failing:

inductor / cuda11.8-py3.10-gcc7-sm86 / test (inductor_torchbench_dynamic, 1, 1, linux.g5.4xlarge.nvidia.gpu)

github-actions[bot] commented 1 year ago

These jobs started failing:

inductor / cuda11.8-py3.10-gcc7-sm86 / test (inductor_torchbench, 1, 1, linux.g5.4xlarge.nvidia.gpu)

github-actions[bot] commented 1 year ago

These jobs stopped failing:

inductor / linux-focal-cpu-py3.8-gcc7-inductor / test (inductor_torchbench_cpu_accuracy, 1, 1, linux.4xlarge)

github-actions[bot] commented 1 year ago

These jobs stopped failing:

inductor / cuda11.8-py3.10-gcc7-sm86 / test (inductor_torchbench_dynamic, 1, 1, linux.g5.4xlarge.nvidia.gpu)

github-actions[bot] commented 2 months ago

These jobs started failing:

inductor / cuda12.1-py3.10-gcc9-sm86 / test (inductor, 1, 2, linux.g5.4xlarge.nvidia.gpu, mem_leak_check)
inductor / cuda12.1-py3.10-gcc9-sm86 / test (inductor, 2, 2, linux.g5.4xlarge.nvidia.gpu, mem_leak_check)
inductor / cuda12.1-py3.12-gcc9-sm86 / test (inductor, 1, 2, linux.g5.4xlarge.nvidia.gpu, mem_leak_check)
inductor / cuda12.1-py3.12-gcc9-sm86 / test (inductor, 2, 2, linux.g5.4xlarge.nvidia.gpu, mem_leak_check)
inductor-cu124 / cuda12.4-py3.10-gcc9-sm86 / test (inductor, 1, 2, linux.g5.4xlarge.nvidia.gpu, mem_leak_check)
inductor-cu124 / cuda12.4-py3.10-gcc9-sm86 / test (inductor, 2, 2, linux.g5.4xlarge.nvidia.gpu, mem_leak_check)
inductor-cu124 / cuda12.4-py3.12-gcc9-sm86 / test (inductor, 1, 2, linux.g5.4xlarge.nvidia.gpu, mem_leak_check)
inductor-cu124 / cuda12.4-py3.12-gcc9-sm86 / test (inductor, 2, 2, linux.g5.4xlarge.nvidia.gpu, mem_leak_check)
periodic / linux-focal-cuda11.8-py3.10-gcc9-debug / test (default, 2, 5, linux.4xlarge.nvidia.gpu)
periodic / linux-focal-cuda12.1-py3.10-gcc9 / test (nogpu_AVX512, 1, 1, linux.2xlarge)
periodic / linux-focal-cuda12.1-py3.10-gcc9 / test (nogpu_NO_AVX2, 1, 1, linux.2xlarge)
periodic / linux-focal-cuda12.1-py3.10-gcc9-experimental-split-build / test (nogpu_AVX512, 1, 1, linux.2xlarge)
periodic / linux-focal-cuda12.1-py3.10-gcc9-experimental-split-build / test (nogpu_NO_AVX2, 1, 1, linux.2xlarge)
periodic / linux-focal-cuda12.4-py3.10-gcc9 / test (default, 5, 5, linux.4xlarge.nvidia.gpu)
periodic / linux-focal-cuda12.4-py3.10-gcc9 / test (nogpu_AVX512, 1, 1, linux.2xlarge)
periodic / linux-focal-cuda12.4-py3.10-gcc9 / test (nogpu_NO_AVX2, 1, 1, linux.2xlarge)
periodic / win-vs2019-cuda11.8-py3 / test (default, 2, 4, windows.g5.4xlarge.nvidia.gpu, mem_leak_check)
pull / linux-focal-py3.11-clang10 / test (crossref, 2, 2, linux.2xlarge)
pull / linux-focal-py3.11-clang10 / test (default, 2, 4, linux.2xlarge)
pull / linux-focal-py3.11-clang10 / test (dynamo, 3, 3, linux.2xlarge)
pull / linux-focal-py3.12-clang10 / test (dynamo, 3, 3, linux.2xlarge)
pull / linux-focal-py3.12-clang10-experimental-split-build / test (dynamo, 3, 3, linux.2xlarge)
pull / linux-focal-py3.9-clang10 / test (crossref, 1, 2, linux.2xlarge)
pull / linux-focal-py3.9-clang10 / test (default, 2, 4, linux.2xlarge)
pull / linux-focal-py3.9-clang10 / test (dynamo, 3, 3, linux.2xlarge)
pull / linux-jammy-py3.9-gcc11 / test (default, 4, 4, linux.2xlarge)
slow / linux-focal-cuda12.1-py3-gcc9-slow-gradcheck / test (default, 3, 6, linux.g5.4xlarge.nvidia.gpu)
slow / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (slow, 2, 2, linux.g5.4xlarge.nvidia.gpu)
slow / linux-focal-py3.9-clang10 / test (slow, 1, 2, linux.2xlarge)
slow / linux-focal-py3.9-clang10 / test (slow, 2, 2, linux.2xlarge)
slow / linux-focal-rocm6.1-py3.8 / test (slow, 1, 2, linux.rocm.gpu)
slow / linux-focal-rocm6.1-py3.8 / test (slow, 2, 2, linux.rocm.gpu)
trunk / linux-focal-cuda12.4-py3.10-gcc9-experimental-split-build-test / test (default, 3, 5, linux.4xlarge.nvidia.gpu)
trunk / linux-focal-cuda12.4-py3.10-gcc9-experimental-split-build-test / test (nogpu_AVX512, 1, 1, linux.2xlarge)
trunk / linux-focal-cuda12.4-py3.10-gcc9-experimental-split-build-test / test (nogpu_NO_AVX2, 1, 1, linux.2xlarge)

github-actions[bot] commented 2 months ago

These jobs stopped failing:

slow / linux-focal-cuda12.1-py3-gcc9-slow-gradcheck / test (default, 3, 6, linux.g5.4xlarge.nvidia.gpu)

These jobs started failing:

slow / linux-focal-rocm6.1-py3.8 / test (slow, 1, 2, linux.rocm.gpu, mem_leak_check)

github-actions[bot] commented 2 months ago

These jobs started failing:

slow / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (slow, 2, 2, linux.g5.4xlarge.nvidia.gpu, mem_leak_check)

github-actions[bot] commented 2 months ago

These jobs started failing:

pull / linux-focal-py3.12-clang10-experimental-split-build / test (default, 1, 3, linux.2xlarge)

github-actions[bot] commented 2 months ago

These jobs stopped failing:

pull / linux-focal-py3.12-clang10-experimental-split-build / test (default, 1, 3, linux.2xlarge)
trunk / linux-focal-cuda12.4-py3.10-gcc9-experimental-split-build-test / test (default, 3, 5, linux.4xlarge.nvidia.gpu)
trunk / linux-focal-cuda12.4-py3.10-gcc9-experimental-split-build-test / test (nogpu_AVX512, 1, 1, linux.2xlarge)
trunk / linux-focal-cuda12.4-py3.10-gcc9-experimental-split-build-test / test (nogpu_NO_AVX2, 1, 1, linux.2xlarge)

github-actions[bot] commented 2 months ago

These jobs started failing:

periodic / parallelnative-linux-jammy-py3.9-gcc11 / test (default, 3, 3, linux.2xlarge)

github-actions[bot] commented 2 months ago

These jobs started failing:

periodic / linux-focal-cuda12.1-py3.10-gcc9 / test (nogpu_AVX512, 1, 1, linux.2xlarge, mem_leak_check)
periodic / linux-focal-cuda12.1-py3.10-gcc9 / test (nogpu_NO_AVX2, 1, 1, linux.2xlarge, mem_leak_check)
periodic / linux-focal-cuda12.1-py3.10-gcc9-experimental-split-build / test (nogpu_AVX512, 1, 1, linux.2xlarge, mem_leak_check)
periodic / linux-focal-cuda12.1-py3.10-gcc9-experimental-split-build / test (nogpu_NO_AVX2, 1, 1, linux.2xlarge, mem_leak_check)
periodic / linux-focal-cuda12.4-py3.10-gcc9 / test (nogpu_AVX512, 1, 1, linux.2xlarge, mem_leak_check)
periodic / linux-focal-cuda12.4-py3.10-gcc9 / test (nogpu_NO_AVX2, 1, 1, linux.2xlarge, mem_leak_check)

github-actions[bot] commented 2 months ago

These jobs started failing:

trunk / linux-focal-cuda12.4-py3.10-gcc9-experimental-split-build-test / test (nogpu_AVX512, 1, 1, linux.2xlarge, mem_leak_check)
trunk / linux-focal-cuda12.4-py3.10-gcc9-experimental-split-build-test / test (nogpu_NO_AVX2, 1, 1, linux.2xlarge, mem_leak_check)

github-actions[bot] commented 2 months ago

These jobs started failing:

trunk / linux-focal-cuda12.4-py3.10-gcc9-experimental-split-build-test / test (default, 3, 5, linux.4xlarge.nvidia.gpu)

github-actions[bot] commented 2 months ago

These jobs started failing:

pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 3, 5, linux.4xlarge.nvidia.gpu, mem_leak_check)

github-actions[bot] commented 2 months ago

These jobs started failing:

trunk / linux-focal-cuda12.4-py3.10-gcc9-experimental-split-build-test / test (default, 3, 5, linux.4xlarge.nvidia.gpu, mem_leak_check)

github-actions[bot] commented 2 months ago

These jobs stopped failing:

periodic / linux-focal-cuda12.4-py3.10-gcc9 / test (default, 5, 5, linux.4xlarge.nvidia.gpu)

These jobs started failing:

trunk / linux-focal-cuda12.4-py3.10-gcc9-experimental-split-build-test / test (nogpu_AVX512, 1, 1, linux.2xlarge)
trunk / linux-focal-cuda12.4-py3.10-gcc9-experimental-split-build-test / test (nogpu_NO_AVX2, 1, 1, linux.2xlarge)

github-actions[bot] commented 2 months ago

These jobs stopped failing:

slow / linux-focal-py3.9-clang10 / test (slow, 1, 2, linux.2xlarge)

github-actions[bot] commented 2 months ago

These jobs started failing:

slow / linux-focal-cuda12.1-py3-gcc9-slow-gradcheck / test (default, 4, 6, linux.g5.4xlarge.nvidia.gpu)

github-actions[bot] commented 2 months ago

These jobs stopped failing:

periodic / linux-focal-cuda11.8-py3.10-gcc9-debug / test (default, 2, 5, linux.4xlarge.nvidia.gpu)

github-actions[bot] commented 2 months ago

These jobs stopped failing:

pull / linux-focal-py3.11-clang10 / test (default, 2, 4, linux.2xlarge)

github-actions[bot] commented 2 months ago

These jobs stopped failing:

pull / linux-focal-py3.9-clang10 / test (crossref, 1, 2, linux.2xlarge)
pull / linux-jammy-py3.9-gcc11 / test (default, 4, 4, linux.2xlarge)
trunk / linux-focal-cuda12.4-py3.10-gcc9-experimental-split-build-test / test (default, 3, 5, linux.4xlarge.nvidia.gpu)

github-actions[bot] commented 2 months ago

These jobs stopped failing:

slow / linux-focal-rocm6.1-py3.8 / test (slow, 1, 2, linux.rocm.gpu)

github-actions[bot] commented 2 months ago

These jobs stopped failing:

pull / linux-focal-py3.11-clang10 / test (crossref, 2, 2, linux.2xlarge)
pull / linux-focal-py3.11-clang10 / test (dynamo, 3, 3, linux.2xlarge)
pull / linux-focal-py3.9-clang10 / test (default, 2, 4, linux.2xlarge)
trunk / linux-focal-cuda12.4-py3.10-gcc9-experimental-split-build-test / test (nogpu_AVX512, 1, 1, linux.2xlarge)
trunk / linux-focal-cuda12.4-py3.10-gcc9-experimental-split-build-test / test (nogpu_NO_AVX2, 1, 1, linux.2xlarge)

github-actions[bot] commented 2 months ago

These jobs started failing:

pull / linux-focal-py3.11-clang10 / test (dynamo, 3, 3, linux.2xlarge)

github-actions[bot] commented 2 months ago

These jobs stopped failing:

periodic / win-vs2019-cuda11.8-py3 / test (default, 2, 4, windows.g5.4xlarge.nvidia.gpu, mem_leak_check)

These jobs started failing:

pull / linux-focal-py3.11-clang10 / test (default, 3, 4, linux.2xlarge)
pull / linux-jammy-py3.9-gcc11 / test (default, 1, 4, linux.2xlarge)

github-actions[bot] commented 2 months ago

These jobs started failing:

pull / linux-focal-py3.11-clang10 / test (crossref, 2, 2, linux.2xlarge)
pull / linux-focal-py3.12-clang10-experimental-split-build / test (default, 1, 3, linux.2xlarge)
pull / linux-focal-py3.9-clang10 / test (crossref, 2, 2, linux.2xlarge)
pull / linux-focal-py3.9-clang10 / test (default, 2, 4, linux.2xlarge)

github-actions[bot] commented 2 months ago

These jobs started failing:

trunk / linux-focal-cuda12.4-py3.10-gcc9-experimental-split-build-test / test (nogpu_AVX512, 1, 1, linux.2xlarge)

github-actions[bot] commented 2 months ago

These jobs started failing:

trunk / linux-focal-cuda12.4-py3.10-gcc9-experimental-split-build-test / test (nogpu_NO_AVX2, 1, 1, linux.2xlarge)

github-actions[bot] commented 2 months ago

These jobs stopped failing:

pull / linux-focal-py3.11-clang10 / test (crossref, 2, 2, linux.2xlarge)

github-actions[bot] commented 2 months ago

These jobs started failing:

trunk / linux-focal-cuda12.4-py3.10-gcc9-experimental-split-build-test / test (default, 4, 5, linux.4xlarge.nvidia.gpu)

github-actions[bot] commented 2 months ago

These jobs stopped failing:

pull / linux-focal-py3.11-clang10 / test (default, 3, 4, linux.2xlarge)
pull / linux-focal-py3.12-clang10 / test (dynamo, 3, 3, linux.2xlarge)

These jobs started failing:

pull / linux-focal-py3.9-clang10 / test (default, 3, 4, linux.2xlarge)

github-actions[bot] commented 2 months ago

These jobs stopped failing:

pull / linux-focal-py3.9-clang10 / test (default, 2, 4, linux.2xlarge)

These jobs started failing:

pull / linux-focal-py3.12-clang10-experimental-split-build / test (default, 2, 3, linux.2xlarge)

github-actions[bot] commented 2 months ago

These jobs stopped failing:

pull / linux-jammy-py3.9-gcc11 / test (default, 1, 4, linux.2xlarge)

These jobs started failing:

pull / linux-focal-py3.11-clang10 / test (crossref, 1, 2, linux.2xlarge)

github-actions[bot] commented 2 months ago

These jobs stopped failing:

pull / linux-focal-py3.12-clang10-experimental-split-build / test (default, 1, 3, linux.2xlarge)

github-actions[bot] commented 2 months ago

These jobs stopped failing:

trunk / linux-focal-cuda12.4-py3.10-gcc9-experimental-split-build-test / test (default, 4, 5, linux.4xlarge.nvidia.gpu)
trunk / linux-focal-cuda12.4-py3.10-gcc9-experimental-split-build-test / test (nogpu_NO_AVX2, 1, 1, linux.2xlarge)

github-actions[bot] commented 2 months ago

These jobs started failing:

trunk / linux-focal-cuda12.4-py3.10-gcc9-experimental-split-build-test / test (nogpu_NO_AVX2, 1, 1, linux.2xlarge)

github-actions[bot] commented 2 months ago

These jobs stopped failing:

periodic / parallelnative-linux-jammy-py3.9-gcc11 / test (default, 3, 3, linux.2xlarge)

github-actions[bot] commented 2 months ago

These jobs stopped failing:

pull / linux-focal-py3.11-clang10 / test (crossref, 1, 2, linux.2xlarge)
pull / linux-focal-py3.11-clang10 / test (dynamo, 3, 3, linux.2xlarge)
pull / linux-focal-py3.12-clang10-experimental-split-build / test (default, 2, 3, linux.2xlarge)
pull / linux-focal-py3.12-clang10-experimental-split-build / test (dynamo, 3, 3, linux.2xlarge)
pull / linux-focal-py3.9-clang10 / test (crossref, 2, 2, linux.2xlarge)
pull / linux-focal-py3.9-clang10 / test (default, 3, 4, linux.2xlarge)
pull / linux-focal-py3.9-clang10 / test (dynamo, 3, 3, linux.2xlarge)
trunk / linux-focal-cuda12.4-py3.10-gcc9-experimental-split-build-test / test (nogpu_AVX512, 1, 1, linux.2xlarge)
trunk / linux-focal-cuda12.4-py3.10-gcc9-experimental-split-build-test / test (nogpu_NO_AVX2, 1, 1, linux.2xlarge)

github-actions[bot] commented 2 months ago

These jobs started failing:

pull / linux-focal-py3.12-clang10-experimental-split-build / test (dynamo, 3, 3, linux.2xlarge)
pull / linux-focal-py3.9-clang10 / test (dynamo, 3, 3, linux.2xlarge)

github-actions[bot] commented 2 months ago

These jobs started failing:

pull / linux-focal-py3.11-clang10 / test (dynamo, 3, 3, linux.2xlarge)
pull / linux-focal-py3.12-clang10 / test (dynamo, 2, 3, linux.2xlarge)
pull / linux-focal-py3.12-clang10 / test (dynamo, 3, 3, linux.2xlarge)

github-actions[bot] commented 2 months ago

These jobs started failing:

pull / linux-focal-py3.12-clang10 / test (default, 2, 4, linux.2xlarge)

github-actions[bot] commented 2 months ago

These jobs stopped failing:

pull / linux-focal-py3.9-clang10 / test (dynamo, 3, 3, linux.2xlarge)

These jobs started failing:

pull / linux-focal-py3.12-clang10 / test (default, 4, 4, linux.2xlarge)
pull / linux-focal-py3.12-clang10-experimental-split-build / test (default, 2, 3, linux.2xlarge)
pull / linux-focal-py3.9-clang10 / test (default, 3, 4, linux.2xlarge)

github-actions[bot] commented 2 months ago

These jobs stopped failing:

pull / linux-focal-py3.12-clang10 / test (dynamo, 2, 3, linux.2xlarge)
pull / linux-focal-py3.9-clang10 / test (default, 3, 4, linux.2xlarge)

These jobs started failing:

pull / linux-focal-py3.9-clang10 / test (crossref, 2, 2, linux.2xlarge)

github-actions[bot] commented 2 months ago

These jobs started failing:

pull / linux-focal-py3.11-clang10 / test (crossref, 1, 2, linux.2xlarge)
pull / linux-focal-py3.11-clang10 / test (default, 2, 4, linux.2xlarge)

github-actions[bot] commented 2 months ago

These jobs stopped failing:

pull / linux-focal-py3.12-clang10-experimental-split-build / test (default, 2, 3, linux.2xlarge)

These jobs started failing:

pull / linux-focal-py3.11-clang10 / test (crossref, 2, 2, linux.2xlarge)
pull / linux-focal-py3.9-clang10 / test (crossref, 1, 2, linux.2xlarge)
trunk / linux-focal-cuda12.4-py3.10-gcc9-experimental-split-build-test / test (nogpu_NO_AVX2, 1, 1, linux.2xlarge)

github-actions[bot] commented 2 months ago

These jobs started failing:

trunk / linux-focal-cuda12.4-py3.10-gcc9-experimental-split-build-test / test (nogpu_AVX512, 1, 1, linux.2xlarge)

github-actions[bot] commented 2 months ago

These jobs stopped failing:

trunk / linux-focal-cuda12.4-py3.10-gcc9-experimental-split-build-test / test (nogpu_AVX512, 1, 1, linux.2xlarge)
trunk / linux-focal-cuda12.4-py3.10-gcc9-experimental-split-build-test / test (nogpu_NO_AVX2, 1, 1, linux.2xlarge)

These jobs started failing:

pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 5, 5, linux.4xlarge.nvidia.gpu)

github-actions[bot] commented 2 months ago

These jobs started failing:

trunk / linux-focal-cuda12.4-py3.10-gcc9-experimental-split-build-test / test (nogpu_AVX512, 1, 1, linux.2xlarge)
trunk / linux-focal-cuda12.4-py3.10-gcc9-experimental-split-build-test / test (nogpu_NO_AVX2, 1, 1, linux.2xlarge)
trunk / linux-focal-cuda12.4-py3.10-gcc9-sm86 / test (default, 2, 5, linux.g5.4xlarge.nvidia.gpu)

github-actions[bot] commented 2 months ago

These jobs started failing:

pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 4, 5, linux.g5.4xlarge.nvidia.gpu)

github-actions[bot] commented 2 months ago

These jobs started failing:

trunk / linux-focal-cuda12.4-py3.10-gcc9-sm86 / test (default, 4, 5, linux.g5.4xlarge.nvidia.gpu)

Next