search

ray-project / ray

Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
https://ray.io
Apache License 2.0
33.21k stars 5.61k forks source link

CI test linux://python/ray/dag:tests/experimental/test_accelerated_dag is flaky #45922

Closed can-anyscale closed 3 weeks ago

can-anyscale commented 3 months ago

CI test linux://python/ray/dag:tests/experimental/test_accelerated_dag is consistently_failing. Recent failures:

DataCaseName-linux://python/ray/dag:tests/experimental/test_accelerated_dag-END Managed by OSS Test Policy

can-anyscale commented 3 months ago

Blamed commit: d57765290c511cc5150587b0b0a2ba23960e7ba2 found by bisect job https://buildkite.com/ray-project/release-tests-bisect/builds/1230

can-anyscale commented 3 months ago

This test is now considered as flaky because it has been failing on postmerge for too long. Flaky tests do not run on premerge.

can-anyscale commented 3 months ago

blame reverted

can-anyscale commented 3 months ago

This test is now considered as flaky because it has been failing on postmerge for too long. Flaky tests do not run on premerge.

can-anyscale commented 3 months ago

This test is now considered as flaky because it has been failing on postmerge for too long. Flaky tests do not run on premerge.

can-anyscale commented 3 months ago

still failing; but likely not release blocking, let's confirm on Monday

ruisearch42 commented 3 months ago

These should pass now?

can-anyscale commented 3 months ago

@ruisearch42 hi yes, automation will close this issue automatically once it runs this test on the master branch; if the tests fail it will also comment here

can-anyscale commented 3 months ago

Test passed on latest run: https://buildkite.com/ray-project/postmerge/builds/5027#01903605-07ef-4724-9f64-532f5d6e6a15

can-anyscale commented 3 months ago

CI test linux://python/ray/dag:tests/experimental/test_accelerated_dag is flaky. Recent failures:

DataCaseName-linux://python/ray/dag:tests/experimental/test_accelerated_dag-END Managed by OSS Test Policy

can-anyscale commented 3 months ago

@stephanie-wang , @kevin85421 please remember to add the weekly-release-blocker tag back once it is fixed; otherwise it will never be blocking again; thanks

can-anyscale commented 3 months ago

Test passed on latest run: https://buildkite.com/ray-project/postmerge/builds/5048#01903931-0d21-426e-beba-8b67dd924546

can-anyscale commented 3 months ago

Test passed on latest run: https://buildkite.com/ray-project/postmerge/builds/5065#01903c93-d917-49b4-b82d-fc074a136033

can-anyscale commented 3 months ago

CI test linux://python/ray/dag:tests/experimental/test_accelerated_dag is consistently_failing. Recent failures:

DataCaseName-linux://python/ray/dag:tests/experimental/test_accelerated_dag-END Managed by OSS Test Policy

can-anyscale commented 3 months ago

CI test linux://python/ray/dag:tests/experimental/test_accelerated_dag is consistently_failing. Recent failures:

DataCaseName-linux://python/ray/dag:tests/experimental/test_accelerated_dag-END Managed by OSS Test Policy

can-anyscale commented 3 months ago

This test is now considered as flaky because it has been failing on postmerge for too long. Flaky tests do not run on premerge.

can-anyscale commented 3 months ago

Blamed commit: b582905987632b38a7711ae8512a66545b538c75 found by bisect job https://buildkite.com/ray-project/release-tests-bisect/builds/1265

can-anyscale commented 3 months ago

Blamed commit: b582905987632b38a7711ae8512a66545b538c75 found by bisect job https://buildkite.com/ray-project/release-tests-bisect/builds/1265

can-anyscale commented 3 months ago

this test is broken now but is not considered as release blocking, let me know otherwise ;) @ruisearch42

can-anyscale commented 3 months ago

This test is now considered as flaky because it has been failing on postmerge for too long. Flaky tests do not run on premerge.

can-anyscale commented 3 months ago

Test passed on latest run: https://buildkite.com/ray-project/postmerge/builds/5110#01904f98-f888-45e2-a868-9a074a4a0d25

can-anyscale commented 3 months ago

CI test linux://python/ray/dag:tests/experimental/test_accelerated_dag is consistently_failing. Recent failures:

DataCaseName-linux://python/ray/dag:tests/experimental/test_accelerated_dag-END Managed by OSS Test Policy

stephanie-wang commented 3 months ago

Probably because of #46154, let me take a quick look.

can-anyscale commented 3 months ago

also this test is currently not a release blocker; remember to add the weekly-release-blocker tag back if you want to stop us from releasing without it ;)

can-anyscale commented 3 months ago

CI test linux://python/ray/dag:tests/experimental/test_accelerated_dag is consistently_failing. Recent failures:

DataCaseName-linux://python/ray/dag:tests/experimental/test_accelerated_dag-END Managed by OSS Test Policy

can-anyscale commented 3 months ago

This test is now considered as flaky because it has been failing on postmerge for too long. Flaky tests do not run on premerge.

stephanie-wang commented 3 months ago

Thanks, added the tag back :)

stephanie-wang commented 3 months ago

Should be fixed by #46264.

can-anyscale commented 3 months ago

Blamed commit: d090d86468e9a54a6ad2403b1943412cdcbf231a found by bisect job https://buildkite.com/ray-project/release-tests-bisect/builds/1277

can-anyscale commented 3 months ago

This test is now considered as flaky because it has been failing on postmerge for too long. Flaky tests do not run on premerge.

can-anyscale commented 3 months ago

This test is now considered as flaky because it has been failing on postmerge for too long. Flaky tests do not run on premerge.

can-anyscale commented 3 months ago

Test passed on latest run: https://buildkite.com/ray-project/postmerge/builds/5157#01905a1f-f19f-4814-824d-3ab647d30227

can-anyscale commented 3 weeks ago

CI test linux://python/ray/dag:tests/experimental/test_accelerated_dag is consistently_failing. Recent failures:

DataCaseName-linux://python/ray/dag:tests/experimental/test_accelerated_dag-END Managed by OSS Test Policy

can-anyscale commented 3 weeks ago

CI test linux://python/ray/dag:tests/experimental/test_accelerated_dag is consistently_failing. Recent failures:

DataCaseName-linux://python/ray/dag:tests/experimental/test_accelerated_dag-END Managed by OSS Test Policy

can-anyscale commented 3 weeks ago

Test passed on latest run: https://buildkite.com/ray-project/postmerge/builds/6162#0191c5bb-5346-4301-9743-77b6f57f7bdc

can-anyscale commented 3 weeks ago

Test passed on latest run: https://buildkite.com/ray-project/postmerge/builds/6162#0191c5fd-ce5a-4def-8703-338b139229a3

can-anyscale commented 3 weeks ago

CI test linux://python/ray/dag:tests/experimental/test_accelerated_dag is flaky. Recent failures:

DataCaseName-linux://python/ray/dag:tests/experimental/test_accelerated_dag-END Managed by OSS Test Policy

can-anyscale commented 3 weeks ago

CI test linux://python/ray/dag:tests/experimental/test_accelerated_dag is flaky. Recent failures:

DataCaseName-linux://python/ray/dag:tests/experimental/test_accelerated_dag-END Managed by OSS Test Policy

can-anyscale commented 3 weeks ago

Test passed on latest run: https://buildkite.com/ray-project/postmerge/builds/6172#0191cae0-e5dc-4396-9a71-b21c781ea0a3

can-anyscale commented 3 weeks ago

Test passed on latest run: https://buildkite.com/ray-project/postmerge/builds/6173#0191cb45-ea88-4ba2-854c-e11ec7b38662