ray-project / kuberay

A toolkit to run Ray applications on Kubernetes
Apache License 2.0
990 stars 330 forks source link

[Test][RayCluster] Test redis cleanup job in the e2e compatibility test #2026

Closed rueian closed 3 months ago

rueian commented 3 months ago

Why are these changes needed?

As discussed with @kevin85421 offline, we think that we should test the redis cleanup job in the tests/compatibility-test.py.

This PR appends the cleanup_redis procedure to every test in the RayFTTestCase to test the cleanup job by deleting the RayCluster CR and watching the Redis' DBSIZE to zero. If it doesn't go zero before timeout, then the test will fail.

Related issue number

Fixes https://github.com/ray-project/kuberay/issues/1997

Checks

kevin85421 commented 3 months ago

The CI stuck for 10 hours. I cancel it.

Screen Shot 2024-03-20 at 3 15 32 PM Screen Shot 2024-03-20 at 3 16 56 PM
rueian commented 3 months ago

The CI stuck for 10 hours. I cancel it.

Screen Shot 2024-03-20 at 3 15 32 PM Screen Shot 2024-03-20 at 3 16 56 PM

I will look into why did it stuck.