[RayService] Address Recent Flakiness in RayService Zero Downtime Rollout Test

[Yicheng-Lu-llll]

Why are these changes needed?

Recently, RayService zero downtime rollout test becomes flaky:

• https://buildkite.com/ray-project/ray-ecosystem-ci-kuberay-ci/builds/3418#018dfb4f-8302-4e53-b41f-75d82dd3eb0d/243-2083
• https://buildkite.com/ray-project/ray-ecosystem-ci-kuberay-ci/builds/3416#018dfb25-22b7-4b2c-8978-f2a56721b47c/6-759
• https://buildkite.com/ray-project/ray-ecosystem-ci-kuberay-ci/builds/3372#018ded1d-5412-486a-ad0a-0224040c2c36/243-1998

https://github.com/ray-project/kuberay/pull/1837 describes the root cause and introduces a hotfix. My previous pr https://github.com/ray-project/kuberay/pull/1928 attempts to address the issue and remove the hotfix but as shown avoce, it actually fails to resolve them.

This PR introduces logic to ensure that, for zero-downtime rollout tests, during the period when the serve service has not been fully updated and the old RayCluster has not been deleted, RayServiceUpdateCREvent expects the query request to be processed by either the old or the new RayCluster.

Related issue number

Checks

• [ ] I've made sure the tests are passing.
• Testing Strategy
  • [ ] Unit tests
  • [ ] Manual tests
  • [ ] This PR is not tested :(

[Yicheng-Lu-llll]

I have triggered the CI test 5 times, and the RayService test has passed every time. The RayService Zero Downtime Rollout Test seems stable now. Note, the failed CI tests are RayCluster tests, which are unrelated to this PR.