I frequently see recurring test flakes, most often at the cleanup stage of test_configure_index_with_deletion_protection because the index cannot be deleted while still in an "upgrading" state. The index is in that state while configuration changes are being applied.
Solution
In deletion protection test, wait for the index to be ready before attempting the delete.
In dependency tests which do a basic sanity test, loosen the assertions. We're not really intending to test the backend functionality with these tests; we just want to make sure network calls are successfully made with a variety of different grpc / protobuff / etc dependency versions.
Remove some assertions that rely on non-deterministic behavior on the backend. E.g. how long it takes for a serverless index to be considered ready.
Check for deletion protection status before trying to delete.
Don't fail the job when cleanup fails; even if we re-run the entire job, there will still be an orphaned index from a previous run that needs to be cleaned up. Best to just attempt to delete, then let the nightly cleanup job garbage collect anything left over.
Type of Change
[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
Problem
I frequently see recurring test flakes, most often at the cleanup stage of
test_configure_index_with_deletion_protectionbecause the index cannot be deleted while still in an "upgrading" state. The index is in that state while configuration changes are being applied.Solution
Type of Change