Always cleanup cloud indexes from tests

[vicilliar]

• Please check if the PR fulfills these requirements

• [x] The commit message follows our guidelines

• [ ] Tests for the changes have been added (for bug fixes/features)

• [ ] Docs have been added / updated (for bug fixes / features)

• What kind of change does this PR introduce? (Bug fix, feature, docs update, ...) Bug fix

• What is the current behavior? (You can also link to an open issue here) When cloud integration test run is cancelled, indexes are not cleaned up.

• What is the new behavior (if this is a feature change)?

Several test quality of life updates: -Concurrency control. Only 1 pipeline can run at a time. Prevents overlapping indexes. -Use last 4 digits of github run ID for test unique identifier (instead of UUID) so it can be shared between jobs. -Run cleanup_indexes regardless of outcome of integration test. -delete indexes script now always has wait_for_readiness=True -Remove wait_for_readiness in the cleanup section of the run_integration_tests job. This is because cleanup_indexes job will always run and wait til completion. Also so the deletion time is not recorded in the test run section. -wait_for_readiness loop now also waits for CREATING, MODIFYING, DELETING indexes, then sends a new delete request when they are READY or FAILED. -Add timeouts for workflow: 1 hour for github workflow, 45 minutes for python tests.

• Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?) No

• Other information:

[vicilliar]

Proof that script will only clean up associated indexes (for this run).

• Only 2 indexes are attempting to be deleted, even if 7 are present in the account.
• Constantly waits for indexes to be READY or FAILED before sending delete request again. 30s sleep confirmed.
• Run seen here: https://github.com/marqo-ai/py-marqo/actions/runs/10917291511/job/30300318636

[vicilliar]

Test Case:

• 1 smooth successful run, with cleanup

• Test runs succeed, and cleanup indexes happens in another job. Works as intended. No timeout is hit. https://github.com/marqo-ai/py-marqo/actions/runs/10952799139/job/30412029355

[vicilliar]

1 Full Test run cancelled (5 indices deleted by cleanup script) Run: https://github.com/marqo-ai/py-marqo/actions/runs/10917619817/job/30301648628 ~37 minutes to delete all (including waiting for index to disappear from list

[vicilliar]

Cancel run with no indexes to delete:

[vicilliar]

Concurrency check (only 1 run is allowed at a time)

Works as intended.

[vicilliar]

Workflow TIMEOUT: After hitting 1 hour, with tests being stuck at test_config.py, entire test job failed out. Then Cleanup cloud indexes job was run automatically. This is intended behavior.

Will investigate why tests were stuck at test_config.py