Test failure: `TestStorageTieringContinueInxMigration.test_put_object_limit_lt`

[alanking]

I have observed two modes of failure for test_plugin_unified_storage_tiering.TestStorageTieringContinueInxMigration.test_put_object_limit_lt as of commit 511cba8e55268bce8cb2f254fd2cbe4a2fe814d6 (NOTE: This commit is not necessarily the commit which introduced the problem, just the earliest one with which I tested).

1. The test intermittently fails in about the same amount of time as a passing test (~135 seconds).
2. The test intermittently hangs forever after migrating a few objects out of the 255 objects it was supposed to migrate.

We need to investigate whether this is due to the test or if this is a real issue. We at least need to ensure that the test will fail eventually by removing the infinite loop found here: https://github.com/irods/irods_capability_storage_tiering/blob/511cba8e55268bce8cb2f254fd2cbe4a2fe814d6/packaging/test_plugin_unified_storage_tiering.py#L162-L171

[alanking]

For more context, I see hundreds of messages like this in the log:

{
  "log_category": "database",
  "log_level": "error",
  "log_message": "Could not find a delay rule with id [10356].",
  "request_api_name": "",
  "request_api_number": 20013,
  "request_api_version": "d",
  "request_client_user": "rods",
  "request_host": "172.25.0.3",
  "request_proxy_user": "rods",
  "request_release_version": "rods4.3.1",
  "server_host": "2d81d8c90eae",
  "server_pid": 2505,
  "server_timestamp": "2023-12-19T22:54:19.628Z",
  "server_type": "agent",
  "server_zone": "tempZone"
}
{
  "log_category": "api",
  "log_level": "error",
  "log_message": "Could not get delay rule information [rule id=[10356]]",
  "request_api_name": "",
  "request_api_number": 20013,
  "request_api_version": "d",
  "request_client_user": "rods",
  "request_host": "172.25.0.3",
  "request_proxy_user": "rods",
  "request_release_version": "rods4.3.1",
  "server_host": "2d81d8c90eae",
  "server_pid": 2505,
  "server_timestamp": "2023-12-19T22:54:19.628Z",
  "server_type": "agent",
  "server_zone": "tempZone"
}