In the case where an entry in the cache was expired, if the data could not be refreshed by requesting it to the API, an error was returned. This implied a blocking behavior for various actions, such as:
Publish events: If tenant remotes could not be obtained.
These actions should continue to run even if the agents lose connection with the API, or the API is behaving abnormally. Therefore, this change favours returning stale tenants data from the cache in case it can not be refreshed by requesting it to the API.
The worst case scenario for each mentioned case after this change is:
Check execution: The check will be executed with limits that might have changed.
Publish: The metrics and logs will be pushed using remotes that might have changed.
Both cases can lead to metrics and logs backends to reject our requests. But these changes happen infrequently, so most often the data returned by the cache will still be valid and checks will be able to run even if the agent is disconnected from the API. And even in the case on which these errors happen, is better than blocking preventively.
In the case where an entry in the cache was expired, if the data could not be refreshed by requesting it to the API, an error was returned. This implied a blocking behavior for various actions, such as:
These actions should continue to run even if the agents lose connection with the API, or the API is behaving abnormally. Therefore, this change favours returning stale tenants data from the cache in case it can not be refreshed by requesting it to the API.
The worst case scenario for each mentioned case after this change is:
Both cases can lead to metrics and logs backends to reject our requests. But these changes happen infrequently, so most often the data returned by the cache will still be valid and checks will be able to run even if the agent is disconnected from the API. And even in the case on which these errors happen, is better than blocking preventively.