search

grafana / synthetic-monitoring-app

Synthetic Monitoring frontend application
GNU Affero General Public License v3.0
144 stars 19 forks source link

Warning about leaving too small a window between checks #865

Open peterschretlen opened 4 months ago

peterschretlen commented 4 months ago

The planned infrastructure for scripted checks GA will have a retry mechanism when internal failures happen (these may happen when a new agent version is deployed to public probes for example)

However these retries will only happen until the next scheduled execution. For example if a synthetic check runs on a 1 minute interval and has a run-time of close to 1 minute there is no window for retries.

We should warn when the check interval and timeout suggest there is not enough buffer between checks to allow for retries. The risk is that there may be the ocassional gaps between test runs when they are not spaced far enough apart.

peterschretlen commented 4 months ago
  1. What are the numbers we should warn on?
  2. Wording this in such a way that it will make sense to users. "Recommendation or best practice to leave some buffer between test executions"
peterschretlen commented 1 month ago

Do we "recommend" a buffer, or "enforce" a buffer.