[Heartbeat] Perform last state request exactly once for retry/transition scenario

[vigneshshanmugam]

Summary

Heartbeat uses state loader to get the last status from the ES cluster and loads the current monitor state after the monitor has been successfully ran. Once the monitor run is successfully completed, we are performing two requests to the ES to get the last known state for the monitor

• First time, before we figure out the retry/max_attempts for the monitor
• Second time, transitioning to the new monitor state.

These double requests affects the execution time of the monitor when running in tight schedules on the Synthetics SAAS service. We can optimize this and perform the last state check only once and use that effectively on both places.

Relevant code - https://github.com/elastic/beats/blob/fe9b7ce5a9180ba820d1c0719fe12f8cf7738917/heartbeat/monitors/wrappers/summarizer/plugstatestat.go#L154

[elasticmachine]

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

[emilioalvap]

@vigneshshanmugam there shouldn't be two requests for a single monitor, even through retries. States should be cached underneath

Do we have an example trace of the double request in action?

[vigneshshanmugam]

We discussed this on slack, The problem happened because of error responses and we are delaying the next executions by periodically retrying these failed requests. We need a strategy to short circuit these requests - https://github.com/elastic/beats/issues/37424

I am closing this as we can track these failed requests and short circuiting strategy on the other issue.