Adding a more flexible retry function to the retryService

[maryam-saeidi]

Summary

In this PR, the security team (@jpdjere) added a retry utility that is very flexible and it accepts parameters such as:

• timeout
• retries
• retryDelay

I copied this utility to our observability API integration tests, and I did some adjustment regarding logging as we don't need to have error logs in case of a failure attempt (I prefer the logging that we have in retryService instead.)

I think it would be good to have this code shared between different tests, either in a package or maybe moving it to the retryService itself.

The end goal is to have this functionality shared between tests and replace all the instances of pRetry with this utility instead: (Second item in the description of this PR)

• In case of throwing an error in pRetry, when we have 10 retries, it does not log the retry attempts and we end up in the situation that is mentioned in this comment, item 3
  • This can be related to exponential backoff stated here and not logging the attempts until either timeout or successful try.

    The example below will retry a potentially failing dns.resolve operation 10 times using an exponential backoff strategy. With the default settings, this means the last attempt is made after 17 minutes and 3 seconds.

Before	After

[elasticmachine]

Pinging @elastic/appex-qa (Team:QA)

[dmlemeshko]

I agree it is a useful improvement. I also think that having retry capabilities within single service is easy to support and re-use across different tests in the long run, opened #178660

[maryam-saeidi]

@dmlemeshko Shall we close this ticket? I think you covered it in your PR, right?