Fill out the template below. Any pull request that does not include enough information to be reviewed in a timely manner may be closed at the maintainers' discretion.
The pull request must only fix one issue at the time.
The pull request must update the test suite to demonstrate the changed functionality.
After you create the pull request, all status checks must be pass before a maintainer reviews your contribution. For more details, please see CONTRIBUTING.
What does this PR do?
Adds exponential backoff to re-initialisation of HTTP logger, StatsD client and HTTP client that are used to communicate with the agent.
A customer might misconfigure their agent integration, in which case one or more of the clients listed above will not be able to initialise properly.
If a client is not fully initialised, every request to it will lead to a re-init attempt.
This is problematic, as initialising these clients is expensive.
The problem is especially pronounced with the logger, as it receives the biggest number of request.
This constant reinitialisation loop can consume a lot of resources (depending on the customer's workload) and lead to severe performance degradation.
While the plugin is not expected to work correctly if it is misconfigured, it should degrade gracefully and should not affect customer's jobs and pipelines.
Description of the Change
This change adds exponential back-off to initialisation of the three clients listed above.
There is now a delay between re-initialisation attempts, starting at 100 ms and doubling every time initialisation fails, up to a maximum of 60 seconds.
In case initialisation is successful, the delay is reset back to 100 ms so that subsequent errors lead to faster re-tries.
Alternate Designs
Possible Drawbacks
In case there is an intermittent network issue, clients will take longer to re-initialise (up to 1 minute in the absolute worst case, that is if the connectivity is lost for a period longer than this).
Verification Process
Verified manually with a dockerized Jenkins instance.
Additional Notes
Release Notes
Review checklist (to be filled by reviewers)
[ ] Feature or bug fix MUST have appropriate tests (unit, integration, etc...)
[ ] PR title must be written as a CHANGELOG entry (see why)
[ ] Files changes must correspond to the primary purpose of the PR as described in the title (small unrelated changes should have their own PR)
[ ] PR must have one changelog/ label attached. If applicable it should have the backward-incompatible label attached.
[ ] PR should not have do-not-merge/ label attached.
[ ] If Applicable, issue must have kind/ and severity/ labels attached at least.
Requirements for Contributing to this repository
What does this PR do?
Adds exponential backoff to re-initialisation of HTTP logger, StatsD client and HTTP client that are used to communicate with the agent.
A customer might misconfigure their agent integration, in which case one or more of the clients listed above will not be able to initialise properly. If a client is not fully initialised, every request to it will lead to a re-init attempt. This is problematic, as initialising these clients is expensive. The problem is especially pronounced with the logger, as it receives the biggest number of request. This constant reinitialisation loop can consume a lot of resources (depending on the customer's workload) and lead to severe performance degradation.
While the plugin is not expected to work correctly if it is misconfigured, it should degrade gracefully and should not affect customer's jobs and pipelines.
Description of the Change
This change adds exponential back-off to initialisation of the three clients listed above. There is now a delay between re-initialisation attempts, starting at 100 ms and doubling every time initialisation fails, up to a maximum of 60 seconds. In case initialisation is successful, the delay is reset back to 100 ms so that subsequent errors lead to faster re-tries.
Alternate Designs
Possible Drawbacks
In case there is an intermittent network issue, clients will take longer to re-initialise (up to 1 minute in the absolute worst case, that is if the connectivity is lost for a period longer than this).
Verification Process
Verified manually with a dockerized Jenkins instance.
Additional Notes
Release Notes
Review checklist (to be filled by reviewers)
changelog/label attached. If applicable it should have thebackward-incompatiblelabel attached.do-not-merge/label attached.kind/andseverity/labels attached at least.