Bug fix: deadlocked worker pool threads during a controlplane connection failure.

[VirajSalaka]

Purpose

Bug Fix: When the retry limit reaches its maximum limit, the worker threads which calls the control plane gets blocked.

Issues

Fixes #

Automation tests

• Unit tests added: No
• Integration tests added: No

Tested environments

Locally tested in the following manner. Started choreo-product-apim locally then Started choreo-connect locally pointing to local choreo-product-apim and ASB connection string. (I used PDP mode in choreo-connect here) Stopped the choreo-product-apim so that we could replicate a network unreachable type failure. Sent deploy events via ASB

Monitored the behavior with and without the fix. And with the fix it did not blocked.

---

Maintainers: Check before merge

• [x] Assigned 'Type' label
• [x] Assigned the project
• [x] Validated respective github issues
• [ ] Assigned milestone to the github issue(s)

[choreo-cicd]

[succeeded] Dataplane(NorthEU) cluster : dev-deployment-v2 : 20240129.9

[choreo-cicd]

[succeeded] Dataplane(EastUS) cluster : dev-deployment-v2 : 20240129.9

[choreo-cicd]

[succeeded] Controlplane cluster : dev-deployment-v2 : 20240129.9

[choreo-cicd]

[succeeded] Dataplane(NorthEU) cluster : stage-deployment-v2 : 20240201.1

[choreo-cicd]

[succeeded] Dataplane(EastUS) cluster : stage-deployment-v2 : 20240201.1

[choreo-cicd]

[succeeded] Controlplane cluster : stage-deployment-v2 : 20240201.1

[choreo-cicd]

[succeeded] Controlplane cluster : prod-deployment-v2 : 20240201.3

[choreo-cicd]

[succeeded] Dataplane(EastUS) cluster : prod-deployment-v2 : 20240201.3

[choreo-cicd]

[succeeded] Dataplane(NorthEU) cluster : prod-deployment-v2 : 20240201.3