[TT-2372 ] - CircuitBreaker cooldown does not work

[maciejwojciechowski]

Branch/Environment/Version

• Branch/Version: release-1.8 / release-1.9
• Environment: on-prem

Describe the bug Seems that the CB cooldown time defined on API level return_to_service_after does not work. My cooldown is 60 seconds and CB was tripped but I still see some of my requests go thru to upstream.

In gateway logs I see the CB is OPEN and then CLOSED again (all in just 10 seconds)

[Nov 15 12:15:08] DEBUG ON REQUEST: Circuit Breaker is in CLOSED or HALF-OPEN state

[Nov 15 12:15:09] DEBUG ON REQUEST: Circuit Breaker is in CLOSED or HALF-OPEN state
[Nov 15 12:15:09]  WARN PROXY: [CIRCUIT BREAKER] Breaker tripped for path: /status/500

[Nov 15 12:15:10] DEBUG ON REQUEST: Circuit Breaker is in CLOSED or HALF-OPEN state
[Nov 15 12:15:10]  WARN PROXY: [CIRCUIT BREAKER] Breaker tripped for path: /status/500

[Nov 15 12:15:10] DEBUG ON REQUEST: Circuit Breaker is in OPEN state

[Nov 15 12:15:13] DEBUG ON REQUEST: Circuit Breaker is in CLOSED or HALF-OPEN state
[Nov 15 12:15:13]  WARN PROXY: [CIRCUIT BREAKER] Breaker tripped for path: /status/500

[Nov 15 12:15:16] DEBUG ON REQUEST: Circuit Breaker is in CLOSED or HALF-OPEN state
[Nov 15 12:15:16]  WARN PROXY: [CIRCUIT BREAKER] Breaker tripped for path: /status/500

[Nov 15 12:15:17] DEBUG ON REQUEST: Circuit Breaker is in OPEN state

[Nov 15 12:15:17] DEBUG ON REQUEST: Circuit Breaker is in CLOSED or HALF-OPEN state

[Nov 15 12:15:18] DEBUG ON REQUEST: Circuit Breaker is in OPEN state

[Nov 15 12:15:19] DEBUG ON REQUEST: Circuit Breaker is in OPEN state

[Nov 15 12:15:19] DEBUG ON REQUEST: Circuit Breaker is in CLOSED or HALF-OPEN state

Reproduction steps

1. Add API with Circuit Breaker, set return_to_service_after = 60 sec
2. Keep hitting that endpoint until CB is tripped.
3. Keep hitting it

Actual behavior The cooldown time is not working. After CB is tripped some of the requests continue to go through to upstream

Expected behavior When CB is tripped then no requests should go through during defined period of time (in my case return_to_service_after = 60 seconds)

Logs (debug mode or log file): gw_log.txt

API definition api_def.txt

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs, please add comments to this ticket if you would like it to stay open. Thank you for your contributions.

[Zavierazo]

+1 after CB tiggered recover with the first 200 request