bug: api-breaker - concurrent traffic skip some breaker time steps

[zhoujiexiong]

Current Behavior

In case of continuous failure, some breaker time steps are skipped. The higher the degree of concurrency, the more obvious the problem.

Expected Behavior

In case of continuous failure, breaker time steps should not be skipped even in the high concurrent scenario.

Error Logs

Steps to Reproduce

=== TEST 2: [reproduce] concurrent traffic skip some breaker time steps
--- config
    location /t {
        content_by_lua_block {
            local t = require("lib.test_admin").test

            -- test configs
            local reqs = 128 / 4
            local breaker_times = { 2 }
            local uri = "/api_breaker?code=500"

            local function concurrent_traffic()
                local threads = {}
                for i = 1, reqs do
                    local t = ngx.thread.spawn(function()
                        t(uri, ngx.HTTP_GET)
                    end)
                    table.insert(threads, t)
                end
                local start_time = ngx.now()
                for i = 1, #threads do
                    ngx.thread.wait(threads[i])
                end
                local elapsed = ngx.now() - start_time
                assert(elapsed <= breaker_times[1],
                       string.format("concurrent traffic should finish in the " ..
                                     "time of not more than the smallest breaker_time(%d), " ..
                                     "but elapsed %f seconds", breaker_times[1], elapsed))
            end

            -- many 500 & 599
            concurrent_traffic()

            -- 599: breaking
            local code = t(uri, ngx.HTTP_GET)
            ngx.say(code)

            -- expecte breaking in 2 seconds
            ngx.sleep(2+1)

            -- 599: expect breaking timeout, but the 2s step is skipped, still breaking...
            code = t(uri, ngx.HTTP_GET)
            ngx.say(code)
        }
    }
--- request
GET /t
--- response_body
599
599

Environment

• APISIX version (3.9.0):
• Operating system (Linux 5.15.0-112-generic):
• OpenResty / Nginx version (nginx version: openresty/1.25.3.1):

[zhoujiexiong]

@dosu-bot WDYT