kong lua-resty-lock lock timeout 500 error {"message":"An unexpected error occurred"}

dubuqingfeng commented 5 months ago

Is there an existing issue for this?

[X] I have searched the existing issues

Kong version (`$ kong version`)

3.0.0

Current Behavior

Few requests： {"message":"An unexpected error occurred"} status: 500

Expected Behavior

Normal

Steps To Reproduce

Few requests

500 log:

remote_addr: 10.1.33.25 realip_remote_addr: 10.1.33.25 upstream_addr: - body_bytes_sent: 42 time_local: 27/Apr/2024:01:06:47 request_method: POST upstream_host_name: HOST_NAME http_host: fxxx.com request_uri: /rpc http_user_agent: okhttp/4.9.3 remote_user: - upstream_status: - request_time: 5.017 rpc_method: request_length: 35 http_referer: - msec: 1714180007.468 http_x_forwarded_for: - servicename: service_name upstream_response_time: - request_body_head: server_protocol: HTTP/2.0 status: 500

Anything else?

After modifying the code, print relevant logs:

failed the initial dns/balancer resolve for 'UPSTREAM_HOST' with: failed to get from node cache: could not acquire callback lock: timeout

Depoly: Hybrid mode

https://github.com/openresty/lua-resty-lock

timeout Specifies the maximal waiting time (in seconds) for the lock method calls on the current object instance. You can specify up to 0.001 seconds. Default to 5 (seconds). This option value cannot be bigger than exptime. This timeout is to prevent a lock method call from waiting forever. You can specify 0 to make the lock method return immediately without waiting if it cannot acquire the lock right away.

https://github.com/Kong/kong/issues/12811

KONG_MEM_CACHE_SIZE=128m，This value has not been changed，so maybe 128 m

The code involved in 500 is below：

https://github.com/Kong/kong/blob/3.0.0/kong/runloop/handler.lua#L1011

local function balancer_execute(ctx)
  local balancer_data = ctx.balancer_data
  local ok, err, errcode = balancer.execute(balancer_data, ctx)
  if not ok and errcode == 500 then
    err = "failed the initial dns/balancer resolve for '" ..
          balancer_data.host .. "' with: " .. tostring(err)
  end
  return ok, err, errcode
end

chronolaw commented 4 months ago

Kong 3.0 is EOL, could you try the latest 3.6 or 3.4 LTS? thanks.

dubuqingfeng commented 4 months ago

It can be confirmed that this is a problem, but there is no way to confirm whether 3.6 has fixed this, because our upgrade process is quite long. Can you explain the reason for this problem

github-actions[bot] commented 4 months ago

This issue is marked as stale because it has been open for 14 days with no activity.

ms2008 commented 4 months ago

This can be mitigated by #12956.

github-actions[bot] commented 3 months ago

This issue is marked as stale because it has been open for 14 days with no activity.

dubuqingfeng commented 3 months ago

@ms2008 Are there any updates?

github-actions[bot] commented 3 months ago

This issue is marked as stale because it has been open for 14 days with no activity.

github-actions[bot] commented 2 months ago

Dear contributor,

We are automatically closing this issue because it has not seen any activity for three weeks. We're sorry that your issue could not be resolved. If any new information comes up that could help resolving it, please feel free to reopen it.

Your contribution is greatly appreciated!

Please have a look our pledge to the community for more information.

Sincerely, Your Kong Gateway team

Kong / kong