Closed binaryseed closed 4 years ago
@benoitc any debugging pointers?
probably a race condition. i'm investigating
FYI, I was able to create a situation where the errors stopped happening by increasing max_connections
from the default value to 200. Looking at the metrics emitted by hackney that makes sense because in_use_count
now never hits 200 so we never have to queue in the first place
It also makes me think it might be easier to reproduce by lowering max_connections
...
@benoitc any update on this? I've started to see this issue myself with the default pool in 1.15.1
. Thanks.
@alinpopa how/when does it happen? More generally there is a WIP on fixing connection reusing, i will provide a more detailled status tomorrow, should have been earlier but I've been side-tracked.
@benoitc it's a bit difficult to know exactly how it happened, but the only thing I can tell you (from my logs), is that at that time we had a high requests load (again, don't really have numbers to back me up on this), and then it crashed with this:
no case clause matching: {:EXIT, {{{:badmatch, {:empty, {[], []}}},
[{:hackney_pool, :queue_out, 2, [file: '/app/deps/hackney/src/hackney_pool.erl', line: 509]},
{:hackney_pool, :deliver_socket, 3, [file: '/app/deps/hackney/src/hackney_pool.erl', line: 524]},
{:hackney_pool, :handle_call, 3, [file: '/app/deps/hackney/src/hackney_pool.erl', line: 294]}, {:gen_server,
:try_handle_call, 4, [file: 'gen_server.erl', line: 661]}, {:gen_server, :handle_msg, 6, [file: 'gen_server.erl', line:
690]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 249]}]}, {:gen_server, :call, [#PID<0.3296.0>,
{:checkout, {'.....', 443, :hackney_ssl}, #PID<0.30153.1917>,
#Reference<0.692853367.1751121921.124719>}, 8000]}}}
Hope this helps. Thanks.
fixed in cd96ce79cee73ae8294b8b42249f6923f866a105
I'm seeing a match error for somewhat sporadically... (Elixir stacktrace):
https://github.com/benoitc/hackney/blob/master/src/hackney_pool.erl#L504-L517
It looks like the call to
queue:out
on L509 is returning{:empty, _queue}
but it's always expecting there to be a valueHackney
1.15.2
Erlang/OTP22.1.3
Any ideas what could cause this situation?