search

ninenines / gun

HTTP/1.1, HTTP/2, Websocket client (and more) for Erlang/OTP.
ISC License
898 stars 231 forks source link

Continuous spawning of proc_lib_hack process #183

Closed halidm closed 5 years ago

halidm commented 5 years ago

We have tested gun in production over the past few days and noticed a very weird behavior.

The way we setup gun was through a pool of processes that connect on init to the server, with anything between 30-300 connections (pre-set). Process send request and receives proper gun response. However, for no obvious reason, after running the application for 30-40mins all of a sudden, process numbers and open ports start growing--to be more precise (proc_lib_hack). And within minutes we reach 30000 processes instead of the original 300 and of course the CPU is at 100%.

The load that is using these processes during our tests was negligible 30 reqs/sec.

We are not that versed in debugging gun, but it seems as if it gets stuck in retries and as if it is not happy anymore with existing open connections.

PS: with another http client (hackney) we haven't seen this issue

essen commented 5 years ago

If you see processes growing then that's not the retry. Processes are only started when the gun:open function is called, and restarted when they crash.

essen commented 5 years ago

Could it be that your pool restarts a connection via gun:open or equivalent when the connection is down, and you didn't configure retry to 0 to prevent Gun to reconnect? That would lead to that issue.

Either way I don't think the issue is in Gun itself. Closing, thanks!