Open msmykowski opened 3 years ago
So, this is interesting. http/2 connections definitely can't handle >1 million requests over their lifetime, but the GenStage implementation should properly queue while it sets up a new connection. It's been a while since I tested it, and my production workloads generally don't hit that limit very often.
I'll dig a little bit and see what's up. If you're sending that kind of volume though, I'd recommend starting multiple workers and round-robining between them. I may consider adding a pool_size
option in v2 to make this easier, but most workloads typically run just fine on one connection.
@hpopp If it helps we identified the race condition happening here -- https://github.com/codedge-llc/pigeon/blob/master/lib/pigeon/connection.ex#L93. Since Genstage.cancel
is async, at a high enough throughput this guarantees messages will be processed with a nil
socket.
Our quick patch was to reopen a new socket when the old one is closed rather than setting the socket to nil
and shutting down the consumer process. It has fixed the issue posted above, but I believe there is still a condition where push notifications can be pushed into a kadabra connection just prior to it being closed, resulting in on_response
not being called.
Environment
elixir --version
): 1.11.2Current behavior
We send a high volume of alerts in the magnitude of millions in single bursts. We are seeing very frequent FCM connection crashes resulting in dropped messages.
We are starting our connection in a bit of a dated way within a Genserver.
Expected behavior
We would expect the connection to be reestablished before attempting to send the request.