unhandled `:gun_error` for websocket connection needing to be established before pushing

[the-mikedavis]

unknown message {:gun_error, #PID<0.16322.1>, {:badstate, 'Connection needs to be upgraded to Websocket before the gun:ws_send/1 function can be used.'}}
heard in Slipstream.Connection
please open an issue in NFIBrokerage/slipstream with this message and
any available information.

starting to feel a need for #4 so we can understand how a connection gets in a state where this happens

[the-mikedavis]

closed by https://github.com/NFIBrokerage/slipstream/commit/9d8fd35d6d6917b739e279d81a6f452afcfc5346

[the-mikedavis]

of course the first try finds us a nice :gun_error once the changes from #14 (see also #13) were merged. as you can see in the trace above (going bottom to top in reverse chronological order):

• SendHeartbeat command that got translated into a ChannelClosed event with reason: :heartbeat_timeout
• gun connection is up (this comes from a retry mechanism in gun)
• gun connection is down
• :gun_error about us not being able to send the heartbeat from 2 spans up (2 bullets down in this list)
• gun connection is up
• SendHeartbeat command that would fail with :gun_error on the send

I chaos-monkeyed this failure state by performing a rollout-restart (kubernetes) on the back-end to this front-end (the server to this client). (That back-end service has a RollingUpdate recreation strategy)

downtime for this bug would have been minimal, but leaving the above case unhandled would allow up to the heartbeat-timeout interval in 'dead' time for the connection. by fixing this we fallback from gun retry strategies to slipstream retry strategies, which is potentially faster because of the heartbeat-timeout mechanism

---

:point_up: that graph there comes from https://honeycomb.io, with connection telemetry shipped by our NFIBrokerage/slipstream_honeycomb adapter