Not receiving/reading event type leads to stream server crash

[Tuxified]

Describe the bug We recently found an error log indicating shotgun is not passing (or receiving) the event type for SSE. It passed unnoticed for some time as it only happens in our staging env, on which we don't keep a close eye.

To reproduce Use latest Erlang SDK, wait a bit. The eld_stream_server process crashes because it receives empty data, without an event type.

Expected behavior No crashes

Logs (eld) /tmp/nix-project.drv-0/project-9d01937/deps/eld/src/eld_stream_server.erl:195: :eld_stream_server.process_event(%{data: ""}, :eld_storage_ets, :default) (stdlib) lists.erl:1338: :lists.foreach/2 (shotgun) src/shotgun.erl:661: :shotgun.manage_chunk/4 (shotgun) src/shotgun.erl:582: :shotgun.receive_chunk/2 (stdlib) gen_fsm.erl:497: :gen_fsm.handle_msg/8 (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3 (shotgun) :shotgun.init(:Argument__1)

SDK version 1.0.0-alpha3

Language version, developer tools Erlang/OTP 22 [erts-10.4.3], Elixir 1.9.1, nix package manager

Additional context I'll try to find out what the real culprit is, but I wanted to notify you in the meantime :)

[unix1]

@Tuxified thank you for reporting. A couple of questions:

• This looks like a function_clause crash at eld_stream_server:process_event, is that correct - did the logs show that?

• If yes, the eld_stream_server should be restarted by the supervisor and attempt to create a new shotgun SSE connection. Were you able to observe whether the process would crash continuously as eld_stream_server would keep attempting to reconnect unsuccessfully; or was it able to keep the normal streaming connection up, only crashing once so often? If it's the former, I believe we know the cause (now reverted until I can make the shotgun PR and get it released); if latter, it's not yet known and I'll have to try to reproduce.

[zurab-darkly]

I believe inaka/shotgun#182 was probably the root cause of this.

[aj-foster]

Hello there. 👋 We recently encountered this error as well.

Were you able to observe whether the process would crash continuously as eld_stream_server would keep attempting to reconnect unsuccessfully; or was it able to keep the normal streaming connection up, only crashing once so often?

For us, the crashes were continuous.

[zurab-darkly]

@aj-foster thanks for reporting and sorry for the trouble. You are correct - this is the same issue. The quick and dirty fix is to override the dependencies to use https://github.com/inaka/shotgun/ master branch. We'll have a release of the SDK with the fix out shortly, and we'll revert the SSE update in our service to avoid breaking existing SDKs.

[aj-foster]

@zurab-darkly Thank you! The quick fix worked for us, and we'll keep an eye out for the release of the SDK. 🙂

[zurab-darkly]

@aj-foster @Tuxified this is now fixed in 1.0.0-beta3 release. Thank you for reporting again.