[Wocky/us1] ErlangError: Erlang error: :noproc

[bernardd]

Backtrace

View full backtrace and more info at honeybadger.io

[bernardd]

This corresponds to an error in the logs caused by the push notification connection process having died:

{ json: { msg: "GenServer #PID<0.17890.9> terminating ** (stop) exited in: GenServer.call(#PID<0.17891.9>, {:request, [%Kadabra.Request{body: "{\"uri\":\"tinyrobot://bot/us1.prod.tinyrobot.com/08471fc0-7ec7-11e8-b7e0-0a580a0203ed/visitors\",\"aps\":{\"badge\":1,\"alert\":\"@sydnie is at Softball Field\"}}", headers: [{":method", "POST"}, {":path", "/3/device/d211275a85a8708c067d7eb106c5ac8910fdfa5d54c6d50c8162af5129c282f7"}, {"content-length", "151"}, {"apns-topic", "com.hippware.ios.Chat"}], on_response: nil}]}, 5000) ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started (elixir) lib/gen_server.ex:924: GenServer.call/3 (pigeon) lib/pigeon/connection.ex:145: Pigeon.Connection.send_push/3 (elixir) lib/enum.ex:1925: Enum."-reduce/3-lists^foldl/2-0-"/3 (pigeon) lib/pigeon/connection.ex:97: Pigeon.Connection.handle_events/3 (gen_stage) lib/gen_stage.ex:2315: GenStage.consumer_dispatch/6 (stdlib) gen_server.erl:637: :gen_server.try_dispatch/4 (stdlib) gen_server.erl:711: :gen_server.handle_msg/6 (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3 Last message: {:"$gen_consumer", {#PID<0.3748.0>, #Reference<0.2819615548.829685761.176736>}, [{:push, %Pigeon.APNS.Notification{collapse_id: nil, device_token: "d211275a85a8708c067d7eb106c5ac8910fdfa5d54c6d50c8162af5129c282f7", expiration: nil, id: nil, payload: %{"aps" => %{"alert" => "@sydnie is at Softball Field", "badge" => 1}, "uri" => "tinyrobot://bot/us1.prod.tinyrobot.com/08471fc0-7ec7-11e8-b7e0-0a580a0203ed/visitors"}, response: nil, topic: "com.hippware.ios.Chat"}, [on_response: #Function<1.110771641/1 in Wocky.Push.do_notify/4>]}]}", kubernetes: { container_name: "wocky", host: "ip-10-1-154-114.us-west-2.compute.internal", master_url: "https://10.3.0.1:443/api", pod_id: "9cc00a66-a261-11e8-880f-02e908953d3a", namespace_name: "wocky-us1", pod_name: "wocky-79c6476f68-7k8mt", labels: { app: "wocky", pod-template-hash: "3572032924", env: "us1" } }, level: "error", log: "{"time":"2018-08-27 17:33:51.468","pid":"#PID<0.17890.9>","msg":"GenServer #PID<0.17890.9> terminating\n** (stop) exited in: GenServer.call(#PID<0.17891.9>, {:request, [%Kadabra.Request{body: \"{\\\"uri\\\":\\\"tinyrobot://bot/us1.prod.tinyrobot.com/08471fc0-7ec7-11e8-b7e0-0a580a0203ed/visitors\\\",\\\"aps\\\":{\\\"badge\\\":1,\\\"alert\\\":\\\"@sydnie is at Softball Field\\\"}}\", headers: [{\":method\", \"POST\"}, {\":path\", \"/3/device/d211275a85a8708c067d7eb106c5ac8910fdfa5d54c6d50c8162af5129c282f7\"}, {\"content-length\", \"151\"}, {\"apns-topic\", \"com.hippware.ios.Chat\"}], on_response: nil}]}, 5000)\n ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started\n (elixir) lib/gen_server.ex:924: GenServer.call/3\n (pigeon) lib/pigeon/connection.ex:145: Pigeon.Connection.send_push/3\n (elixir) lib/enum.ex:1925: Enum.\"-reduce/3-lists^foldl/2-0-\"/3\n (pigeon) lib/pigeon/connection.ex:97: Pigeon.Connection.handle_events/3\n (gen_stage) lib/gen_stage.ex:2315: GenStage.consumer_dispatch/6\n (stdlib) gen_server.erl:637: :gen_server.try_dispatch/4\n (stdlib) gen_server.erl:711: :gen_server.handle_msg/6\n (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3\nLast message: {:\"$gen_consumer\", {#PID<0.3748.0>, #Reference<0.2819615548.829685761.176736>}, [{:push, %Pigeon.APNS.Notification{collapse_id: nil, device_token: \"d211275a85a8708c067d7eb106c5ac8910fdfa5d54c6d50c8162af5129c282f7\", expiration: nil, id: nil, payload: %{\"aps\" => %{\"alert\" => \"@sydnie is at Softball Field\", \"badge\" => 1}, \"uri\" => \"tinyrobot://bot/us1.prod.tinyrobot.com/08471fc0-7ec7-11e8-b7e0-0a580a0203ed/visitors\"}, response: nil, topic: \"com.hippware.ios.Chat\"}, [on_response: #Function<1.110771641/1 in Wocky.Push.do_notify/4>]}]}","module":"gen_server","line":888,"level":"error","function":"error_info/7","file":"gen_server.erl"} ", line: 888, module: "gen_server", pid: "#PID<0.17890.9>", docker: { container_id: "e14cf7a2c8824a2ba22459d4c8f9504a215634de0b89947a895c73744320486f" }, file: "gen_server.erl", stream: "stdout", function: "error_info/7", time: "2018-08-27 17:33:51.468", timestamp: "2018-08-27T17:33:51+00:00" }, http: { clientHost: "52.33.167.239", contentType: "text" } }

Unfortunately there's no indication as to why the process died (presumably the remote end closed the connection). We're seeing about 2% of our push requests time out at this point which, while not catastrophic, is still not great. We could consider doing a retry ourselves as one option to reduce this - I'm not really sure what else we can do at this point - the kadabra HTTP client that pigeon uses is complex enough that I gave up trying to dig into it, and the author has acknowledged that it could stand some tweaking in its failure handling.

[toland]

I think this one might be worth fixing upstream. I may have a different opinion after looking at the Kadabra code, though.

[bernardd]

Yeah, I think the proper upstream fix is going to be "non-trivial" - like, a fairly significant rework of the whole approach. By all means take a look though.