Kafka_ex process exit without restart when request timeout

[yli-brex]

Hey,

I'm currently experiencing a problem that if Kafka_ex request timeout without broker response, the whole process crashed. Wonder is there a way / config to let kafka_ex application and worker restart automatically?

 {"crash_reason":"** (RuntimeError) Parse error during %Kayrock.Metadata.V5.Request{allow_auto_topic_creation: true, client_id: \"kafka_ex\", correlation_id: 2, topics: [\"topic_name\"]} response deserializer. Couldn't parse: [%KafkaEx.New.Broker{host: \"10.2.212.123\", node_id: 0, port: 9092, rack: nil, socket: nil}]\n    (kayrock) lib/generated/metadata.ex:1806: Kayrock.Metadata.V5.Response.deserialize/1\n    (kafka_ex) lib/kafka_ex/new/client.ex:668: KafkaEx.New.Client.deserialize/2\n    (kafka_ex) lib/kafka_ex/new/client.ex:566: KafkaEx.New.Client.run_client_request/3\n    (kafka_ex) lib/kafka_ex/new/client.ex:541: KafkaEx.New.Client.kayrock_network_request/4\n    (kafka_ex) lib/kafka_ex/new/client.ex:319: KafkaEx.New.Client.retrieve_metadata/5\n    (kafka_ex) lib/kafka_ex/new/client.ex:212: KafkaEx.New.Client.update_metadata/2\n    (kafka_ex) lib/kafka_ex/new/client.ex:176: KafkaEx.New.Client.handle_call/3\n    (stdlib) gen_server.erl:661: :gen_server.try_handle_call/4\n    (stdlib) gen_server.erl:690: :gen_server.handle_msg/6\n    (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3\n","erlang_context":{"pid":"#PID<0.1147.0>"},"file_info":{"file":"gen_server.erl","function":"error_info/7","line":889,"module":"gen_server"},"git_commit":"fb3b8ea691ff1c61d316bbadb06d81ff3ff9f407","identity":{},"level":"error","message":"GenServer :my_worker terminating\n** (RuntimeError) Parse error during %Kayrock.Metadata.V5.Request{allow_auto_topic_creation: true, client_id: \"kafka_ex\", correlation_id: 2, topics: [\"topic_name\"]} response deserializer. Couldn't parse: [%KafkaEx.New.Broker{host: \"10.2.212.123\", node_id: 0, port: 9092, rack: nil, socket: nil}]\n    (kayrock) lib/generated/metadata.ex:1806: Kayrock.Metadata.V5.Response.deserialize/1\n    (kafka_ex) lib/kafka_ex/new/client.ex:668: KafkaEx.New.Client.deserialize/2\n    (kafka_ex) lib/kafka_ex/new/client.ex:566: KafkaEx.New.Client.run_client_request/3\n    (kafka_ex) lib/kafka_ex/new/client.ex:541: KafkaEx.New.Client.kayrock_network_request/4\n    (kafka_ex) lib/kafka_ex/new/client.ex:319: KafkaEx.New.Client.retrieve_metadata/5\n    (kafka_ex) lib/kafka_ex/new/client.ex:212: KafkaEx.New.Client.update_metadata/2\n    (kafka_ex) lib/kafka_ex/new/client.ex:176: KafkaEx.New.Client.handle_call/3\n    (stdlib) gen_server.erl:661: :gen_server.try_handle_call/4\n    (stdlib) gen_server.erl:690: :gen_server.handle_msg/6\n    (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3\nLast message (from :topic_name): {:metadata, \"topic_name\"}","programming_language":"Elixir/1.8.1","registered_name":"my_worker","runtime":"Erlang/OTP/22","timestamp":"2022-03-21T22:22:27.330000Z","unsafe_labels":{"crash_reason":"** (RuntimeError) Parse error during %Kayrock.Metadata.V5.Request{allow_auto_topic_creation: true, client_id: \"kafka_ex\", correlation_id: 2, topics: [\"topic_name\"]} response deserializer. Couldn't parse: [%KafkaEx.New.Broker{host: \"10.2.212.123\", node_id: 0, port: 9092, rack: nil, socket: nil}]\n    (kayrock) lib/generated/metadata.ex:1806: Kayrock.Metadata.V5.Response.deserialize/1\n    (kafka_ex) lib/kafka_ex/new/client.ex:668: KafkaEx.New.Client.deserialize/2\n    (kafka_ex) lib/kafka_ex/new/client.ex:566: KafkaEx.New.Client.run_client_request/3\n    (kafka_ex) lib/kafka_ex/new/client.ex:541: KafkaEx.New.Client.kayrock_network_request/4\n    (kafka_ex) lib/kafka_ex/new/client.ex:319: KafkaEx.New.Client.retrieve_metadata/5\n    (kafka_ex) lib/kafka_ex/new/client.ex:212: KafkaEx.New.Client.update_metadata/2\n    (kafka_ex) lib/kafka_ex/new/client.ex:176: KafkaEx.New.Client.handle_call/3\n    (stdlib) gen_server.erl:661: :gen_server.try_handle_call/4\n    (stdlib) gen_server.erl:690: :gen_server.handle_msg/6\n    (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3\n","registered_name":"my_worker"}}

Thanks, Yujie

[dantswain]

Hi @yli-brex I'm a little confused. Having the process crash seems like the Erlang/Elixir way here, and it should restart automatically if it's under a supervision tree. Is that not what's happening for you? If so, can you give a little more detail?

[yli-brex]

@dantswain Thanks for the inputs, I'm also confused with why kafka_ex does not restart automatically.

This exit issue happen during sync produce call and will cascading crash the application without catch exit.

After notice this process exit issue, I was testing though elixir interactive terminal but not seen kafka_ex process restart automatically.