Consumers do not create topics automatically

skateinmars commented 5 years ago

On the server side, kafka has a auto.create.topics.enable setting that determines whether a topic can be used "on the fly" without having to create it manually beforehand.

In other languages/libraries, such as the kafka-console-consumer "official" CLI that uses the java library, or https://github.com/Shopify/sarama (a pure go implementation), starting a consumer or a consumer group for a non-existing topic will trigger the auto create and start consuming from the newly created topic.

With kafka_ex, topics do not get created automatically.

KafkaEx.fetch will return :topic_not_found, which is probably the desired behaviour here (regardless of the auto.create.topics.enable setting)
KafkaEx.stream will also return :topic_not_found, but with the standard use case of a pipe, you risk running into issues

iex(1)> KafkaEx.stream("willfail", 0, offset: 0) |> Enum.take(1)
** (Protocol.UndefinedError) protocol Enumerable not implemented for :topic_not_found. This protocol is implemented for: [...]

With consumer groups, a single warning will be displayed

Consumer group groupname encountered nonexistent topic topicname

Afterwards the consumer will get "stuck", even if the topics gets created manually later.

I believe starting a consumer group should follow the expected behaviour and trigger a topic creation.

If that helps, I noticed that in lib/kafka_ex/consumer_group/manager.ex, KafkaEx.metadata is called to fetch partition info for every topic. On the other hand, calling KafkaEx.metadata(topic: "topicname") will trigger the automatic topic creation, and return with metadata for the created topic. I'm not sure though which implications replacing the global metadata call with a series of per-topic metadata calls may have?

bjhaid commented 5 years ago

I'll rather see this implemented explicitly via CreateTopics protocol:

https://kafka.apache.org/protocol.html#The_Messages_CreateTopics

We'll accept PRs for the explicit CreateTopics, see a similar PR for DeleteTopics:

https://github.com/kafkaex/kafka_ex/pull/338/files

bjhaid commented 5 years ago

I think this:

Afterwards the consumer will get "stuck", even if the topics gets created manually later.

is a bug that probably needs a separate issue/PR, I expect the consumer group creation to halt if the topic does not exist. Also we welcome PRs to address it :)

skateinmars commented 5 years ago

I'll rather see this implemented explicitly via CreateTopics protocol:

I'll admit auto creation is usually considered an anti-pattern!

I expect the consumer group creation to halt if the topic does not exist. Also we welcome PRs to address it :)

What do you think the behaviour should be? Should the KafkaEx.ConsumerGroup crash?

bjhaid commented 5 years ago

Should the KafkaEx.ConsumerGroup crash?

Maybe or not return with an error at creation, I don't have strong opinions...

joshuawscott commented 5 years ago

I think autocreation could be an option; I'd be concerned about doing it by default.

I think returning an error at creation vs crashing is basically the same thing, since returning anything besides an :ok tuple during start_link is going to force the supervisor to crash. I'm fine with it crashing if it cannot find the topic (I would have expected that this would happen already)

kafkaex / kafka_ex

Consumers do not create topics automatically #340