karafka / karafka

Ruby and Rails efficient Kafka processing framework
https://karafka.io
Other
2.09k stars 178 forks source link

Introduce active flag for topics to enable/disable them #1224

Closed mensfeld closed 1 year ago

mensfeld commented 1 year ago

Routing style like this:

consumer_groups.draw do
    unless FlipperHelper.enabled? :shutdown_consumer_a
      topic "topic_a" do
        consumer ConsumerA
      end
    end

    unless FlipperHelper.enabled? :shutdown_consumer_b
      topic "topic_b" do
        consumer ConsumerB
      end
    end

    unless FlipperHelper.enabled? :shutdown_consumer_c
      topic "topic_c" do
        consumer ConsumerC
      end
    end
  end
end

messes up with subscription groups count and makes it hard to build proper client ids for static group membership.

We need to be able to deactivate given topics (hence sub groups in some cases) but keep them in routes, not to impact indexing

mensfeld commented 1 year ago

Done and described: https://karafka.io/docs/Routing/#topic-level-options