Key not found :jobs - Githubissues

BrandonHicks-msr commented 3 weeks ago

Expected behavior

While using v0.9.1 it works as expected, karafka is able to consume and produce topic messages as expected. It is not used as an ActiveJob replacement (we have sidekiq for this). We are using karafka as a message bus to pass topic messages between applications

Actual behavior

When upgrading to v0.10 this seems to no longer be the case, with 0.10 when accessing the karafka-web ui key not found :jobs is immediately thrown.

Your setup details

$ [bundle exec] karafka info
Karafka version: 2.4.8
Ruby version: ruby 3.3.4 (2024-07-09 revision be1089c8ec) [arm64-darwin23]
Rdkafka version: 0.17.3
Consumer groups count: 2
Subscription groups count: 2
Workers count: 5
Application client id: measurabl_api
Boot file: /Users/******/karafka.rb
Environment: development
License: LGPL-3.0
Shutting down background worker

The karafka config is setup and very basic:

# frozen_string_literal: true

# Logs
require 'datadog'
require 'karafka/instrumentation/vendors/datadog/logger_listener'

class KarafkaApp < Karafka::App
  setup do |config|
    config.kafka = {
      'bootstrap.servers': ENV.fetch('KAFKA_URL', ''),
      'security.protocol': ENV.fetch('KAFKA_SECURITY_PROTOCOL', 'ssl'),
      'api.version.request': true,
      'ssl.key.pem': ENV.fetch('KAFKA_SERVICE_KEY', ''),
      'ssl.certificate.pem': ENV.fetch('KAFKA_SERVICE_CERT', ''),
      'ssl.ca.pem': ENV.fetch('KAFKA_CA_CERT', ''),
      partitioner: 'murmur2_random',
    }
    config.client_id = 'app_client'
    config.consumer_persistence = true
  end

  Karafka.monitor.subscribe(Karafka::Instrumentation::LoggerListener.new)
  # Log Karafka exceptions to sentry
  Karafka.monitor.subscribe 'error.occurred' do |event|
    Sentry.capture_exception(event[:error])
  end

  # This logger prints the producer development info using the Karafka logger.
  # It is similar to the consumer logger listener but producer oriented.
  Karafka.producer.monitor.subscribe(
    WaterDrop::Instrumentation::LoggerListener.new(
      Karafka.logger,
      log_messages: false
    )
  )

  routes.draw do
    topic :sample_topic do
      consumer Sample::ResultsConsumer
    end
  end
end

##############################
# Karafka instrumentation and logging
##############################
# Initialize the listener for logging with Datadog
dd_logger_listener = Karafka::Instrumentation::Vendors::Datadog::LoggerListener.new do |config|
  config.client = Datadog::Tracing
end
# Use the DD tracing only for staging and production
Karafka.monitor.subscribe(dd_logger_listener) if %w[staging production].include?(Rails.env)

Karafka::Web.setup do |config|
  config.ui.sessions.secret = ENV.fetch('KARAFKA_WEB_SECRET', SecureRandom.hex(64))
end

Karafka::Web.enable!