search

karafka / karafka-web

Web UI for monitoring and managing Karafka consumers
Other
55 stars 8 forks source link

installation fails for new project setting up karafka and web-ui #259

Closed juanpedrojose closed 8 months ago

juanpedrojose commented 8 months ago

Both lib/karafka/web/management/migrator.rb:55

Karafka::Web::Management::Migrator#consumers_state

and lib/karafka/web/management/migrator.rb:62

Karafka::Web::Management::Migrator#consumers_metrics

return a Hash with stringified keys while the ensure_migrable! expects symbolized keys for schema version comparison. This results in fails when running bundle exec karafka-web install.

Running data migrations...
[50, 59] in ~/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/karafka-web-0.8.0/lib/karafka/web/management/migrator.rb
    50|
    51|         # Raise an exception if there would be an attempt to run migrations on a newer schema for
    52|         # any states we manage. We can only move forward, so attempt to migrate for example from
    53|         # 1.0.0 to 0.9.0 should be considered and error.
    54|         def ensure_migrable!
=>  55|           debugger
    56|           if consumers_state[:schema_version] > State::SCHEMA_VERSION
    57|             raise(
    58|               Errors::Management::IncompatibleSchemaError,
    59|               'consumers state newer than supported'
=>#0    Karafka::Web::Management::Migrator#ensure_migrable! at ~/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/karafka-web-0.8.0/lib/karafka/web/management/migrator.rb:55
  #1    Karafka::Web::Management::Migrator#call at ~/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/karafka-web-0.8.0/lib/karafka/web/management/migrator.rb:44
  # and 22 frames (use `bt' command for all frames)
(ruby) consumers_state
{"schema_version"=>"0.0.0"}
(ruby) consumers_metrics
{"schema_version"=>"0.0.0"}

A possible solution is to change lib/karafka/web/processing/consumers/state.rb:23 and lib/karafka/web/processing/consumers/metrics.rb:23 from

return state_message.payload if state_message
return metrics_message.payload if metrics_message

to

return state_message.payload.stringify_keys if state_message
return metrics_message.payload.symbolize_keys if metrics_message

But it's a "dirt" hack rather than a mature solution.

Expected behavior

$ bundle exec karafka-web install

Installing Karafka Web UI...

Creating necessary topics and populating state data...

Creating topic karafka_errors...
Topic karafka_errors successfully created.
Creating topic karafka_consumers_reports...
Topic karafka_consumers_reports successfully created.
Creating topic karafka_consumers_metrics...
Topic karafka_consumers_metrics successfully created.
Creating topic karafka_consumers_states...
Topic karafka_consumers_states successfully created.

Waiting for the topics to synchronize in the cluster.....

Creating consumers state initial record...
[3f3a932b547a] Sync producing of a message to 'karafka_consumers_states' topic took 23.16200017929077 ms
Initial consumers state record successfully created.
Creating consumers metrics initial record...
[3f3a932b547a] Sync producing of a message to 'karafka_consumers_metrics' topic took 966.6679999828339 ms
Initial consumers metrics record successfully created.

Running data migrations...
[3f3a932b547a] Sync producing of 2 messages to 2 topics took 32.43499994277954 ms

Updating the Karafka boot file...
Karafka boot file successfully updated.

Installation completed. Have fun!

[3f3a932b547a] Closing producer took 0.023000001907348633 ms

Actual behavior

$ bundle exec karafka-web install

Installing Karafka Web UI...

Creating necessary topics and populating state data...

Creating topic karafka_errors...
Topic karafka_errors successfully created.
Creating topic karafka_consumers_reports...
Topic karafka_consumers_reports successfully created.
Creating topic karafka_consumers_metrics...
Topic karafka_consumers_metrics successfully created.
Creating topic karafka_consumers_states...
Topic karafka_consumers_states successfully created.

Waiting for the topics to synchronize in the cluster.....

Creating consumers state initial record...
[1ab7a0e684ef] Sync producing of a message to 'karafka_consumers_states' topic took 83.43699979782104 ms
Initial consumers state record successfully created.
Creating consumers metrics initial record...
[1ab7a0e684ef] Sync producing of a message to 'karafka_consumers_metrics' topic took 1057.284999847412 ms
Initial consumers metrics record successfully created.

Running data migrations...
bundler: failed to load command: karafka-web (/Users/alex/.rbenv/versions/3.2.3/bin/karafka-web)
/Users/alex/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/karafka-web-0.8.0/lib/karafka/web/management/migrator.rb:55:in `ensure_migrable!': undefined method `>' for nil:NilClass (NoMethodError)

          if consumers_state[:schema_version] > State::SCHEMA_VERSION
                                              ^
    from /Users/alex/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/karafka-web-0.8.0/lib/karafka/web/management/migrator.rb:44:in `call'
    from /Users/alex/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/karafka-web-0.8.0/lib/karafka/web/management/actions/migrate_states_data.rb:12:in `call'
    from /Users/alex/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/karafka-web-0.8.0/lib/karafka/web/installer.rb:25:in `install'
    from /Users/alex/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/karafka-web-0.8.0/lib/karafka/web/cli/install.rb:20:in `call'
    from /Users/alex/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/karafka-2.3.0/lib/karafka/cli.rb:23:in `start'
    from /Users/alex/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/karafka-web-0.8.0/bin/karafka-web:9:in `<top (required)>'
    from /Users/alex/.rbenv/versions/3.2.3/bin/karafka-web:25:in `load'
    from /Users/alex/.rbenv/versions/3.2.3/bin/karafka-web:25:in `<top (required)>'
    from /Users/alex/.rbenv/versions/3.2.3/lib/ruby/site_ruby/3.2.0/bundler/cli/exec.rb:58:in `load'
    from /Users/alex/.rbenv/versions/3.2.3/lib/ruby/site_ruby/3.2.0/bundler/cli/exec.rb:58:in `kernel_load'
    from /Users/alex/.rbenv/versions/3.2.3/lib/ruby/site_ruby/3.2.0/bundler/cli/exec.rb:23:in `run'
    from /Users/alex/.rbenv/versions/3.2.3/lib/ruby/site_ruby/3.2.0/bundler/cli.rb:451:in `exec'
    from /Users/alex/.rbenv/versions/3.2.3/lib/ruby/site_ruby/3.2.0/bundler/vendor/thor/lib/thor/command.rb:28:in `run'
    from /Users/alex/.rbenv/versions/3.2.3/lib/ruby/site_ruby/3.2.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
    from /Users/alex/.rbenv/versions/3.2.3/lib/ruby/site_ruby/3.2.0/bundler/vendor/thor/lib/thor.rb:527:in `dispatch'
    from /Users/alex/.rbenv/versions/3.2.3/lib/ruby/site_ruby/3.2.0/bundler/cli.rb:34:in `dispatch'
    from /Users/alex/.rbenv/versions/3.2.3/lib/ruby/site_ruby/3.2.0/bundler/vendor/thor/lib/thor/base.rb:584:in `start'
    from /Users/alex/.rbenv/versions/3.2.3/lib/ruby/site_ruby/3.2.0/bundler/cli.rb:28:in `start'
    from /Users/alex/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/bundler-2.5.5/exe/bundle:28:in `block in <top (required)>'
    from /Users/alex/.rbenv/versions/3.2.3/lib/ruby/site_ruby/3.2.0/bundler/friendly_errors.rb:117:in `with_friendly_errors'
    from /Users/alex/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/bundler-2.5.5/exe/bundle:20:in `<top (required)>'
    from /Users/alex/.rbenv/versions/3.2.3/bin/bundle:25:in `load'
    from /Users/alex/.rbenv/versions/3.2.3/bin/bundle:25:in `<main>'
[1ab7a0e684ef] Closing producer took 0.41499972343444824 ms

Steps to reproduce the problem

$ rails new testapp
$ cd testapp
$ bundle add karafka --version ">= 2.3.0"
$ bundle exec karafka install
$ bundle add karafka-web --version ">= 0.8.0"
$ bundle exec karafka-web install

Setup details

$ bundle exec karafka info
Karafka version: 2.3.0
Ruby version: ruby 3.2.3 (2024-01-18 revision 52bb2ac0a6) [arm64-darwin23]
Rdkafka version: 0.14.9
Consumer groups count: 2
Subscription groups count: 2
Workers count: 5
Application client id: example_app
Boot file: /Users/alex/projects/testapp/karafka.rb
Environment: development
License: LGPL-3.0
mensfeld commented 8 months ago

hmm how did my integrations not catch it :thinking:

mensfeld commented 8 months ago

ah I know! The template is not enabling the web UI. I should alter the setup flow. There is no need to change anything in the framework. Look at the end of karafka.rb:

# Karafka now features a Web UI!
# Visit the setup documentation to get started and enhance your experience.
#
# https://karafka.io/docs/Web-UI-Getting-Started
#
# Karafka::Web.enable!

if you uncomment the last line things work as expected. So the expected flow here is:

$ rails new testapp
$ cd testapp
$ bundle add karafka --version ">= 2.3.0"
$ bundle exec karafka install
$ bundle add karafka-web --version ">= 0.8.0"

# uncomment the setup

$ bundle exec karafka-web install

everything works then.

mensfeld commented 8 months ago

this is why my tests didn't detect it. I enable the web UI.

Web UI should add the web enabled at the end and indeed the current flow is suboptimal. I will today enhance docs to make sure this is documented (to make sure enable is not commented out)

mensfeld commented 8 months ago

It actually shows a different issue, wrong template is being selected. Fixing

mensfeld commented 8 months ago

Funny, this will require update to both karafka and karafka-web to fully fix it because:

nonetheless I will make sure that it is handled in web without a karafka release for now

great report!

mensfeld commented 8 months ago

@juanpedrojose I was able to write integration specs for it. Sorry for the issue :pray:

It is already fixed in master and will be released tomorrow.

mensfeld commented 8 months ago

I will keep it open until the release just in case someone looks for it

mensfeld commented 8 months ago

0.8.1 has been released. Specs in karafka were added and pass. Will not happen again. Thank you and sorry :pray: