missing `waiting` attributes in underlying hash in Karafka::Web::Ui::Lib::HashProxy

[nonnenmacher]

When upgrading from Karafka-web 0.7.10 to 0.8.0

Launching the web-ui fails with a 'missing method' in the waiting counter

NameError: undefined local variable or method `waiting' for #<Karafka::Web::Ui::Models::Counters:0x00000001046cdb58 @hash={:batches=>98, :messages=>114, :retries=>17, :dead=>0, :busy=>0, :enqueued=>0, :processing=>0, :workers=>5, :processes=>1, :rss=>58036, :listeners=>2, :utilization=>0.009067093286916215, :lag_stored=>0, :errors=>17, :lag=>3}, @visited={:messages=>{{:batches=>98, :messages=>114, :retries=>17, :dead=>0, :busy=>0, :enqueued=>0, :processing=>0, :workers=>5, :processes=>1, :rss=>58036, :listeners=>2, :utilization=>0.009067093286916215, :lag_stored=>0, :errors=>17, :lag=>3}=>nil}, :batches=>{{:batches=>98, :messages=>114, :retries=>17, :dead=>0, :busy=>0, :enqueued=>0, :processing=>0, :workers=>5, :processes=>1, :rss=>58036, :listeners=>2, :utilization=>0.009067093286916215, :lag_stored=>0, :errors=>17, :lag=>3}=>nil}, :lag_stored=>{{:batches=>98, :messages=>114, :retries=>17, :dead=>0, :busy=>0, :enqueued=>0, :processing=>0, :workers=>5, :processes=>1, :rss=>58036, :listeners=>2, :utilization=>0.009067093286916215, :lag_stored=>0, :errors=>17, :lag=>3}=>nil}, :busy=>{{:batches=>98, :messages=>114, :retries=>17, :dead=>0, :busy=>0, :enqueued=>0, :processing=>0, :workers=>5, :processes=>1, :rss=>58036, :listeners=>2, :utilization=>0.009067093286916215, :lag_stored=>0, :errors=>17, :lag=>3}=>nil}, :enqueued=>{{:batches=>98, :messages=>114, :retries=>17, :dead=>0, :busy=>0, :enqueued=>0, :processing=>0, :workers=>5, :processes=>1, :rss=>58036, :listeners=>2, :utilization=>0.009067093286916215, :lag_stored=>0, :errors=>17, :lag=>3}=>nil}, :waiting=>{{:batches=>98, :messages=>114, :retries=>17, :dead=>0, :busy=>0, :enqueued=>0, :processing=>0, :workers=>5, :processes=>1, :rss=>58036, :listeners=>2, :utilization=>0.009067093286916215, :lag_stored=>0, :errors=>17, :lag=>3}=>nil, [:batches, 98]=>nil, :batches=>nil, 98=>nil, [:messages, 114]=>nil, :messages=>nil, 114=>nil, [:retries, 17]=>nil, :retries=>nil, 17=>nil, [:dead, 0]=>nil, :dead=>nil, 0=>nil, [:busy, 0]=>nil, :busy=>nil, [:enqueued, 0]=>nil, :enqueued=>nil, [:processing, 0]=>nil, :processing=>nil, [:workers, 5]=>nil, :workers=>nil, 5=>nil, [:processes, 1]=>nil, :processes=>nil, 1=>nil, [:rss, 58036]=>nil, :rss=>nil, 58036=>nil, [:listeners, 2]=>nil, :listeners=>nil, 2=>nil, [:utilization, 0.009067093286916215]=>nil, :utilization=>nil, 0.009067093286916215=>nil, [:lag_stored, 0]=>nil, :lag_stored=>nil, [:errors, 17]=>nil, :errors=>nil, [:lag, 3]=>nil, :lag=>nil, 3=>nil}}, @results={:messages=>114, :batches=>98, :lag_stored=>0, :busy=>0, :enqueued=>0}> (NameError)
    /Users/vincentnonnenmacher/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/karafka-web-0.8.0/lib/karafka/web/ui/lib/hash_proxy.rb:49:in `method_missing'
    /Users/vincentnonnenmacher/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/karafka-web-0.8.0/lib/karafka/web/ui/models/counters.rb:23:in `pending'

perhaps a missing step on the gem upgrade karafka from 2.2.14 to 2.3.0 ?

[mensfeld]

Did you upgrade karafka to 2.3.0 and redeployed? only when puma and karafka servers are deployed and in sync this will dissapear

[mensfeld]

quote from web ui upgrade docs:

The Web UI interface may throw 500 errors during the upgrade because of schema incompatibility (until Puma is deployed). This will have no long-term effects and can be ignored.

https://karafka.io/docs/Changelog-Karafka-Web-UI/#upgrade-notes

[mensfeld]

Closing as I see no reply and I was able to reproduce and handle in the described way.

[nonnenmacher]

din't see any notion of Puma here and missed certainly the 'redeployed' thing in the doc.

Updated all gems according to changeling, still need to retry and look at this counter from the source.

[mensfeld]

Can you give me more stacktrace?

[mensfeld]

Ok I think I found it. Now let me try to reproduce

[mensfeld]

Did you roll all of the karafka server including the one that deals with web UI materialization? Because as far as I can see here a migration didn't kick in. This migration should run automatically in 0.8.0 karafka consumer. Can you show me the last payload from the karafka_consumers_states topic?

[mensfeld]

Yeah migrations didn't kick in (yet):

undefined local variable or method `waiting' for an instance of Karafka::Web::Ui::Models::Counters

[Application Trace](http://localhost:3000/karafka/dashboard#) | [Framework Trace](http://localhost:3000/karafka/dashboard#) | [Full Trace](http://localhost:3000/karafka/dashboard#)
karafka-web (0.8.0) lib/karafka/web/ui/lib/hash_proxy.rb:49:in `method_missing'
karafka-web (0.8.0) lib/karafka/web/ui/models/counters.rb:23:in `pending'
karafka-web (0.8.0) lib/karafka/web/ui/views/consumers/_counters.erb:37:in `__tilt_11220'
tilt (2.3.0) lib/tilt/template.rb:207:in `bind_call'
tilt (2.3.0) lib/tilt/template.rb:207:in `evaluate'
tilt (2.3.0) lib/tilt/template.rb:102:in `render'
roda (3.76.0) lib/roda/plugins/render.rb:500:in `render'
roda (3.76.0) lib/roda/plugins/partials.rb:58:in `partial'
karafka-web (0.8.0) lib/karafka/web/ui/views/dashboard/index.erb:11:in `__tilt_11220'
roda (3.76.0) lib/roda/plugins/render.rb:620:in `_optimized_view_content'
roda (3.76.0) lib/roda/plugins/render.rb:514:in `view'
karafka-web (0.8.0) lib/karafka/web/ui/base.rb:151:in `render_response'
karafka-web (0.8.0) lib/karafka/web/ui/base.rb:59:in `block in <class:Base>'

looks exactly like your issue. This issue arises from old 0.7.10 materialization consumer running with 0.8.0 web ui

[mensfeld]

If you need any extra help understanding or fixing this feel free to reach out to me via https://slack.karafka.io/ - as of now as said, I was able to reproduce and get beyond that by updating consumers as well and deploying.

[nonnenmacher]

I'll RTFM (but as the guy who follow all the small print in the readme/changelog, I was thinking something is missing but perhaps it is because I rewrapped the whole 'Application' using dry-system and I perhaps missed a point here.

anyway we'll buy a PRO licence anyway, but I need to create perceptual value as usual ;-)

[mensfeld]

No worries. While I was able to reproduce it I do accept my shortcomings and if there indeed is a bug, I assure you it will be fixed. On top of that, if you ended up with a "weird" intermediate UI state I am also willing to help you to fix it :)

[nonnenmacher]

Great help from @mensfeld !!!

When starting an 'app' construct not from the template, don't forget to call Karafka::Web.enable! as this take care of 'upgrading' internal Karafka topics to new schema (as expected by the web-ui gem).