search

steelThread / redmon

A web interface for managing redis: cli, admin, and live monitoring
http://steelthread.github.com/redmon/
1.57k stars 127 forks source link

crash on one of our instances -- source sequence is illegal/malformed utf-8 #82

Open SteveDevOps opened 8 years ago

SteveDevOps commented 8 years ago

any ideas? It loops through rescue, never resolves, and I have to manually stop it.

[16-03-03 13:28:05] listening on http#0.0.0.0:4567
[16-03-03 13:28:15] !!! Redmon has shit the bed, restarting... source sequence is illegal/malformed utf-8
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon/worker.rb:27:in `to_json'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon/worker.rb:27:in `stats'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon/worker.rb:13:in `record_stats'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon/worker.rb:7:in `block in run!'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/eventmachine-1.0.9.1/lib/em/timers.rb:56:in `fire'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:193:in `call'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:193:in `run_machine'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:193:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:20:in `start_em'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:9:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:15:in `rescue in run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/lib/redmon.rb:8:in `run'
[16-03-03 13:28:15] /usr/local/rvm/gems/ruby-2.3.0/gems/redmon-0.0.10/bin/redmon:78:in `<top (required)>'^C
[root@evest-redis-1 redmon]# service redmon_4567 stop
SteveDevOps commented 8 years ago

we traced through your code and found where it's trying to load stats json from the associated redis key which we identified as:

redmon:redis:<ip address>:6379:stats

We then attempted to clear it out from the redis-cli

del redmon:redis:<ip address>:6379:stats

and restart redmon, but ui fails to load and we re-enter this crash cycle. We are running redmon standalone, no integrations. Other things to try?

SteveDevOps commented 8 years ago

I have been able to determine that the cause of the break in run is slowlog contents. There are some HMGETs included in the slowlog get and these end with (the following for example) : 

835510:516818
839761:520340
843986:524044
654885:20391
... (995 more arguments)

which I believe is what is breaking the json convert. For the moment we are running without slowlog data but would like to know a workaround?

steelThread commented 8 years ago

@SteveDevOps - I haven't really touched this code for a couple of years. I can take a look this weekend and get back with you. Thanks for reporting this.

SteveDevOps commented 8 years ago

Greatly appreciated.. let me know. Thx!