search

gemhome / fnordmetric

(possible new home for) FnordMetric is a redis/ruby-based realtime Event-Tracking app
0 stars 1 forks source link

Queue Length #9

Open bf4 opened 10 years ago

bf4 commented 10 years ago

Issue by chrismoos Wednesday Jan 18, 2012 at 16:12 GMT Originally opened as https://github.com/paulasmuth/fnordmetric/issues/45

Hi,

If I have a lot of events fnordmetric can't keep up and the queue length keeps growing until eventually Redis is claiming all memory on the box. (queue length of 8 million). I'm not sure if the queue is the only reason to blame for Redis taking up so much memory, but maybe also data stored on the various metrics? (I'm not too sure of how everything is stored).

5.4G Redis, here is last log from fnordmetric:

events_received: 25300317, events_processed: 17397513, queue_length: 7901866

bf4 commented 10 years ago

Comment by kazjote Thursday Jan 19, 2012 at 14:23 GMT

All events are stored as separate values in Redis (am I right?). It is quite possible that you are just out of space.

You could try decreasing :event_data_ttl parameter as shown here: https://github.com/paulasmuth/fnordmetric/blob/master/doc/full_example.rb#L635 . By default it is set to 1 month. But it will only take effect for newly added events.

bf4 commented 10 years ago

Comment by chrismoos Thursday Jan 19, 2012 at 14:53 GMT

The TTL isn't a bad idea but I run out of memory within hours, so I don't think its going to help.

bf4 commented 10 years ago

Comment by kazjote Thursday Jan 19, 2012 at 15:01 GMT

You can experiment and set it to 5 minutes :) Do you use uniq gauges and sessions?

bf4 commented 10 years ago

Comment by JohnMurray Wednesday Feb 08, 2012 at 03:48 GMT

Have you tried running multiple instances? If you run multiple instances on the same machine (even with the same configuration), it will notice a web-server interface running and just start up another worker process.

I'm working on a commit now that will allow Fnordmetric to spawn off multiple workers via multiple processes.

bf4 commented 10 years ago

Comment by catz Sunday Mar 04, 2012 at 17:43 GMT

you should definitely run more workers and decrease event_data_ttl parameter. I run several workers via self.embedded (thin) monitored by god. My queue is always 0 now. It's about 2000 events for 3 seconds flow now.

imo 8 millions unprocessed events - totally unreal case

bf4 commented 10 years ago

Comment by chrismoos Monday Mar 05, 2012 at 18:39 GMT

Yeah seems like more workers and TTL is the way to go, going to try this out. Thanks everyone.