search

FirebaseExtended / firebase-queue

MIT License
786 stars 108 forks source link

How do I monitor load and scale? #27

Closed morgs32 closed 4 years ago

morgs32 commented 9 years ago

Hi, I asked this question on one of the examples @drtriumph provided.

I think I'd use the firebase-queue to handle incoming messages (like the sanichat example). Except I want them to get handled after exactly 60 seconds from when they are "published" (from that example). So as long as a worker gets to it within the 60 seconds it can run a timeout for the time left, process it, and resolve it.

But, I'm wondering how to know when is too many workers for the cpu. Because that's the number of workers I'd use.

Then, I'd set the cpu threshold to something less than that - when it gets reached, Azure will add another vm with the same number of workers...

Any ideas? Go Firebase! (Label this a question of course)

cbraynor commented 9 years ago

This is a great question, and my initial answer is that there's not a whole lot of instrumentation for this use-case currently. I do, however see great value in this, so I'll have a think about how more can be built-in to the queue itself.

I think in your position I would start with monitoring the number of outstanding queue items, and the timeout lengths with some stats. As the timeout lengths creep down towards 0 you will be seeing the saturation of your pipeline, though that's not directly able to be automated. Timeouts in Node are not of course guaranteed to fire at the exact moment you schedule them for, so I'd also monitor the actual length of time between the item being added to the queue and it being processed.

Under load I'd then experiment with the number of workers. I usually suggest a single worker per CPU, but given the process is likely to just be waiting for a while, you probably want more. How many more will depend on how long your processing function usually takes - if it takes 5 seconds you might want to start with something like 12 workers per CPU (60 / 5).

Once you've settled on a number of processes per CPU, under load you will probably be able to come across some kind of heuristic for the number of outstanding requests per machine and then scale on that.

k1ng440 commented 7 years ago

hi @drtriumph, can you give us an example please?