what kind of bandwidth does worker queue use?

[brandonmp]

I've been working through a firebase queue of ~300k tasks, each of which is a tuple of ~20 chars. I had anywhere from 13-25 workers on it at a time.

Was each one of those workers downloading the entire queue each time? Once they'd done that, would they have re-downloading the entire thing at some point?

Trying to understand (a) how better to optimize the flow and (b) how to avoid in the future the 70 GB of download I've incurred on this project ;)

[cbraynor]

The queue workers do not download the entire queue - they only download the latest task. However, if there's a lot of contention on the queue, the bandwidth spikes might be caused by the optimistic concurrency. Take this example:

1. A new task is added to the queue
2. All n workers download the latest task
3. All available workers (m) start a transaction to claim it
   • They perform the transformation function on the task and attempt to write back to the server
   • One succeeds, and starts processing the task
   • m-1 workers fail because they have an old version of the data, the transaction automatically fetches the latest version of that task (now claimed), runs the transformation function again, which cancels early
   • Meanwhile, the original task has been modified in a such a way that it's removed from the worker's listeners and the next task is sent to all n workers

Contention can be caused by a combination of short processing functions, large numbers of workers, or large numbers of tasks, and reducing it will probably be your best bet for reducing your bill.

If your processing function can't change, and the number of tasks / workers can't really be reduced in your case, then you could try sharding your queue - have the clients push onto one of many queues (either a random choice based on the number of queues, or through some application-specific hashing algorithm that suits your needs), then smaller pools of workers process a smaller pool of incoming tasks and there's less contention

[brandonmp]

thanks @drtriumph , i think i follow the example.

I'm still left wondering, though, how my bandwidth went nuts:

The only process happening during that spike (which hit ~2-3GB/hr for the better part of a day) was a worker queue w/ ~12-25 workers (each in a docker, w/ numWorkers === 1), and their only interaction with firebase was the queue and an update() call.

Even if every task was in contention among all 25 workers, that's still a fraction of that kind of bandwidth (each task was {"someKeyName":"some 18 char value"} -- the size the queue w/ ~250k tasks was <5MB).

Is there any feasible way this could be some kind of queue-related thing, either user error or otherwise?

I'm working w/ firebase support to get a more granular look at the traffic, but until then, just trying to rule-out the worker queue so I can continue using it on other projects.

[cbraynor]

I'll let you work with support on the specifics. The queue does add some fields that are non-negligible in relation to your payload (maybe multiple of a factor of 3 or 4), but that doesn't explain the huge increase

[katowulf]

Each of the workers would download the entire queue each time the service is started. After this it will receive deltas of adds/updates/deletes. Ideally, if you need to optimize, you shouldn't store more data in the queue than needed. Instead, try passing the queue workers some ids and letting them fetch individual records as needed.

However, a random guess is that your queue is probably not consuming the majority of bandwidth. I'd start by making sure your indices are set up correctly and that queries are working as expected. Also, if you have clients doing once() or on('value') style ops, that's a much more likely culprit.

Note that this sounds like something you can guesstimate fairly quickly. Try turning on debug logging, using firebase.database().enableLogging(true);. You'll be able to see all socket traffic, which looks similar to the following:

p:0: from server: {"r":3,"b":{"s":"ok","d":{}}}  
p:0: from server: {"r":4,"b":{"s":"ok","d":""}}  
p:0: from server: {"r":5,"b":{"s":"ok","d":""}}