sender-cli — parallelism is a lie

[elmigranto]

So I just realized node streams are processing stuff in order — before starting to work on item#2, item#1 needs to be done with. When I wrote sender-cli, I was a bit confused on all the .pipe() interactions, highWaterMark option and backpressure stuff. So… pretty sure we pull X things from redis, and send them one by one instead of in parallel.

Well, one learns new things every day!

[j3k0]

Side note. The push worker has been doing its job without issues for months, so it might not be a problem, until we reach the point were inputs get faster that what worker can output, but I suspect we're far from that.

Checking out right now, output rate is approx 12 notifications / second

Input rate around 5 notifications / second at peak hour. If number of users doubles, we'll have to watch this out... If I'm right we can just simply start a second worker.

"name":"notifications","hostname":"xxx","pid":30264,"sender":true,"gcm":true,"level":30,"msg":"sending GCM 249773412","time":"2017-03-08T19:37:34.009Z","v":0}
{"name":"notifications","hostname":"xxx","pid":30264,"sender":true,"gcm":true,"level":30,"msg":"sending GCM 249773415","time":"2017-03-08T19:37:34.089Z","v":0}
{"name":"notifications","hostname":"xxx","pid":30264,"sender":true,"gcm":true,"level":30,"msg":"sending GCM 249773418","time":"2017-03-08T19:37:34.148Z","v":0}
{"name":"notifications","hostname":"xxx","pid":30264,"sender":true,"gcm":true,"level":30,"msg":"sending GCM 249773421","time":"2017-03-08T19:37:34.240Z","v":0}
{"name":"notifications","hostname":"xxx","pid":30264,"sender":true,"apn":true,"level":30,"msg":"sending APN 249773424","time":"2017-03-08T19:37:34.319Z","v":0}
{"name":"notifications","hostname":"xx","pid":30264,"sender":true,"gcm":true,"level":30,"msg":"sending GCM 249773427","time":"2017-03-08T19:37:34.958Z","v":0}
{"name":"notifications","hostname":"xxx","pid":30264,"sender":true,"apn":true,"level":30,"msg":"sending APN 249773430","time":"2017-03-08T19:37:35.044Z","v":0}
{"name":"notifications","hostname":"xxx","pid":30264,"sender":true,"gcm":true,"level":30,"msg":"sending GCM 249773433","time":"2017-03-08T19:37:35.045Z","v":0}
{"name":"notifications","hostname":"xxx","pid":30264,"sender":true,"apn":true,"level":30,"msg":"sending APN 249773442","time":"2017-03-08T19:37:35.126Z","v":0}
{"name":"notifications","hostname":"xxx","pid":30264,"sender":true,"gcm":true,"level":30,"msg":"sending GCM 249773445","time":"2017-03-08T19:37:35.126Z","v":0}
{"name":"notifications","hostname":"xxx","pid":30264,"sender":true,"apn":true,"level":30,"msg":"sending APN 249773449","time":"2017-03-08T19:37:35.215Z","v":0}
{"name":"notifications","hostname":"xxx","pid":30264,"sender":true,"gcm":true,"level":30,"msg":"sending GCM 249773463","time":"2017-03-08T19:37:35.216Z","v":0}
{"name":"notifications","hostname":"xxx","pid":30264,"sender":true,"gcm":true,"level":30,"msg":"sending GCM 249773470","time":"2017-03-08T19:37:35.307Z","v":0}
{"name":"notifications","hostname":"xxx","pid":30264,"sender":true,"apn":true,"level":30,"msg":"sending APN 249773473","time":"2017-03-08T19:37:35.563Z","v":0}
{"name":"notifications","hostname":"xxx","pid":30264,"sender":true,"gcm":true,"level":30,"msg":"sending GCM 249773482","time":"2017-03-08T19:37:35.563Z","v":0}
{"name":"notifications","hostname":"xxx","pid":30264,"sender":true,"gcm":true,"level":30,"msg":"sending GCM 249773486","time":"2017-03-08T19:37:35.625Z","v":0}
{"name":"notifications","hostname":"xxx","pid":30264,"sender":true,"gcm":true,"level":30,"msg":"sending GCM 249773489","time":"2017-03-08T19:37:35.701Z","v":0}
{"name":"notifications","hostname":"xxx","pid":30264,"sender":true,"gcm":true,"level":30,"msg":"sending GCM 249773492","time":"2017-03-08T19:37:35.813Z","v":0}
{"name":"notifications","hostname":"xxx","pid":30264,"sender":true,"gcm":true,"level":30,"msg":"sending GCM 249773498","time":"2017-03-08T19:37:35.918Z","v":0}
{"name":"notifications","hostname":"xxx","pid":30264,"sender":true,"gcm":true,"level":30,"msg":"sending GCM 249773501","time":"2017-03-08T19:37:36.002Z","v":0}
{"name":"notifications","hostname":"xxx","pid":30264,"sender":true,"gcm":true,"level":30,"msg":"sending GCM 249773504","time":"2017-03-08T19:37:36.073Z","v":0}
{"name":"notifications","hostname":"xxx","pid":30264,"sender":true,"gcm":true,"level":30,"msg":"sending GCM 249773507","time":"2017-03-08T19:37:36.154Z","v":0}
{"name":"notifications","hostname":"xxx","pid":30264,"sender":true,"gcm":true,"level":30,"msg":"sending GCM 249773512","time":"2017-03-08T19:37:36.239Z","v":0}
{"name":"notifications","hostname":"xxx","pid":30264,"sender":true,"gcm":true,"level":30,"msg":"sending GCM 249773518","time":"2017-03-08T19:37:36.329Z","v":0}
{"name":"notifications","hostname":"xxx","pid":30264,"sender":true,"gcm":true,"level":30,"msg":"sending GCM 249773521","time":"2017-03-08T19:37:36.403Z","v":0}
{"name":"notifications","hostname":"xxx","pid":30264,"sender":true,"gcm":true,"level":30,"msg":"sending GCM 249773530","time":"2017-03-08T19:37:36.504Z","v":0}
{"name":"notifications","hostname":"xxx","pid":30271,"sender":true,"gcm":true,"level":30,"msg":"sending GCM 249773536","time":"2017-03-08T19:37:40.618Z","v":0}

[elmigranto]

iOS one is "async" and just dumps data to socket with feedback on separate connection, but android one might be significantly slower b/c of this IIRC.