Performance improvement: Pub/sub should pass messages by reference

[emadum]

Messaging systems work best with small message payloads. Redis in particular suffers huge performance losses when pub/sub message payloads get large.

Since UMF messages already have a unique message ID, we can cache the message payload in redis when a message is sent and only send the message ID over pub/sub, minimizing the message payload.

[sjmcdowall]

My gut tells me there is a "cutoff" in payload size where doing this makes sense but just keeping the message in the payload makes sense to avoid another trip to Redis for EvERY message. I can't imagine too many messages being passed that the receiver doesn't need the payload for it to actually do work. So, that process would need to fetch the payload from Redis (somehow?)

This is fine for LARGE payloads but I think for smaller ones (TBD but maybe 1K or less?) it maybe more efficient to just pass the payload itself. This is just a hunch from 30+ years of doing things like this. :)

Or -- perhaps I am missing the whole point. LOL

[emadum]

Yes you're 100% right. We did a little benchmarking of this PR and that quickly became obvious.

I've been busy lately, but plan on doing some more benchmarking using 1k, 5k and 10k message size cutoffs.

[sjmcdowall]

I would go even more than that unless someone knows the “Redis” limit of bad payload sizes (I haven’t seen one).

Perhaps the benchmarks should go a “logarithmic” route and start with like 1K 10K 100K 1M 10M 100M 1G (the last should probably be enough!!) :)

I would also suggest, if going this route, serious consideration be made to look into possible compression of payload to/from Redis. Again, from experience there will be some “magic” # where compression makes sense and below not ..

Finally, I would HIGHLY HIGHLY suggest that no matter what set of numbers are good defaults, give the “user” the ability to over-ride within the config. init area. Or heck, even “on the fly” perhaps. The point though is that any one number based on the benchmark results on a specific set of h/w may not be the best when running native in the wild. :)

Cheers!

On Feb 16, 2018, at 9:13 AM, Eric Adum notifications@github.com wrote:

Yes you're 100% right. We did a little benchmarking of this PR and that quickly became obvious.

I've been busy lately, but plan on doing some more benchmarking using 1k, 5k and 10k message size cutoffs.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/flywheelsports/hydra/issues/169#issuecomment-366245842, or mute the thread https://github.com/notifications/unsubscribe-auth/AB8M7UgsXoi6HmHoIfbK4ueaAgXI8ty0ks5tVYz9gaJpZM4R4tMh.