fonty422
commented
3 years ago Just one additional point of information as to why we were hoping to not use a key to force it to be sent to the same partition every time. Of the 50k keys, most of them are quiet and only produce one message every few minutes or so, but occasionally a few of them start producing a few a second. Since we plan to keep things in the log to 3 months for some mid-term statistical analysis (super basic - just counting and aggregating of time) we may end up with one partition having many millions of records, while others have only a few hundred. The hope was not to overload a single topic and share the load. But it's entirely possible that I've not got this very wrong and misunderstood the platform and in fact the Kafka system will more than happily handle this level of information so it's actually not a problem to force each instance of a particular key to the same partition.
Question I noted that with the word_count example, that the count table can be recalculated on a re-balance (using the @app.table_route option with the @app.page option). But is it possible to do this without the page option? Or can we call that internally on each worker to get the 'real' live count and then add to it?
Reason If we have possibly a few thousand messages per second and we need to have live counts for the last 10, 30, 60, and 300s for up to 50k keys, it seems like
It makes sense that this would be split across multiple workers It seems necessary for fault tolerance - one worker dies, then the others pick up the task. Point 2 is especially valid as this is a critical step in the processing pipeline for the end use so if it stops, then the whole thing stops.