Buffering producer

[dongdongcai]

Added buffering producer, buffer is flushed by timeout, count and time interval between arrivals

[bartelink]

This seems a bit specific for inclusion in Kafunk itself, except as an an example of a usage pattern. I have some requirements that have some degree of overlap with these needs, but my sense is that for example attempting to accomplish the union of those and the ones encompassed by this likely to yield a very complex API.

If on the other hand, there is an intention to add something like this to Kafunk, I may be able to find time to define the concerns of my usage patterns in more depth @eulerfx

[bartelink]

@dongdongcai Perhaps you can clarify the context of your submission - what usage pattern and load are you accomodating in your context ? Does this abstraction cover multiple scenarios you have; has it been used in >1 different place ?

[eulerfx]

@bartelink this covers the usage pattern we discussed during our call. The idea is to provide an async mechanism for producing messages to Kafka. A common use-case is telemetry/log data.

• The reason for including in Kafunk directly is to provide a batteries included way to get going with this. While I agree that the logic is itself external to Kafunk and can even be adapted to other use-cases, some care is required to get it right and I'd prefer to have it here rather than hope that users get it right. Alternatively, we can simply show an example implementation in the docs.

• What concerns have you?

[bartelink]

@eulerfx I don't have any particular concerns re the impl

I appreciate the desire for batteries included implementations; knowing that a particular specific producer arrangement is working in production over a significant timespan is immensely valuable.

My main goal here is to assess the maturity if this particular implementation; I have some upcoming work which won't be directly addressable by this as it is, but also isn't that far away. I'll likely do that independently (possibly based on this), prove it in prod and then circle back here with either a PR that converges this scheme with the scheme I arrive at, or venture it as a separated thing.

Unfortunately I'm not in a position to e.g. proffer a PR or detailed comments on this one so would not wish to block it's inclusion. (In short, I'm looking to provide a specific mix of backpressure / guarantees of specific handling if the Kafka cluster is unreachable which are specific to the app in question)

Perhaps (if this is the case) signalling that this is intended for a specific scenario (putting it in an Experimental or Topologies namespace or module (or giving it a very specific-sounding name)) and/or whether it's intended to be generalized over time by a doc comment or a docs page regarding it can leave it open to the degree necessary for us to arrive at a set of strategies that can be flexibly rearranged to meet diverse needs vs ending up with a set of baroque implementations that overlap in semantics but not in terms of a converged API.

In summary, definitely behind getting something in and gaining experience (esp with multiple usage scenarios), not seeking to block or have any specific concerns addressed.

[eulerfx]

Closing as I merged these in separately.