search

salesforce / storm-dynamic-spout

A framework for building spouts for Apache Storm and a Kafka based spout for dynamically skipping messages to be processed later.
BSD 3-Clause "New" or "Revised" License
40 stars 13 forks source link

Add consumer id to consumer record #41

Closed daniel-dara closed 6 years ago

daniel-dara commented 6 years ago

The consumer id is a handy piece of information that could be used by a topology component to determine which virtual spout a tuple is associated with. Adding the consumer id to the consumer record and passing it into the deserializer's deserialize method should allow users to extend the deserializer to make use of it. This came out of a conversation with Stan so he should have more details if necessary.

stanlemon commented 6 years ago

@Crim curious if you have an opinion as to whether or not this is a String for consumer id or an actual VirtualSpoutIdentifier. Depends how loosely coupled we want the VirtualSpout from the Consumer interface I suppose.

stanlemon commented 6 years ago

Closing per comment on #44

Not 100% sure I agree. Part of the goal of sidelining is so that you don't have to worry what spout something originates from. The more I think about this the less I am sold that we should add this API surface area without a clear use case to back it up. Adding this parameter requires every implementing interface to pass this data. Not sure that's a necessary requirement just yet. If we can come up with a good use case for adding this I'm game, but I think for now I'm going to sideline (har har) this PR until we do.