search

wintoncode / winton-kafka-streams

A Python implementation of Apache Kafka Streams
Apache License 2.0
313 stars 55 forks source link

Correctly associate source with a topic #23

Closed llawall closed 7 years ago

llawall commented 7 years ago

Minor refactor of building a topology. Ensures sources are kept in a dictionary so a stream task can call process on correct source for the current record.

llawall commented 7 years ago

This commit is pulled out of some changes I was making to implement a proof of concept DSL for the project. It should (I hope) be useful to @phoenix10k

phoenix10k commented 7 years ago

Great, I'm hoping to submit a PR adding a test/example based on https://github.com/confluentinc/examples/blob/3.2.x/kafka-streams/src/test/java/io/confluent/examples/streams/StreamToStreamJoinIntegrationTest.java soon (and any code required to get it to work :) )

llawall commented 7 years ago

@phoenix10k Excellent - looking forward to it. Are you intending to put together the DSL pieces too or will you implement the example using the more fundamental processsors?

llawall commented 7 years ago

@satlank Fair point about calling the private function. I'm not 100% happy with the way this is now either though I prefer it slightly more than before where nodes had to be passed into every one of the builder functions.

phoenix10k commented 7 years ago

I should have said soon™. How is the DSL coming along? I think it would be nice to use it for this simple example if we know pretty much what we want it to look like already, but I'm happy to stick to the processor API too if we're still a little way off, and I don't want to bite off too much at once.