Open austinnichols101 opened 5 years ago
Hi, To workaround this problem, I tried to replace the agent with a task and combine streams generated using topics t1 and t2 using " &"
@app.task()
async def count_page_views_task():
s1 = app.stream(t1)
s2 = app.stream(t2)
async for view in (s1 & s2).group_by(PageView.id):
page_views[view.id] +=1
@app.timer(5)
async def process1():
await t1.send(value={"id": "foo1", "user": "bar"})
await t2.send(value={"id": "foo2", "user": "bar"})
$KAFKA_HOME/bin/kafka-console-consumer.sh --topic page_views-page_views-changelog --bootstrap-server localhost:9092 --property print.key=True --from-beginning
However, this prints counts only for foo1. Counts for foo2 are missing
“foo1” 1 “foo1" 2 “foo1” 3 “foo1" 4 “foo1” 5 “foo1" 6 “foo1” 7 “foo1" 8 “foo1” 9 “foo1" 10 “foo1” 11 “foo1" 12 “foo1” 13 “foo1" 14 “foo1” 15 “foo1" 16 “foo1” 17 “foo1" 18 “foo1” 19 “foo1" 20 “foo1” 21 “foo1" 22 “foo1” 23 “foo1" 24 “foo1” 25 “foo1" 26 “foo1” 27 “foo1" 28
Combining is not supported, we removed support for it as some point as it was not fully implemented.
In this case you should specify the repartitioning topic directly:
repartition_topic = app.topic('foo-bar-repartition-by-id', value_type=PageView)
async for view in stream.group_by(PageView.id, topic=repartition_topic):
ValueError: Topic with multiple topic names cannot be identified
The following code fails (taken from the PageView examples) fails when attempting to use
multiple_topics
. Switching the agent to a single topic (t1
ort2
instead ofmultiple_topics
) works perfectly.Versions