search

AirSage / Petrel

Tools for writing, submitting, debugging, and monitoring Storm topologies in pure Python
BSD 3-Clause "New" or "Revised" License
247 stars 70 forks source link

Emitting from thread #63

Closed mdrio closed 9 years ago

mdrio commented 10 years ago

Hi, is it possible to emit from a thread in a bolt? I have a timer that periodically processes a list populated in MyBolt.process, then emits a tuple. Anyway nothing arrives to the receiving bolt. Am i missing something? Thanks in advance

barrywhart commented 10 years ago

The Petrel emit() function is pretty similar to the one provided in storm.py. I believe it just writes to stdout. If you're writing to stdout from another thread (especially if there are multiple threads doing so), it's possible you could have issues with the output being jumbled together.

Can you confirm that if you emit the same tuple from the main thread that the downstream bolt does receive it?

I recommend asking this question on the Storm discussion forum as well -- more people will see it and you may get a more definitive answer.

mdrio commented 9 years ago

I implemented a simple topology with a spout emitting from a thread and it fails with: java.lang.RuntimeException: java.io.IOException: unable to parse: {"command": "emit", "need_task_ids": false, "tuple": ["test"]} {"command": "sync"}

So it seems the stdout is jumbled as you said.