search

Yelp / pyleus

Pyleus is a Python framework for developing and launching Storm topologies.
Apache License 2.0
404 stars 113 forks source link

debug pylues in the cluster #119

Closed hadisotudeh closed 9 years ago

hadisotudeh commented 9 years ago

Hi I want to know how could we debug pyleus projects in the cluster? Where are the errors stored? For example: I got the following error: java.lang.RuntimeException: backtype.storm.multilang.NoOutputException: Pipe to subprocess seems to be broken! No output read. Serializer Exception: Traceback (most recent call last): File "/usr/lib but I don't know what is the reason of this error. I've used json as a serializer , should I use msgpack?

ecanzonieri commented 9 years ago

Hi, This error is usually due to a failure in the python process (perhaps a bug in your python spout / bolt). I would suggest you to enable logging in your python code ( https://yelp.github.io/pyleus/logging.html).

On Sunday, May 10, 2015, hadisotudeh notifications@github.com wrote:

Hi I want to know how could we debug pyleus projects in the cluster? Where are the errors stored? For example: I got the following error: java.lang.RuntimeException: backtype.storm.multilang.NoOutputException: Pipe to subprocess seems to be broken! No output read. Serializer Exception: Traceback (most recent call last): File "/usr/lib but I don't know what is the reason of this error. I've used json as a serializer , should I use msgpack?

— Reply to this email directly or view it on GitHub https://github.com/Yelp/pyleus/issues/119.

hadisotudeh commented 9 years ago

Thanks, but I think if we have a runtime error we should get the traceback of that error not put log.info in every line of our code to find the problem. For example:
My project is OK at local mode but in the cluster I get the following error in one of my bolts. java.lang.RuntimeException: Error when launching multilang subprocess Traceback (most recent call last): File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main "main", fname, l How should I detect the cause?

poros commented 9 years ago

If you enable loggin in your Python code, the traceback you are looking for is going to be logged there as well. Look at component.py code if you want to have more insight about how it works.

I would recommend setting up your topology for logging in any case. This is good practice for any sane application.

poros commented 9 years ago

Hello, is this still an issue?

hadisotudeh commented 9 years ago

No, It's OK