search

Yelp / pyleus

Pyleus is a Python framework for developing and launching Storm topologies.
Apache License 2.0
403 stars 107 forks source link

Error when running examples or dummy sample #87

Closed msmakhlouf closed 9 years ago

msmakhlouf commented 9 years ago

I created the structure for the dummy sample and copied the code, tried to submit to a single node cluster on my local machine and I keep getting these errors in the logs:

2014-12-15T14:42:14.739+0300 my-first-bolt [INFO] No handlers could be found for logger "pyleus.storm.component"

2014-12-15T14:42:14.743+0300 b.s.d.executor [ERROR] java.lang.Exception: Shell Process Exception: Traceback (most recent call last): File "/home/mak/qarti/apache-storm-0.9.3/storm-local/supervisor/stormdist/my_first_topology-1-1418643533/resources/pyleus_venv/lib/python2.7/site-packages/pyleus/storm/component.py", line 233, in run self.run_component() File "/home/mak/qarti/apache-storm-0.9.3/storm-local/supervisor/stormdist/my_first_topology-1-1418643533/resources/pyleus_venv/lib/python2.7/site-packages/pyleus/storm/bolt.py", line 46, in run_component self._process_tuple(tup) File "/home/mak/qarti/apache-storm-0.9.3/storm-local/supervisor/stormdist/my_first_topology-1-1418643533/resources/pyleus_venv/lib/python2.7/site-packages/pyleus/storm/bolt.py", line 164, in _process_tuple self.process_tuple(tup) File "/home/mak/qarti/apache-storm-0.9.3/storm-local/supervisor/stormdist/my_first_topology-1-1418643533/resources/my_first_topology/dummy_bolt.py", line 11, in process_tuple sentence, name = tup.values ValueError: need more than 0 values to unpack

at backtype.storm.task.ShellBolt.handleError(ShellBolt.java:188) [storm-core-0.9.3.jar:0.9.3]
at backtype.storm.task.ShellBolt.access$1100(ShellBolt.java:69) [storm-core-0.9.3.jar:0.9.3]
at backtype.storm.task.ShellBolt$BoltReaderRunnable.run(ShellBolt.java:331) [storm-core-0.9.3.jar:0.9.3]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_72]

2014-12-15T14:42:14.747+0300 b.s.t.ShellBolt [ERROR] Halting process: ShellBolt died. java.io.IOException: Broken pipe at java.io.FileOutputStream.writeBytes(Native Method) ~[na:1.7.0_72] at java.io.FileOutputStream.write(FileOutputStream.java:345) ~[na:1.7.0_72] at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[na:1.7.0_72] at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[na:1.7.0_72] at java.io.DataOutputStream.flush(DataOutputStream.java:123) ~[na:1.7.0_72] at com.yelp.pyleus.serializer.MessagePackSerializer.writeMessage(MessagePackSerializer.java:208) ~[stormjar.jar:na] at com.yelp.pyleus.serializer.MessagePackSerializer.writeBoltMsg(MessagePackSerializer.java:181) ~[stormjar.jar:na] at backtype.storm.utils.ShellProcess.writeBoltMsg(ShellProcess.java:106) ~[storm-core-0.9.3.jar:0.9.3] at backtype.storm.task.ShellBolt$BoltWriterRunnable.run(ShellBolt.java:361) ~[storm-core-0.9.3.jar:0.9.3] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_72] 2014-12-15T14:42:14.747+0300 b.s.d.executor [ERROR] java.io.IOException: Broken pipe at java.io.FileOutputStream.writeBytes(Native Method) ~[na:1.7.0_72] at java.io.FileOutputStream.write(FileOutputStream.java:345) ~[na:1.7.0_72] at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[na:1.7.0_72] at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[na:1.7.0_72] at java.io.DataOutputStream.flush(DataOutputStream.java:123) ~[na:1.7.0_72] at com.yelp.pyleus.serializer.MessagePackSerializer.writeMessage(MessagePackSerializer.java:208) ~[stormjar.jar:na] at com.yelp.pyleus.serializer.MessagePackSerializer.writeBoltMsg(MessagePackSerializer.java:181) ~[stormjar.jar:na] at backtype.storm.utils.ShellProcess.writeBoltMsg(ShellProcess.java:106) ~[storm-core-0.9.3.jar:0.9.3] at backtype.storm.task.ShellBolt$BoltWriterRunnable.run(ShellBolt.java:361) ~[storm-core-0.9.3.jar:0.9.3] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_72] 2014-12-15T14:42:14.775+0300 b.s.t.ShellBolt [ERROR] Halting process: ShellBolt died. java.io.EOFException: null at org.msgpack.io.StreamInput.readByte(StreamInput.java:60) ~[stormjar.jar:na] at org.msgpack.unpacker.MessagePackUnpacker.getHeadByte(MessagePackUnpacker.java:66) ~[stormjar.jar:na] at org.msgpack.unpacker.MessagePackUnpacker.trySkipNil(MessagePackUnpacker.java:396) ~[stormjar.jar:na] at org.msgpack.template.MapTemplate.read(MapTemplate.java:59) ~[stormjar.jar:na] at org.msgpack.template.MapTemplate.read(MapTemplate.java:27) ~[stormjar.jar:na] at org.msgpack.template.AbstractTemplate.read(AbstractTemplate.java:31) ~[stormjar.jar:na] at org.msgpack.MessagePack.read(MessagePack.java:527) ~[stormjar.jar:na] at org.msgpack.MessagePack.read(MessagePack.java:496) ~[stormjar.jar:na] at com.yelp.pyleus.serializer.MessagePackSerializer.readMessage(MessagePackSerializer.java:198) ~[stormjar.jar:na] at com.yelp.pyleus.serializer.MessagePackSerializer.readShellMsg(MessagePackSerializer.java:74) ~[stormjar.jar:na] at backtype.storm.utils.ShellProcess.readShellMsg(ShellProcess.java:99) ~[storm-core-0.9.3.jar:0.9.3] at backtype.storm.task.ShellBolt$BoltReaderRunnable.run(ShellBolt.java:318) ~[storm-core-0.9.3.jar:0.9.3] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_72] 2014-12-15T14:42:14.775+0300 b.s.d.executor [ERROR] java.io.EOFException: null at org.msgpack.io.StreamInput.readByte(StreamInput.java:60) ~[stormjar.jar:na] at org.msgpack.unpacker.MessagePackUnpacker.getHeadByte(MessagePackUnpacker.java:66) ~[stormjar.jar:na] at org.msgpack.unpacker.MessagePackUnpacker.trySkipNil(MessagePackUnpacker.java:396) ~[stormjar.jar:na] at org.msgpack.template.MapTemplate.read(MapTemplate.java:59) ~[stormjar.jar:na] at org.msgpack.template.MapTemplate.read(MapTemplate.java:27) ~[stormjar.jar:na] at org.msgpack.template.AbstractTemplate.read(AbstractTemplate.java:31) ~[stormjar.jar:na] at org.msgpack.MessagePack.read(MessagePack.java:527) ~[stormjar.jar:na] at org.msgpack.MessagePack.read(MessagePack.java:496) ~[stormjar.jar:na] at com.yelp.pyleus.serializer.MessagePackSerializer.readMessage(MessagePackSerializer.java:198) ~[stormjar.jar:na] at com.yelp.pyleus.serializer.MessagePackSerializer.readShellMsg(MessagePackSerializer.java:74) ~[stormjar.jar:na] at backtype.storm.utils.ShellProcess.readShellMsg(ShellProcess.java:99) ~[storm-core-0.9.3.jar:0.9.3] at backtype.storm.task.ShellBolt$BoltReaderRunnable.run(ShellBolt.java:318) ~[storm-core-0.9.3.jar:0.9.3] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_72] 2014-12-15T14:42:14.778+0300 b.s.d.worker [INFO] Shutting down worker my_first_topology-1-1418643533 9b4f9ba9-1497-4122-940d-59730ebdd72c 6702 2014-12-15T14:42:14.778+0300 b.s.d.worker [INFO] Shutting down receive thread 2014-12-15T14:42:14.783+0300 o.a.s.c.r.ExponentialBackoffRetry [WARN] maxRetries too large (300). Pinning to 29 2014-12-15T14:42:14.783+0300 b.s.u.StormBoundedExponentialBackoffRetry [INFO] The baseSleepTimeMs [100] the maxSleepTimeMs [1000] the maxRetries [300] 2014-12-15T14:42:14.784+0300 b.s.m.n.Client [INFO] New Netty Client, connect to localhost, 6702, config: , buffer_size: 5242880 2014-12-15T14:42:14.785+0300 b.s.m.n.Client [INFO] Reconnect started for Netty-Client-localhost/127.0.0.1:6702... [0] 2014-12-15T14:42:14.785+0300 b.s.m.loader [INFO] Shutting down receiving-thread: [my_first_topology-1-1418643533, 6702]

msmakhlouf commented 9 years ago

When the bolt receives a heartbeat tuple the tup.values is empty and it causes this error leading to breaking the async loop since garbage text is written to stdout so storm protocol some how breaks or stops.

Can some one correct me if I am wrong here?

msmakhlouf commented 9 years ago

Seems to be duplicate of #86 .. just ran it against wrong version 0.9.3