search

Yelp / pyleus

Pyleus is a Python framework for developing and launching Storm topologies.
Apache License 2.0
404 stars 113 forks source link

Is pyleus 0.2.4 compatible with Storm 0.9.3 ? #86

Closed William-Sang closed 9 years ago

William-Sang commented 9 years ago

Today I set up a storm 0.9.3 cluster, and run 

$ git clone https://github.com/Yelp/pyleus.git
$ pyleus build pyleus/examples/exclamation_topology/pyleus_topology.yaml
$ pyleus --verbose local exclamation_topology.jar

Than it occurs: 

......
9491 [Thread-11-exclaim2] INFO  backtype.storm.task.ShellBolt - Start checking heartbeat...
9492 [Thread-11-exclaim2] INFO  backtype.storm.daemon.executor - Prepared bolt exclaim2:(3)
10502 [Thread-22] ERROR backtype.storm.daemon.executor -
java.lang.Exception: Shell Process Exception: Traceback (most recent call last):
  File "/tmp/d05ae596-2b00-4eaa-9c4f-17c7f6ee5667/supervisor/stormdist/exclamation_topology-1-1418406574/resources/pyleus_venv/lib/python2.7/site-packages/pyleus/storm/component.py", line 233, in run
    self.run_component()
  File "/tmp/d05ae596-2b00-4eaa-9c4f-17c7f6ee5667/supervisor/stormdist/exclamation_topology-1-1418406574/resources/pyleus_venv/lib/python2.7/site-packages/pyleus/storm/bolt.py", line 46, in run_component
    self._process_tuple(tup)
  File "/tmp/d05ae596-2b00-4eaa-9c4f-17c7f6ee5667/supervisor/stormdist/exclamation_topology-1-1418406574/resources/pyleus_venv/lib/python2.7/site-packages/pyleus/storm/bolt.py", line 164, in _process_tuple
    self.process_tuple(tup)
  File "/tmp/d05ae596-2b00-4eaa-9c4f-17c7f6ee5667/supervisor/stormdist/exclamation_topology-1-1418406574/resources/exclamation_topology/exclamation_bolt.py", line 18, in process_tuple
    word = tup.values[0] + "!!!"
IndexError: list index out of range

    at backtype.storm.task.ShellBolt.handleError(ShellBolt.java:188) [storm-core-0.9.3.jar:0.9.3]
    at backtype.storm.task.ShellBolt.access$1100(ShellBolt.java:69) [storm-core-0.9.3.jar:0.9.3]
    at backtype.storm.task.ShellBolt$BoltReaderRunnable.run(ShellBolt.java:331) [storm-core-0.9.3.jar:0.9.3]
    at java.lang.Thread.run(Thread.java:745) [na:1.7.0_72]
10503 [Thread-20] ERROR backtype.storm.daemon.executor -
java.lang.Exception: Shell Process Exception: Traceback (most recent call last):
  File "/tmp/d05ae596-2b00-4eaa-9c4f-17c7f6ee5667/supervisor/stormdist/exclamation_topology-1-1418406574/resources/pyleus_venv/lib/python2.7/site-packages/pyleus/storm/component.py", line 233, in run
    self.run_component()
  File "/tmp/d05ae596-2b00-4eaa-9c4f-17c7f6ee5667/supervisor/stormdist/exclamation_topology-1-1418406574/resources/pyleus_venv/lib/python2.7/site-packages/pyleus/storm/bolt.py", line 46, in run_component
    self._process_tuple(tup)
  File "/tmp/d05ae596-2b00-4eaa-9c4f-17c7f6ee5667/supervisor/stormdist/exclamation_topology-1-1418406574/resources/pyleus_venv/lib/python2.7/site-packages/pyleus/storm/bolt.py", line 164, in _process_tuple
    self.process_tuple(tup)
  File "/tmp/d05ae596-2b00-4eaa-9c4f-17c7f6ee5667/supervisor/stormdist/exclamation_topology-1-1418406574/resources/exclamation_topology/exclamation_bolt.py", line 18, in process_tuple
    word = tup.values[0] + "!!!"
IndexError: list index out of range

    at backtype.storm.task.ShellBolt.handleError(ShellBolt.java:188) [storm-core-0.9.3.jar:0.9.3]
    at backtype.storm.task.ShellBolt.access$1100(ShellBolt.java:69) [storm-core-0.9.3.jar:0.9.3]
    at backtype.storm.task.ShellBolt$BoltReaderRunnable.run(ShellBolt.java:331) [storm-core-0.9.3.jar:0.9.3]
    at java.lang.Thread.run(Thread.java:745) [na:1.7.0_72]
10509 [ProcessThread(sid:0 cport:-1):] INFO  org.apache.storm.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x14a3f9ea9e1000b type:create cxid:0x33 zxid:0x22 txntype:-1 reqpath:n/a Error Path:/storm/errors/exclamation_topology-1-1418406574 Error:KeeperErrorCode = NodeExists for /storm/errors/exclamation_topology-1-1418406574
10519 [Thread-22] ERROR backtype.storm.task.ShellBolt - Halting process: ShellBolt died.
java.io.EOFException: null
    at org.msgpack.io.StreamInput.readByte(StreamInput.java:60) ~[exclamation_topology.jar:na]
    at org.msgpack.unpacker.MessagePackUnpacker.getHeadByte(MessagePackUnpacker.java:66) ~[exclamation_topology.jar:na]
    at org.msgpack.unpacker.MessagePackUnpacker.trySkipNil(MessagePackUnpacker.java:396) ~[exclamation_topology.jar:na]
    at org.msgpack.template.MapTemplate.read(MapTemplate.java:59) ~[exclamation_topology.jar:na]
    at org.msgpack.template.MapTemplate.read(MapTemplate.java:27) ~[exclamation_topology.jar:na]
    at org.msgpack.template.AbstractTemplate.read(AbstractTemplate.java:31) ~[exclamation_topology.jar:na]
    at org.msgpack.MessagePack.read(MessagePack.java:527) ~[exclamation_topology.jar:na]
    at org.msgpack.MessagePack.read(MessagePack.java:496) ~[exclamation_topology.jar:na]
    at com.yelp.pyleus.serializer.MessagePackSerializer.readMessage(MessagePackSerializer.java:198) ~[exclamation_topology.jar:na]
    at com.yelp.pyleus.serializer.MessagePackSerializer.readShellMsg(MessagePackSerializer.java:74) ~[exclamation_topology.jar:na]
    at backtype.storm.utils.ShellProcess.readShellMsg(ShellProcess.java:99) ~[storm-core-0.9.3.jar:0.9.3]
    at backtype.storm.task.ShellBolt$BoltReaderRunnable.run(ShellBolt.java:318) ~[storm-core-0.9.3.jar:0.9.3]
    at java.lang.Thread.run(Thread.java:745) [na:1.7.0_72]
10519 [Thread-22] ERROR backtype.storm.daemon.executor -
java.io.EOFException: null
    at org.msgpack.io.StreamInput.readByte(StreamInput.java:60) ~[exclamation_topology.jar:na]
    at org.msgpack.unpacker.MessagePackUnpacker.getHeadByte(MessagePackUnpacker.java:66) ~[exclamation_topology.jar:na]
    at org.msgpack.unpacker.MessagePackUnpacker.trySkipNil(MessagePackUnpacker.java:396) ~[exclamation_topology.jar:na]
    at org.msgpack.template.MapTemplate.read(MapTemplate.java:59) ~[exclamation_topology.jar:na]
    at org.msgpack.template.MapTemplate.read(MapTemplate.java:27) ~[exclamation_topology.jar:na]
    at org.msgpack.template.AbstractTemplate.read(AbstractTemplate.java:31) ~[exclamation_topology.jar:na]
    at org.msgpack.MessagePack.read(MessagePack.java:527) ~[exclamation_topology.jar:na]
    at org.msgpack.MessagePack.read(MessagePack.java:496) ~[exclamation_topology.jar:na]
    at com.yelp.pyleus.serializer.MessagePackSerializer.readMessage(MessagePackSerializer.java:198) ~[exclamation_topology.jar:na]
    at com.yelp.pyleus.serializer.MessagePackSerializer.readShellMsg(MessagePackSerializer.java:74) ~[exclamation_topology.jar:na]
    at backtype.storm.utils.ShellProcess.readShellMsg(ShellProcess.java:99) ~[storm-core-0.9.3.jar:0.9.3]
    at backtype.storm.task.ShellBolt$BoltReaderRunnable.run(ShellBolt.java:318) ~[storm-core-0.9.3.jar:0.9.3]
    at java.lang.Thread.run(Thread.java:745) [na:1.7.0_72]
10520 [Thread-20] ERROR backtype.storm.task.ShellBolt - Halting process: ShellBolt died.
java.io.EOFException: null
    at org.msgpack.io.StreamInput.readByte(StreamInput.java:60) ~[exclamation_topology.jar:na]
    at org.msgpack.unpacker.MessagePackUnpacker.getHeadByte(MessagePackUnpacker.java:66) ~[exclamation_topology.jar:na]
    at org.msgpack.unpacker.MessagePackUnpacker.trySkipNil(MessagePackUnpacker.java:396) ~[exclamation_topology.jar:na]
    at org.msgpack.template.MapTemplate.read(MapTemplate.java:59) ~[exclamation_topology.jar:na]
    at org.msgpack.template.MapTemplate.read(MapTemplate.java:27) ~[exclamation_topology.jar:na]
    at org.msgpack.template.AbstractTemplate.read(AbstractTemplate.java:31) ~[exclamation_topology.jar:na]
    at org.msgpack.MessagePack.read(MessagePack.java:527) ~[exclamation_topology.jar:na]
    at org.msgpack.MessagePack.read(MessagePack.java:496) ~[exclamation_topology.jar:na]
    at com.yelp.pyleus.serializer.MessagePackSerializer.readMessage(MessagePackSerializer.java:198) ~[exclamation_topology.jar:na]
    at com.yelp.pyleus.serializer.MessagePackSerializer.readShellMsg(MessagePackSerializer.java:74) ~[exclamation_topology.jar:na]
    at backtype.storm.utils.ShellProcess.readShellMsg(ShellProcess.java:99) ~[storm-core-0.9.3.jar:0.9.3]
    at backtype.storm.task.ShellBolt$BoltReaderRunnable.run(ShellBolt.java:318) ~[storm-core-0.9.3.jar:0.9.3]
    at java.lang.Thread.run(Thread.java:745) [na:1.7.0_72]
10521 [Thread-20] ERROR backtype.storm.daemon.executor -
java.io.EOFException: null
    at org.msgpack.io.StreamInput.readByte(StreamInput.java:60) ~[exclamation_topology.jar:na]
    at org.msgpack.unpacker.MessagePackUnpacker.getHeadByte(MessagePackUnpacker.java:66) ~[exclamation_topology.jar:na]
    at org.msgpack.unpacker.MessagePackUnpacker.trySkipNil(MessagePackUnpacker.java:396) ~[exclamation_topology.jar:na]
    at org.msgpack.template.MapTemplate.read(MapTemplate.java:59) ~[exclamation_topology.jar:na]
    at org.msgpack.template.MapTemplate.read(MapTemplate.java:27) ~[exclamation_topology.jar:na]
    at org.msgpack.template.AbstractTemplate.read(AbstractTemplate.java:31) ~[exclamation_topology.jar:na]
    at org.msgpack.MessagePack.read(MessagePack.java:527) ~[exclamation_topology.jar:na]
    at org.msgpack.MessagePack.read(MessagePack.java:496) ~[exclamation_topology.jar:na]
    at com.yelp.pyleus.serializer.MessagePackSerializer.readMessage(MessagePackSerializer.java:198) ~[exclamation_topology.jar:na]
    at com.yelp.pyleus.serializer.MessagePackSerializer.readShellMsg(MessagePackSerializer.java:74) ~[exclamation_topology.jar:na]
    at backtype.storm.utils.ShellProcess.readShellMsg(ShellProcess.java:99) ~[storm-core-0.9.3.jar:0.9.3]
    at backtype.storm.task.ShellBolt$BoltReaderRunnable.run(ShellBolt.java:318) ~[storm-core-0.9.3.jar:0.9.3]
    at java.lang.Thread.run(Thread.java:745) [na:1.7.0_72]
10526 [Thread-7] INFO  backtype.storm.daemon.nimbus - Shutting down master

......
ecanzonieri commented 9 years ago

It should come soon, there is a pending pull request to make it compatible #9. But currently we do not support storm 0.9.3.

William-Sang commented 9 years ago

@ecanzonieri We really appreciate your effort.

moandcompany commented 9 years ago

Hi,

I've been able to run Pyleus successfully with Storm 0.9.3.2.2.0.0 (Hortonworks HDP 2.2 Build).

kayrus commented 9 years ago

Any update on Storm 0.9.3 support?

poros commented 9 years ago

We have support for heartbeats (see #97), but we still lack a pull request for #10 and manual testing with Storm 0.9.3

jwestboston commented 9 years ago

Is #10 required for releasing a version (latest -develop) that supports Storm 0.9.3?

I am using Pyleus 0.2.4-develop with Storm 0.9.3 and can confirm heartbeats are working.

poros commented 9 years ago

As far as I know, it should be required. But I didn't try myself

hellp commented 9 years ago

For the record, I'm also using the latest develop branch and just changed the Storm dependency in pom.xml to "0.9.3", built is custom release of pyleus, and use this to build, deploy and run a topology on a 0.9.3 Storm. Works fine.

jwestboston commented 9 years ago

@poros Is the lack of a PR for #10 holding up the next release of Pyleus?

kayrus commented 9 years ago

I've just tried to compile pyleus-base.jar with storm 0.9.3, now I can build and submit topologies. But they don't work. Here is 0.9.3 storm-supervisors' log: http://pastebin.com/0LvrNizZ

The same code works fine on 0.9.2

kayrus commented 9 years ago

Here is 0.9.2 storm-supervisor log: http://pastebin.com/kn0Bwbxt

jwestboston commented 9 years ago

@kayrus are you sure you're running the latest -develop of Pyleus? I saw similar errors to that when running Pyleus 0.2.4 ... it was heartbeats that was choking things up, since Pyleus 0.2.4 doesn't know how to handle or unpack the heartbeat tuples.

kayrus commented 9 years ago

Yes, I use latest develop branch. Topology works, but very, very slow. And it produces error messages. Here is the full log from 0.9.3: http://pastebin.com/ZhW3xDZM

I've tried to reproduce the same behavior with 0.9.2 - it works well.

poros commented 9 years ago

@jwestboston As I said in another thread, I considered #10 as a blocking issue together with extensive testing with Storm 0.9.3. Storm is a complex system and it is not trivial to ensure that Pyleus does not trigger exotic bugs or degrades performance (as it seems to happen with 0.9.3).

However, I also feel that it is time for a new release. I'll coordinate with @patricklucas

poros commented 9 years ago

Update: I opened a pull request for #10, but I noticed we also need #107 for properly supporting Storm 0.9.3

poros commented 9 years ago

We switched directly to Storm 0.9.4. Please upgrade pyleus to version 0.3.0