search

pystorm / streamparse

Run Python in Apache Storm topologies. Pythonic API, CLI tooling, and a topology DSL.
http://streamparse.readthedocs.io/
Apache License 2.0
1.5k stars 217 forks source link

Using msgpack with storm 1.0.1 throws KeyError #278

Closed amitripshtos closed 8 years ago

amitripshtos commented 8 years ago

hey, im trying to use msgpack with streamparse using Dan`s guide on one of the issues around this repo. Im using: streamparse 3.0.0 dev3 apache storm 1.0.1 python 3.5.1 java 1.8

It seems that it working on using msgpack in java, but I get KeyError in the "python" side of msgpack:

java.lang.RuntimeException: Error when launching multilang subprocess Traceback (most recent call last): File "/usr/local/bin/streamparse_run", line 9, in <module> load_entry_point('streamparse==3.0.0.dev3', 'console_scripts', 'streamparse_run')() File "/usr/local/lib/python3.5/site-packages/streamparse/run.py", line 37, in main cls(serializer=args.serializer).run() File "/usr/local/lib/python3.5/site-packages/pystorm/component.py", line 474, in run storm_conf, context = self.read_handshake() File "/usr/local/lib/python3.5/site-packages/pystorm/component.py", line 303, in read_handshake pid_dir, _conf, _context = msg['pidDir'], msg['conf'], msg['context'] KeyError: 'pidDir' at org.apache.storm.utils.ShellProcess.launch(ShellProcess.java:89) at org.apache.storm.spout.ShellSpout.open(ShellSpout.java:88) at org.apache.storm.daemon.executor$fn__7885$fn__7900.invoke(executor.clj:601) at org.apache.storm.util$async_loop$fn__625.invoke(util.clj:482) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.EOFException at org.msgpack.io.StreamInput.readByte(StreamInput.java:60) at org.msgpack.unpacker.MessagePackUnpacker.getHeadByte(MessagePackUnpacker.java:66) at org.msgpack.unpacker.MessagePackUnpacker.trySkipNil(MessagePackUnpacker.java:396) at org.msgpack.template.MapTemplate.read(MapTemplate.java:59) at org.msgpack.template.MapTemplate.read(MapTemplate.java:27) at org.msgpack.template.AbstractTemplate.read(AbstractTemplate.java:31) at org.msgpack.MessagePack.read(MessagePack.java:527) at org.msgpack.MessagePack.read(MessagePack.java:496) at com.yelp.pyleus.serializer.MessagePackSerializer.readMessage(MessagePackSerializer.java:205) at com.yelp.pyleus.serializer.MessagePackSerializer.connect(MessagePackSerializer.java:67) at org.apache.storm.utils.ShellProcess.launch(ShellProcess.java:85) ... 5 more

I could not fix that in any cases and I hope you guys have any solution thank you very much!

dan-blanchard commented 8 years ago

This issue was moved to pystorm/pystorm#35