search

Yelp / pyleus

Pyleus is a Python framework for developing and launching Storm topologies.
Apache License 2.0
404 stars 113 forks source link

Kafka Spout Error #113

Closed sumitchawla closed 9 years ago

sumitchawla commented 9 years ago

Hi All

I am trying to run Kafka Spout Example in HDP 2.2. I made some modifications by copying the logger_bolt, and logging.conf files from logging example. I am just trying to log the message received from Kafka. The code works perfect in local, but does not work in cluster. I reduced the Supervisors to 1, and that supervisor happens to be the same machine on which i am testing the local example.

Not sure what i am doing wrong here. Logging example was working without any issue. The error seems to be related to MessagePack. I am very new to Storm. Would really appreciate any help here? Also i cannot seem to figure out why its complaining about "No handlers could be found for logger "pyleus.storm.component"". Do i need to add any new loggers into my log file?

2015-04-21 13:47:12 b.s.util [ERROR] Async loop died!
java.lang.RuntimeException: Error when launching multilang subprocess
No handlers could be found for logger "pyleus.storm.component"
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/hadoop/storm/supervisor/stormdist/kafka_spout_example-11-1429648265/resources/sdc/logger_bolt.py", line 21, in <module>
    LoggerBolt().run()
  File "/usr/local/lib/python2.7/dist-packages/pyleus/storm/component.py", line 236, in run
    self.error(traceback.format_exc())
  File "/usr/local/lib/python2.7/dist-packages/pyleus/storm/component.py", line 332, in error
    'msg': msg,
  File "/usr/local/lib/python2.7/dist-packages/pyleus/storm/component.py", line 321, in send_command
    self._serializer.send_msg(command_dict)
AttributeError: 'NoneType' object has no attribute 'send_msg'

    at backtype.storm.utils.ShellProcess.launch(ShellProcess.java:66) ~[storm-core-0.9.3.2.2.4.2-2.jar:0.9.3.2.2.4.2-2]
    at backtype.storm.task.ShellBolt.prepare(ShellBolt.java:108) ~[storm-core-0.9.3.2.2.4.2-2.jar:0.9.3.2.2.4.2-2]
    at backtype.storm.daemon.executor$fn__4641$fn__4654.invoke(executor.clj:732) ~[storm-core-0.9.3.2.2.4.2-2.jar:0.9.3.2.2.4.2-2]
    at backtype.storm.util$async_loop$fn__551.invoke(util.clj:463) ~[storm-core-0.9.3.2.2.4.2-2.jar:0.9.3.2.2.4.2-2]
    at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
    at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
Caused by: java.io.EOFException: null
    at org.msgpack.io.StreamInput.readByte(StreamInput.java:60) ~[stormjar.jar:na]
    at org.msgpack.unpacker.MessagePackUnpacker.getHeadByte(MessagePackUnpacker.java:66) ~[stormjar.jar:na]
    at org.msgpack.unpacker.MessagePackUnpacker.trySkipNil(MessagePackUnpacker.java:396) ~[stormjar.jar:na]
    at org.msgpack.template.MapTemplate.read(MapTemplate.java:59) ~[stormjar.jar:na]
    at org.msgpack.template.MapTemplate.read(MapTemplate.java:27) ~[stormjar.jar:na]
    at org.msgpack.template.AbstractTemplate.read(AbstractTemplate.java:31) ~[stormjar.jar:na]
    at org.msgpack.MessagePack.read(MessagePack.java:527) ~[stormjar.jar:na]
    at org.msgpack.MessagePack.read(MessagePack.java:496) ~[stormjar.jar:na]
    at com.yelp.pyleus.serializer.MessagePackSerializer.readMessage(MessagePackSerializer.java:198) ~[stormjar.jar:na]
    at com.yelp.pyleus.serializer.MessagePackSerializer.connect(MessagePackSerializer.java:67) ~[stormjar.jar:na]
    at backtype.storm.utils.ShellProcess.launch(ShellProcess.java:64) ~[storm-core-0.9.3.2.2.4.2-2.jar:0.9.3.2.2.4.2-2]
    ... 5 common frames omitted
2015-04-21 13:47:12 o.a.z.ZooKeeper [INFO] Client environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
2015-04-21 13:47:12 o.a.z.ZooKeeper [INFO] Client environment:host.name=sumitc-ub12vm-hdp-1.dev.support.com
2015-04-21 13:47:12 o.a.z.ZooKeeper [INFO] Client environment:java.version=1.7.0_67
2015-04-21 13:47:12 o.a.z.ZooKeeper [INFO] Client environment:java.vendor=Oracle Corporation
2015-04-21 13:47:12 o.a.z.ZooKeeper [INFO] Client environment:java.home=/usr/jdk64/jdk1.7.0_67/jre
2015-04-21 13:47:12 b.s.d.executor [ERROR]
java.lang.RuntimeException: Error when launching multilang subprocess
No handlers could be found for logger "pyleus.storm.component"
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/hadoop/storm/supervisor/stormdist/kafka_spout_example-11-1429648265/resources/sdc/logger_bolt.py", line 21, in <module>
    LoggerBolt().run()
  File "/usr/local/lib/python2.7/dist-packages/pyleus/storm/component.py", line 236, in run
    self.error(traceback.format_exc())
  File "/usr/local/lib/python2.7/dist-packages/pyleus/storm/component.py", line 332, in error
    'msg': msg,
  File "/usr/local/lib/python2.7/dist-packages/pyleus/storm/component.py", line 321, in send_command
    self._serializer.send_msg(command_dict)
AttributeError: 'NoneType' object has no attribute 'send_msg'

    at backtype.storm.utils.ShellProcess.launch(ShellProcess.java:66) ~[storm-core-0.9.3.2.2.4.2-2.jar:0.9.3.2.2.4.2-2]
    at backtype.storm.task.ShellBolt.prepare(ShellBolt.java:108) ~[storm-core-0.9.3.2.2.4.2-2.jar:0.9.3.2.2.4.2-2]
    at backtype.storm.daemon.executor$fn__4641$fn__4654.invoke(executor.clj:732) ~[storm-core-0.9.3.2.2.4.2-2.jar:0.9.3.2.2.4.2-2]
    at backtype.storm.util$async_loop$fn__551.invoke(util.clj:463) ~[storm-core-0.9.3.2.2.4.2-2.jar:0.9.3.2.2.4.2-2]
    at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
    at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
Caused by: java.io.EOFException: null
    at org.msgpack.io.StreamInput.readByte(StreamInput.java:60) ~[stormjar.jar:na]
    at org.msgpack.unpacker.MessagePackUnpacker.getHeadByte(MessagePackUnpacker.java:66) ~[stormjar.jar:na]
    at org.msgpack.unpacker.MessagePackUnpacker.trySkipNil(MessagePackUnpacker.java:396) ~[stormjar.jar:na]
    at org.msgpack.template.MapTemplate.read(MapTemplate.java:59) ~[stormjar.jar:na]
    at org.msgpack.template.MapTemplate.read(MapTemplate.java:27) ~[stormjar.jar:na]
    at org.msgpack.template.AbstractTemplate.read(AbstractTemplate.java:31) ~[stormjar.jar:na]
    at org.msgpack.MessagePack.read(MessagePack.java:527) ~[stormjar.jar:na]
    at org.msgpack.MessagePack.read(MessagePack.java:496) ~[stormjar.jar:na]
    at com.yelp.pyleus.serializer.MessagePackSerializer.readMessage(MessagePackSerializer.java:198) ~[stormjar.jar:na]
    at com.yelp.pyleus.serializer.MessagePackSerializer.connect(MessagePackSerializer.java:67) ~[stormjar.jar:na]
    at backtype.storm.utils.ShellProcess.launch(ShellProcess.java:64) ~[storm-core-0.9.3.2.2.4.2-2.jar:0.9.3.2.2.4.2-2]
    ... 5 common frames omitted
Storm - 0.9.3.2.2.4.2-2
pyleus 0.2.4
virtualenv - 12.1.1

Excerpts from .jar file

resources/pyleus_venv/lib/python2.7/site-packages/pyleus/cli/virtualenv_proxy.py
resources/pyleus_venv/lib/python2.7/site-packages/pyleus/cli/storm_cluster.py
resources/pyleus_venv/lib/python2.7/site-packages/pyleus/cli/topologies.pyc
resources/pyleus_venv/lib/python2.7/site-packages/pyleus/cli/topologies.py
resources/pyleus_venv/lib/python2.7/site-packages/pyleus/cli/build.pyc
resources/pyleus_venv/lib/python2.7/site-packages/pyleus/cli/virtualenv_proxy.pyc

resources/pyleus_venv/lib/python2.7/site-packages/pyleus/storm/spout.py
resources/pyleus_venv/lib/python2.7/site-packages/pyleus/storm/spout.pyc
resources/pyleus_venv/lib/python2.7/site-packages/pyleus/storm/__init__.py
resources/pyleus_venv/lib/python2.7/site-packages/pyleus/storm/component.pyc
resources/pyleus_venv/lib/python2.7/site-packages/pyleus/storm/bolt.py
resources/pyleus_venv/lib/python2.7/site-packages/pyleus/storm/serializers/__init__.pyc
resources/pyleus_venv/lib/python2.7/site-packages/pyleus/storm/serializers/serializer.pyc
resources/pyleus_venv/lib/python2.7/site-packages/pyleus/storm/serializers/json_serializer.py
resources/pyleus_venv/lib/python2.7/site-packages/pyleus/storm/serializers/msgpack_serializer.pyc
resources/pyleus_venv/lib/python2.7/site-packages/pyleus/storm/serializers/msgpack_serializer.py
resources/pyleus_venv/lib/python2.7/site-packages/pyleus/storm/serializers/serializer.py
resources/pyleus_venv/lib/python2.7/site-packages/pyleus/storm/serializers/__init__.py
resources/pyleus_venv/lib/python2.7/site-packages/pyleus/storm/serializers/json_serializer.pyc

Entries in my .pyleus.conf

[storm]
# optional: use -n option of pyleus CLI instead
nimbus_host: 10.1.3.80

[build]
# PyPI server to use during the build of your topologies
pypi_index_url:https://pypi.python.org/pypi

# always use system-site-packages for pyleus virtualenvs (default: false)
#system_site_packages: true

# list of packages to always include in your topologies
include_packages: pyhdfs
poros commented 9 years ago

Let's try to switch to the JSON serializer to understand if that is an issue of our msgpack serializer. If that is the case, we may consider switch to json as default serializer in the next release, since many people are experiencing exotic bugs.

sumitchawla commented 9 years ago

Issue finally turned out to be a misconfigured log path. The log file path was pointing to a file in /tmp directory. Storm user did not seem to have access to that path. The things are working as expected once i corrected the log path.

tdyhacker commented 8 years ago

@sumitchawla hi, I encounter the same error as you, how you solve this, I also set the log path to /tmp