search

pystorm / streamparse

Run Python in Apache Storm topologies. Pythonic API, CLI tooling, and a topology DSL.
http://streamparse.readthedocs.io/
Apache License 2.0
1.49k stars 218 forks source link

`sparse` broken on Python 3.3 #86

Closed hadisotudeh closed 9 years ago

hadisotudeh commented 10 years ago

Hi, I've run the sparse command in the terminal and get the following result:

[hadi@hadi Desktop]$ sparse
Traceback (most recent call last):
  File "/usr/bin/sparse", line 9, in <module>
    load_entry_point('streamparse==1.0.1', 'console_scripts', 'sparse')()
  File "/usr/lib/python3.3/site-packages/pkg_resources.py", line 339, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python3.3/site-packages/pkg_resources.py", line 2470, in load_entry_point
    return ep.load()
  File "/usr/lib/python3.3/site-packages/pkg_resources.py", line 2184, in load
    ['__name__'])
  File "/usr/lib/python3.3/site-packages/streamparse/__init__.py", line 11, in <module>
    import streamparse.cmdln
  File "/usr/lib/python3.3/site-packages/streamparse/cmdln.py", line 6, in <module>
    from .ext.invoke import (list_topologies, kill_topology, run_local_topology,
  File "/usr/lib/python3.3/site-packages/streamparse/ext/invoke.py", line 27, in <module>
    from .fabric import activate_env, create_or_update_virtualenvs, tail_logs
  File "/usr/lib/python3.3/site-packages/streamparse/ext/fabric.py", line 17, in <module>
    from fabric.api import env, execute, parallel, prefix, put, puts, run, task
  File "/usr/lib/python3.3/site-packages/fabric/api.py", line 9, in <module>
    from fabric.context_managers import (cd, hide, settings, show, path, prefix,
  File "/usr/lib/python3.3/site-packages/fabric/context_managers.py", line 535
    def accept(channel, (src_addr, src_port), (dest_addr, dest_port)):
                        ^
SyntaxError: invalid syntax

What should I do?

dan-blanchard commented 10 years ago

Are you using the fork of fabric listed in the requirements file?

hadisotudeh commented 10 years ago

@dan-blanchard I do that, but now i get the following error.

[hadi@hadi wordcount]$ sparse run -t 12090
Running wordcount topology...
Routing Python logging to /home/hadi/STORM/wordcount/logs.
Running lein command to run local cluster:
lein run -m streamparse.commands.run/-main topologies/wordcount.clj -t 12090 --option 'topology.workers=2' --option 'topology.acker.executors=2' --option 'streamparse.log.path="/home/hadi/STORM/wordcount/logs"' --option 'streamparse.log.level="debug"'
1730 [main] INFO  backtype.storm.zookeeper - Starting inprocess zookeeper at port 2000 and dir /tmp/c95f0304-feae-40fa-a0ff-61a198b59bab
1936 [main] INFO  backtype.storm.daemon.nimbus - Starting Nimbus with conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 0, "topology.skip.missing.kryo.registrations" true, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 50, "storm.messaging.netty.flush.check.interval.ms" 10, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "/tmp/9dabe2bd-bd0e-4c24-950f-3b24b394638f", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "localhost", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2000, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["localhost"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 30, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" [6700 6701 6702 6703], "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer" "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx1024m", "storm.cluster.mode" "local", "topology.max.task.parallelism" nil, "storm.messaging.netty.transfer.batch.size" 262144}
1952 [main] INFO  backtype.storm.daemon.nimbus - Using default scheduler
2041 [main] INFO  org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
2123 [main-SendThread(localhost.localdomain:2000)] ERROR org.apache.zookeeper.ClientCnxnSocketNIO - Unable to open socket to localhost.localdomain/0:0:0:0:0:0:0:1:2000
2128 [main-SendThread(localhost.localdomain:2000)] WARN  org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.SocketException: Network is unreachable
    at sun.nio.ch.Net.connect0(Native Method) ~[na:1.8.0_20]
    at sun.nio.ch.Net.connect(Net.java:457) ~[na:1.8.0_20]
    at sun.nio.ch.Net.connect(Net.java:449) ~[na:1.8.0_20]
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:647) ~[na:1.8.0_20]
    at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:266) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
    at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:276) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
    at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:958) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
2256 [main-EventThread] INFO  org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
2258 [main-EventThread] INFO  backtype.storm.zookeeper - Zookeeper state update: :connected:none
2259 [ConnectionStateManager-0] WARN  org.apache.curator.framework.state.ConnectionStateManager - There are no ConnectionStateListeners registered.
3318 [main] INFO  org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
3321 [main-SendThread(localhost.localdomain:2000)] ERROR org.apache.zookeeper.ClientCnxnSocketNIO - Unable to open socket to localhost.localdomain/0:0:0:0:0:0:0:1:2000
3321 [main-SendThread(localhost.localdomain:2000)] WARN  org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.SocketException: Network is unreachable
    at sun.nio.ch.Net.connect0(Native Method) ~[na:1.8.0_20]
    at sun.nio.ch.Net.connect(Net.java:457) ~[na:1.8.0_20]
    at sun.nio.ch.Net.connect(Net.java:449) ~[na:1.8.0_20]
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:647) ~[na:1.8.0_20]
    at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:266) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
    at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:276) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
    at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:958) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
3425 [main-EventThread] INFO  org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
3425 [ConnectionStateManager-0] WARN  org.apache.curator.framework.state.ConnectionStateManager - There are no ConnectionStateListeners registered.
4478 [main] INFO  org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
4484 [main-EventThread] INFO  org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
4484 [ConnectionStateManager-0] WARN  org.apache.curator.framework.state.ConnectionStateManager - There are no ConnectionStateListeners registered.
4485 [main-EventThread] INFO  backtype.storm.zookeeper - Zookeeper state update: :connected:none
4493 [main] INFO  org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
4498 [main-SendThread(localhost.localdomain:2000)] ERROR org.apache.zookeeper.ClientCnxnSocketNIO - Unable to open socket to localhost.localdomain/0:0:0:0:0:0:0:1:2000
4498 [main-SendThread(localhost.localdomain:2000)] WARN  org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.SocketException: Network is unreachable
    at sun.nio.ch.Net.connect0(Native Method) ~[na:1.8.0_20]
    at sun.nio.ch.Net.connect(Net.java:457) ~[na:1.8.0_20]
    at sun.nio.ch.Net.connect(Net.java:449) ~[na:1.8.0_20]
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:647) ~[na:1.8.0_20]
    at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:266) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
    at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:276) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
    at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:958) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
4499 [main] INFO  org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
4511 [main-SendThread(localhost.localdomain:2000)] ERROR org.apache.zookeeper.ClientCnxnSocketNIO - Unable to open socket to localhost.localdomain/0:0:0:0:0:0:0:1:2000
4511 [main-SendThread(localhost.localdomain:2000)] WARN  org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.SocketException: Network is unreachable
    at sun.nio.ch.Net.connect0(Native Method) ~[na:1.8.0_20]
    at sun.nio.ch.Net.connect(Net.java:457) ~[na:1.8.0_20]
    at sun.nio.ch.Net.connect(Net.java:449) ~[na:1.8.0_20]
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:647) ~[na:1.8.0_20]
    at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:266) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
    at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:276) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
    at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:958) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
4603 [main-EventThread] INFO  org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
4603 [ConnectionStateManager-0] WARN  org.apache.curator.framework.state.ConnectionStateManager - There are no ConnectionStateListeners registered.
4614 [main-EventThread] INFO  org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
4614 [ConnectionStateManager-0] WARN  org.apache.curator.framework.state.ConnectionStateManager - There are no ConnectionStateListeners registered.
4614 [main-EventThread] INFO  backtype.storm.zookeeper - Zookeeper state update: :connected:none
4620 [main] INFO  org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
4626 [main-EventThread] INFO  org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
4626 [ConnectionStateManager-0] WARN  org.apache.curator.framework.state.ConnectionStateManager - There are no ConnectionStateListeners registered.
5644 [main] INFO  backtype.storm.daemon.supervisor - Starting Supervisor with conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 0, "topology.skip.missing.kryo.registrations" true, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 50, "storm.messaging.netty.flush.check.interval.ms" 10, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "/tmp/64c26b8f-f4f8-4ede-a50e-46d57ab34e46", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "localhost", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2000, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["localhost"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 30, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" (1024 1025 1026), "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer" "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx1024m", "storm.cluster.mode" "local", "topology.max.task.parallelism" nil, "storm.messaging.netty.transfer.batch.size" 262144}
5716 [main] INFO  org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
5718 [main-SendThread(localhost.localdomain:2000)] ERROR org.apache.zookeeper.ClientCnxnSocketNIO - Unable to open socket to localhost.localdomain/0:0:0:0:0:0:0:1:2000
5718 [main-SendThread(localhost.localdomain:2000)] WARN  org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.SocketException: Network is unreachable
    at sun.nio.ch.Net.connect0(Native Method) ~[na:1.8.0_20]
    at sun.nio.ch.Net.connect(Net.java:457) ~[na:1.8.0_20]
    at sun.nio.ch.Net.connect(Net.java:449) ~[na:1.8.0_20]
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:647) ~[na:1.8.0_20]
    at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:266) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
    at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:276) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
    at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:958) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
5825 [main-EventThread] INFO  org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
5826 [ConnectionStateManager-0] WARN  org.apache.curator.framework.state.ConnectionStateManager - There are no ConnectionStateListeners registered.
5826 [main-EventThread] INFO  backtype.storm.zookeeper - Zookeeper state update: :connected:none
5835 [main] INFO  org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
5844 [main-EventThread] INFO  org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
5844 [ConnectionStateManager-0] WARN  org.apache.curator.framework.state.ConnectionStateManager - There are no ConnectionStateListeners registered.
6892 [main] INFO  backtype.storm.daemon.supervisor - Starting supervisor with id 2335325e-291c-44a1-a2c0-1c0a9056628e at host hadi
6902 [main] INFO  backtype.storm.daemon.supervisor - Starting Supervisor with conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 0, "topology.skip.missing.kryo.registrations" true, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 50, "storm.messaging.netty.flush.check.interval.ms" 10, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "/tmp/a342c329-6e91-40e5-b342-0a6605228254", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "localhost", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2000, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["localhost"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 30, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" (1027 1028 1029), "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer" "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx1024m", "storm.cluster.mode" "local", "topology.max.task.parallelism" nil, "storm.messaging.netty.transfer.batch.size" 262144}
6917 [main] INFO  org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
6923 [main-EventThread] INFO  org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
6924 [main-EventThread] INFO  backtype.storm.zookeeper - Zookeeper state update: :connected:none
6927 [ConnectionStateManager-0] WARN  org.apache.curator.framework.state.ConnectionStateManager - There are no ConnectionStateListeners registered.
7931 [main] INFO  org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
7939 [main-SendThread(localhost.localdomain:2000)] ERROR org.apache.zookeeper.ClientCnxnSocketNIO - Unable to open socket to localhost.localdomain/0:0:0:0:0:0:0:1:2000
7941 [main-SendThread(localhost.localdomain:2000)] WARN  org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.SocketException: Network is unreachable
    at sun.nio.ch.Net.connect0(Native Method) ~[na:1.8.0_20]
    at sun.nio.ch.Net.connect(Net.java:457) ~[na:1.8.0_20]
    at sun.nio.ch.Net.connect(Net.java:449) ~[na:1.8.0_20]
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:647) ~[na:1.8.0_20]
    at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:266) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
    at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:276) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
    at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:958) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
8044 [main-EventThread] INFO  org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
8044 [ConnectionStateManager-0] WARN  org.apache.curator.framework.state.ConnectionStateManager - There are no ConnectionStateListeners registered.
8050 [main] INFO  backtype.storm.daemon.supervisor - Starting supervisor with id fa66c251-0f21-44a9-a8d8-603f497720c3 at host hadi
8104 [main] INFO  backtype.storm.daemon.nimbus - Received topology submission for wordcount with conf {"storm.id" "wordcount-1-1417787406", "topology.acker.executors" 2, "streamparse.log.path" "/home/hadi/STORM/wordcount/logs", "topology.kryo.decorators" (), "topology.name" "wordcount", "topology.max.spout.pending" 5000, "topology.debug" false, "topology.kryo.register" nil, "topology.message.timeout.secs" 60, "topology.workers" 2, "topology.max.task.parallelism" nil, "streamparse.log.level" "debug"}
8159 [main] INFO  backtype.storm.daemon.nimbus - Activating wordcount: wordcount-1-1417787406
8264 [main] INFO  backtype.storm.scheduler.EvenScheduler - Available slots: (["fa66c251-0f21-44a9-a8d8-603f497720c3" 1027] ["fa66c251-0f21-44a9-a8d8-603f497720c3" 1028] ["fa66c251-0f21-44a9-a8d8-603f497720c3" 1029] ["2335325e-291c-44a1-a2c0-1c0a9056628e" 1024] ["2335325e-291c-44a1-a2c0-1c0a9056628e" 1025] ["2335325e-291c-44a1-a2c0-1c0a9056628e" 1026])
8321 [main] INFO  backtype.storm.daemon.nimbus - Setting new assignment for topology id wordcount-1-1417787406: #backtype.storm.daemon.common.Assignment{:master-code-dir "/tmp/9dabe2bd-bd0e-4c24-950f-3b24b394638f/nimbus/stormdist/wordcount-1-1417787406", :node->host {"fa66c251-0f21-44a9-a8d8-603f497720c3" "hadi", "2335325e-291c-44a1-a2c0-1c0a9056628e" "hadi"}, :executor->node+port {[4 4] ["2335325e-291c-44a1-a2c0-1c0a9056628e" 1024], [3 3] ["fa66c251-0f21-44a9-a8d8-603f497720c3" 1027], [2 2] ["2335325e-291c-44a1-a2c0-1c0a9056628e" 1024], [1 1] ["fa66c251-0f21-44a9-a8d8-603f497720c3" 1027], [5 5] ["fa66c251-0f21-44a9-a8d8-603f497720c3" 1027]}, :executor->start-time-secs {[2 2] 1417787406, [4 4] 1417787406, [5 5] 1417787406, [1 1] 1417787406, [3 3] 1417787406}}
8356 [Thread-3] INFO  backtype.storm.daemon.supervisor - Downloading code for storm id wordcount-1-1417787406 from /tmp/9dabe2bd-bd0e-4c24-950f-3b24b394638f/nimbus/stormdist/wordcount-1-1417787406
8797 [Thread-3] INFO  backtype.storm.daemon.supervisor - Copying resources at file:/home/hadi/STORM/wordcount/_resources/resources to /tmp/64c26b8f-f4f8-4ede-a50e-46d57ab34e46/supervisor/stormdist/wordcount-1-1417787406/resources
8801 [Thread-3] INFO  backtype.storm.daemon.supervisor - Finished downloading code for storm id wordcount-1-1417787406 from /tmp/9dabe2bd-bd0e-4c24-950f-3b24b394638f/nimbus/stormdist/wordcount-1-1417787406
8827 [Thread-4] INFO  backtype.storm.daemon.supervisor - Launching worker with assignment #backtype.storm.daemon.supervisor.LocalAssignment{:storm-id "wordcount-1-1417787406", :executors ([4 4] [2 2])} for this supervisor 2335325e-291c-44a1-a2c0-1c0a9056628e on port 1024 with id b0897d6f-e778-4767-809f-d0b08530f888
8832 [Thread-4] INFO  backtype.storm.daemon.worker - Launching worker for wordcount-1-1417787406 on 2335325e-291c-44a1-a2c0-1c0a9056628e:1024 with id b0897d6f-e778-4767-809f-d0b08530f888 and conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 0, "topology.skip.missing.kryo.registrations" true, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 50, "storm.messaging.netty.flush.check.interval.ms" 10, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "/tmp/64c26b8f-f4f8-4ede-a50e-46d57ab34e46", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "localhost", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2000, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["localhost"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 30, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" (1024 1025 1026), "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer" "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx1024m", "storm.cluster.mode" "local", "topology.max.task.parallelism" nil, "storm.messaging.netty.transfer.batch.size" 262144}
8835 [Thread-4] INFO  org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
8847 [Thread-4-SendThread(localhost.localdomain:2000)] ERROR org.apache.zookeeper.ClientCnxnSocketNIO - Unable to open socket to localhost.localdomain/0:0:0:0:0:0:0:1:2000
8848 [Thread-4-SendThread(localhost.localdomain:2000)] WARN  org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.SocketException: Network is unreachable
    at sun.nio.ch.Net.connect0(Native Method) ~[na:1.8.0_20]
    at sun.nio.ch.Net.connect(Net.java:457) ~[na:1.8.0_20]
    at sun.nio.ch.Net.connect(Net.java:449) ~[na:1.8.0_20]
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:647) ~[na:1.8.0_20]
    at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:266) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
    at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:276) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
    at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:958) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
8927 [Thread-5] INFO  backtype.storm.daemon.supervisor - Downloading code for storm id wordcount-1-1417787406 from /tmp/9dabe2bd-bd0e-4c24-950f-3b24b394638f/nimbus/stormdist/wordcount-1-1417787406
8955 [Thread-4-EventThread] INFO  org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
8955 [Thread-4-EventThread] INFO  backtype.storm.zookeeper - Zookeeper state update: :connected:none
8955 [ConnectionStateManager-0] WARN  org.apache.curator.framework.state.ConnectionStateManager - There are no ConnectionStateListeners registered.
8963 [Thread-4] INFO  org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
8970 [Thread-4-EventThread] INFO  org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
8971 [ConnectionStateManager-0] WARN  org.apache.curator.framework.state.ConnectionStateManager - There are no ConnectionStateListeners registered.
9114 [Thread-5] INFO  backtype.storm.daemon.supervisor - Copying resources at file:/home/hadi/STORM/wordcount/_resources/resources to /tmp/a342c329-6e91-40e5-b342-0a6605228254/supervisor/stormdist/wordcount-1-1417787406/resources
9117 [Thread-5] INFO  backtype.storm.daemon.supervisor - Finished downloading code for storm id wordcount-1-1417787406 from /tmp/9dabe2bd-bd0e-4c24-950f-3b24b394638f/nimbus/stormdist/wordcount-1-1417787406
9134 [Thread-6] INFO  backtype.storm.daemon.supervisor - Launching worker with assignment #backtype.storm.daemon.supervisor.LocalAssignment{:storm-id "wordcount-1-1417787406", :executors ([3 3] [1 1] [5 5])} for this supervisor fa66c251-0f21-44a9-a8d8-603f497720c3 on port 1027 with id 4ebbb85b-ef97-4c44-8790-4cad4107f22f
9140 [Thread-6] INFO  backtype.storm.daemon.worker - Launching worker for wordcount-1-1417787406 on fa66c251-0f21-44a9-a8d8-603f497720c3:1027 with id 4ebbb85b-ef97-4c44-8790-4cad4107f22f and conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 0, "topology.skip.missing.kryo.registrations" true, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 50, "storm.messaging.netty.flush.check.interval.ms" 10, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "/tmp/a342c329-6e91-40e5-b342-0a6605228254", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "localhost", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2000, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["localhost"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 30, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" (1027 1028 1029), "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer" "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx1024m", "storm.cluster.mode" "local", "topology.max.task.parallelism" nil, "storm.messaging.netty.transfer.batch.size" 262144}
9155 [Thread-6] INFO  org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
9186 [Thread-6-EventThread] INFO  org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
9186 [ConnectionStateManager-0] WARN  org.apache.curator.framework.state.ConnectionStateManager - There are no ConnectionStateListeners registered.
9186 [Thread-6-EventThread] INFO  backtype.storm.zookeeper - Zookeeper state update: :connected:none
9404 [Thread-4] INFO  backtype.storm.daemon.executor - Loading executor __acker:[2 2]
9413 [Thread-4] INFO  backtype.storm.daemon.executor - Loaded executor tasks __acker:[2 2]
9423 [Thread-4] INFO  backtype.storm.daemon.executor - Timeouts disabled for executor __acker:[2 2]
9426 [Thread-4] INFO  backtype.storm.daemon.executor - Finished loading executor __acker:[2 2]
9425 [Thread-8-__acker] INFO  backtype.storm.daemon.executor - Preparing bolt __acker:(2)
9435 [Thread-8-__acker] INFO  backtype.storm.daemon.executor - Prepared bolt __acker:(2)
9445 [Thread-4] INFO  backtype.storm.daemon.executor - Loading executor count-bolt:[4 4]
9448 [Thread-4] INFO  backtype.storm.daemon.executor - Loaded executor tasks count-bolt:[4 4]
9451 [Thread-4] INFO  backtype.storm.daemon.executor - Finished loading executor count-bolt:[4 4]
9453 [Thread-10-count-bolt] INFO  backtype.storm.daemon.executor - Preparing bolt count-bolt:(4)
9458 [Thread-4] INFO  backtype.storm.daemon.executor - Loading executor __system:[-1 -1]
9459 [Thread-4] INFO  backtype.storm.daemon.executor - Loaded executor tasks __system:[-1 -1]
9462 [Thread-10-count-bolt] INFO  backtype.storm.utils.ShellProcess - Storm multilang serializer: backtype.storm.multilang.JsonSerializer
9465 [Thread-4] INFO  backtype.storm.daemon.executor - Finished loading executor __system:[-1 -1]
9465 [Thread-4] INFO  backtype.storm.daemon.worker - Launching receive-thread for 2335325e-291c-44a1-a2c0-1c0a9056628e:1024
9465 [Thread-12-__system] INFO  backtype.storm.daemon.executor - Preparing bolt __system:(-1)
9484 [Thread-13-worker-receiver-thread-0] INFO  backtype.storm.messaging.loader - Starting receive-thread: [stormId: wordcount-1-1417787406, port: 1024, thread-id: 0 ]
9486 [Thread-12-__system] INFO  backtype.storm.daemon.executor - Prepared bolt __system:(-1)
9499 [Thread-4] INFO  backtype.storm.daemon.worker - Worker has topology config {"storm.id" "wordcount-1-1417787406", "dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 0, "topology.skip.missing.kryo.registrations" true, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 50, "storm.messaging.netty.flush.check.interval.ms" 10, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "/tmp/64c26b8f-f4f8-4ede-a50e-46d57ab34e46", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "localhost", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2000, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["localhost"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" 2, "streamparse.log.path" "/home/hadi/STORM/wordcount/logs", "topology.kryo.decorators" (), "topology.name" "wordcount", "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 30, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" 5000, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" (1024 1025 1026), "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.kryo.register" nil, "topology.message.timeout.secs" 60, "task.refresh.poll.secs" 10, "topology.workers" 2, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer" "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx1024m", "storm.cluster.mode" "local", "topology.max.task.parallelism" nil, "storm.messaging.netty.transfer.batch.size" 262144, "streamparse.log.level" "debug"}
9504 [Thread-4] INFO  backtype.storm.daemon.worker - Worker b0897d6f-e778-4767-809f-d0b08530f888 for storm wordcount-1-1417787406 on 2335325e-291c-44a1-a2c0-1c0a9056628e:1024 has finished loading
9548 [Thread-10-count-bolt] ERROR backtype.storm.util - Async loop died!
java.lang.RuntimeException: backtype.storm.multilang.NoOutputException: Pipe to subprocess seems to be broken! No output read.
Serializer Exception:
/usr/bin/python: Error while finding spec for 'streamparse.run' (<class 'ImportError'>: No module named 'streamparse')

    at backtype.storm.utils.ShellProcess.launch(ShellProcess.java:68) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
    at backtype.storm.task.ShellBolt.prepare(ShellBolt.java:99) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
    at backtype.storm.daemon.executor$fn__3352$fn__3364.invoke(executor.clj:690) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
    at backtype.storm.util$async_loop$fn__452.invoke(util.clj:429) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
    at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_20]
9549 [Thread-10-count-bolt] ERROR backtype.storm.daemon.executor - 
java.lang.RuntimeException: backtype.storm.multilang.NoOutputException: Pipe to subprocess seems to be broken! No output read.
Serializer Exception:
/usr/bin/python: Error while finding spec for 'streamparse.run' (<class 'ImportError'>: No module named 'streamparse')

    at backtype.storm.utils.ShellProcess.launch(ShellProcess.java:68) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
    at backtype.storm.task.ShellBolt.prepare(ShellBolt.java:99) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
    at backtype.storm.daemon.executor$fn__3352$fn__3364.invoke(executor.clj:690) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
    at backtype.storm.util$async_loop$fn__452.invoke(util.clj:429) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
    at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_20]
9585 [Thread-10-count-bolt] INFO  backtype.storm.util - Halting process: ("Worker died")
Traceback (most recent call last):
  File "/usr/bin/sparse", line 9, in <module>
    load_entry_point('streamparse==1.0.1', 'console_scripts', 'sparse')()
  File "/usr/lib/python3.3/site-packages/streamparse/cmdln.py", line 82, in main
    run_local_topology(args["--name"], time, par, options, args["--debug"])
  File "/usr/lib/python3.3/site-packages/invoke/tasks.py", line 111, in __call__
    result = self.body(*args, **kwargs)
  File "/usr/lib/python3.3/site-packages/streamparse/ext/invoke.py", line 174, in run_local_topology
    run(full_cmd)
  File "/usr/lib/python3.3/site-packages/invoke/runner.py", line 160, in run
    raise Failure(result)
invoke.exceptions.Failure: Command execution failure!

Exit code: 1

Stderr:

Please, help me.

hadisotudeh commented 10 years ago

Finally, I solved to problem. I've installed python3.3 but my default python was python3.4 so the problem was here :D then I set my default python to 3.3 by following commands: ln -sf /usr/bin/python3.3 /usr/local/bin/python ln -sf /usr/bing/python3.3 /usr/bin/python

please, notice that you should install streamparse with pip3.3 not pip ( if your python version is > 3.3 )

ysimonson commented 9 years ago

I'm having the same problem. Here's been my experience so far getting started with streamparse.

1) Install streamparse: pip3 install streamparse -- using pip3 b/c we use python 3 in production 2) Get the SyntaxError, so I manually install the fabric fork: pip3 install -e git+https://github.com/pashinin/fabric@p33#egg=fabric. This works and fixes the error. 3) Run it and get the same second error (No module named 'streamparse').

It's trying to run the scripts via the system python installation at /usr/bin/python, where streamparse is not installed. If I symlinked like @hadisotudeh, it would almost certainly break my setup, as OS X expects python 2.

Is it storm or streamparse that's referencing system python? Is there a way to change it?

dan-blanchard commented 9 years ago

I would recommend setting up an Anaconda Python 3 installation so that it is completely isolated from the OS X system Python. That's what I do.

ysimonson commented 9 years ago

But streamparse is referencing system python, so that won't fix anything right?

dan-blanchard commented 9 years ago

No, streamparse does not use the system Python installation. It will use whichever Python the virtual/conda environment you have it installed in uses.

omus commented 9 years ago

You could also use pyenv which is available from Homebrew.

vlad-dragos commented 9 years ago

Hello,

So from my understanding streamparse only works with python3.3? (no 3.4 support)?

dan-blanchard commented 9 years ago

It should work with 3.3 or 3.4 (probably even 3.2). The key is that you just don't want to use the system Python installation on OS X.

vlad-dragos commented 9 years ago

@dan-blanchard I'm not. I'm using the version for Homebrew. I will do some more research and if I find anything worth sharing I will post an update here. :)

ysimonson commented 9 years ago

Yeah, this worked for me: 

virtualenv -p python3.4 venv
source venv/bin/activate
pip install streamparse
pip install -e git+https://github.com/pashinin/fabric@p33#egg=fabric
sparse
vlad-dragos commented 9 years ago

Thx @ysimonson it works using the snippet you provided. Maybe this should be added to the documenation? Under Dependencies?