search

Yelp / pyleus

Pyleus is a Python framework for developing and launching Storm topologies.
Apache License 2.0
403 stars 107 forks source link

Help needed, spout reading from redis keyspacenotification, bolt storing it in mongo #140

Closed anees042 closed 9 years ago

anees042 commented 9 years ago

Hello All,

I am new to Pyleus, I wrote a topology with one spout and one bolt. The spout is supposed to read redis Keyspace notifications, fetch the changed key and value data. The bolt store the received key and value in mongo. I am always getting java.lang.RuntimeException: subprocess heartbeat timeout

Please help me find out what i am doing wrong.

Below is my code;

.pyleus.conf

[storm]
# path to Storm executable (pyleus will automatically look in PATH)
storm_cmd_path: /home/fali/apache-storm-0.9.4/bin/storm
nimbus_host: 192.168.1.133
jvm_opts: -Djava.io.tmpdir=/home/fali/tmp

pyleus_topology.yaml

name: redistomongotopology

workers: 2

topology:

- spout:
    name: readredisdata-spout
    module: redistomongotopology.readredisdata_spout

- bolt:
    name: writetomongo_bolt
    module: redistomongotopology.writetomongo_bolt
    groupings:
        - shuffle_grouping: readredisdata-spout

readredisdata_spout.py

from pyleus.storm import Spout

import redis 
import threading
import logging
import time
import simplejson as json
from Queue import Queue
log = logging.getLogger("redistomongotopology.readredisdata_spout")

class ReadredisdataSpout(Spout):

#OUTPUT_FIELDS = ['timestamp']
INTERVAL = 5; # seconds

OUTPUT_FIELDS = ['sentence'];
last_emit = None;
r = redis.StrictRedis();
#r = redis.StrictRedis(host='192.168.1.196', port=6379, db=0)
pubsub = r.pubsub();

def initialize(self):
    self.last_emit = time.time()
    self.queue = Queue();
    thread = threading.Thread(target=self.getredisdata,args=())
    thread.start()

def getredisdata(self):
    r = redis.StrictRedis()
    pubsub = r.pubsub()
    pubsub.psubscribe("__keyspace@0__:*")
    while True:
        now = time.time()           
        msg = pubsub.get_message()
        if msg:
            pattern = msg.get('pattern')
            mtype = msg.get('type')
            channel = msg.get('channel')
            data = msg.get('data')
            log.info("Emitted: "+channel+" %r",now)
            if data=='set' or 'SET' and mtype=='pmessage':
                k = channel.split(':')
                #print k
                key = k[1]
                self.queue.put(key)
def next_tuple(self):
    now = time.time()
    if not self.queue.empty():
    key = self.queue.get()
        log.info("Emitting: "+key+" %r",now)
        self.queue.task_done()
        self.emit((key,))
        self.last_emit = now
    else:
        time.sleep(0.100)

if name == 'main': logging.basicConfig( level=logging.DEBUG, filename='/tmp/readredisdata_spout.log', filemode='a', ) ReadredisdataSpout().run()

writetomongo_bolt.py

from pyleus.storm import SimpleBolt
import simplejson as json
import logging
import redis
import time
from pymongo import MongoClient

class WritetomongoBolt(SimpleBolt):

OUTPUT_FIELDS = ['sentence']
mongoclient = MongoClient('mongodb://localhost:27017/')
r = redis.StrictRedis()
db = mongoclient.redislogsdb
def process_tuple(self, tup):
    now = time.time()
    sentence = tup.values[0]
    log = logging.getLogger("redistomongotopology.writetomongo_bolt")
    log.info("sentence: "+sentence+" %r",now)
    new_sentence = "spout says, \"{0}\"".format(sentence)
    #get key from sentence
    key = sentence
    log.info("key: "+key+" %r",now)
    logids=[]
    val = self.r.hgetall(key)
    logs = self.db.logs_collection
    record = {"_id":key+str(time.time()), "value":val}
    logs_id=logs.insert_one(record).inserted_id
    #print logs_id
    log.info("logs_id: "+logs_id+" %r",now)
    #logids.append(log_id)
    self.emit((new_sentence,), anchors=[tup])

if name == 'main': logging.basicConfig( level=logging.DEBUG, filename='/tmp/writetomongo_bolt.log', filemode='a', ) WritetomongoBolt().run()

requirements.txt

simplejson
redis
pymongo

Logs

5039 [main] INFO org.apache.storm.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 5041 [main] INFO org.apache.storm.zookeeper.ZooKeeper - Client environment:host.name=ubuntu 5041 [main] INFO org.apache.storm.zookeeper.ZooKeeper - Client environment:java.version=1.7.0_79 5041 [main] INFO org.apache.storm.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation 5041 [main] INFO org.apache.storm.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre 5042 [main] INFO org.apache.storm.zookeeper.ZooKeeper - Client environment:java.class.path=/home/fali/apache-storm-0.9.5/lib/tools.macro-0.1.0.jar:/home/fali/apache-storm-0.9.5/lib/servlet-api-2.5.jar:/home/fali/apache-storm-0.9.5/lib/slf4j-api-1.7.5.jar:/home/fali/apache-storm-0.9.5/lib/chill-java-0.3.5.jar:/home/fali/apache-storm-0.9.5/lib/commons-io-2.4.jar:/home/fali/apache-storm-0.9.5/lib/kryo-2.21.jar:/home/fali/apache-storm-0.9.5/lib/ring-devel-0.3.11.jar:/home/fali/apache-storm-0.9.5/lib/log4j-over-slf4j-1.6.6.jar:/home/fali/apache-storm-0.9.5/lib/clout-1.0.1.jar:/home/fali/apache-storm-0.9.5/lib/ring-jetty-adapter-0.3.11.jar:/home/fali/apache-storm-0.9.5/lib/ring-servlet-0.3.11.jar:/home/fali/apache-storm-0.9.5/lib/logback-core-1.0.13.jar:/home/fali/apache-storm-0.9.5/lib/json-simple-1.1.jar:/home/fali/apache-storm-0.9.5/lib/jetty-util-6.1.26.jar:/home/fali/apache-storm-0.9.5/lib/logback-classic-1.0.13.jar:/home/fali/apache-storm-0.9.5/lib/storm-core-0.9.5.jar:/home/fali/apache-storm-0.9.5/lib/carbonite-1.4.0.jar:/home/fali/apache-storm-0.9.5/lib/ring-core-1.1.5.jar:/home/fali/apache-storm-0.9.5/lib/clojure-1.5.1.jar:/home/fali/apache-storm-0.9.5/lib/commons-logging-1.1.3.jar:/home/fali/apache-storm-0.9.5/lib/reflectasm-1.07-shaded.jar:/home/fali/apache-storm-0.9.5/lib/commons-codec-1.6.jar:/home/fali/apache-storm-0.9.5/lib/jetty-6.1.26.jar:/home/fali/apache-storm-0.9.5/lib/tools.logging-0.2.3.jar:/home/fali/apache-storm-0.9.5/lib/commons-exec-1.1.jar:/home/fali/apache-storm-0.9.5/lib/jgrapht-core-0.9.0.jar:/home/fali/apache-storm-0.9.5/lib/tools.cli-0.2.4.jar:/home/fali/apache-storm-0.9.5/lib/disruptor-2.10.1.jar:/home/fali/apache-storm-0.9.5/lib/commons-fileupload-1.2.1.jar:/home/fali/apache-storm-0.9.5/lib/core.incubator-0.1.0.jar:/home/fali/apache-storm-0.9.5/lib/compojure-1.1.3.jar:/home/fali/apache-storm-0.9.5/lib/clj-time-0.4.1.jar:/home/fali/apache-storm-0.9.5/lib/minlog-1.2.jar:/home/fali/apache-storm-0.9.5/lib/math.numeric-tower-0.0.1.jar:/home/fali/apache-storm-0.9.5/lib/asm-4.0.jar:/home/fali/apache-storm-0.9.5/lib/hiccup-0.3.6.jar:/home/fali/apache-storm-0.9.5/lib/joda-time-2.0.jar:/home/fali/apache-storm-0.9.5/lib/snakeyaml-1.11.jar:/home/fali/apache-storm-0.9.5/lib/commons-lang-2.5.jar:/home/fali/apache-storm-0.9.5/lib/clj-stacktrace-0.2.2.jar:/home/fali/apache-storm-0.9.5/lib/jline-2.11.jar:/home/fali/apache-storm-0.9.5/lib/objenesis-1.2.jar:redistomongotopology.jar:/home/fali/apache-storm-0.9.5/conf:/home/fali/apache-storm-0.9.5/bin 5042 [main] INFO org.apache.storm.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/lib/jvm/java-7-openjdk-amd64 5042 [main] INFO org.apache.storm.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/home/fali/tmp 5042 [main] INFO org.apache.storm.zookeeper.ZooKeeper - Client environment:java.compiler= 5042 [main] INFO org.apache.storm.zookeeper.ZooKeeper - Client environment:os.name=Linux 5042 [main] INFO org.apache.storm.zookeeper.ZooKeeper - Client environment:os.arch=amd64 5042 [main] INFO org.apache.storm.zookeeper.ZooKeeper - Client environment:os.version=3.13.0-51-generic 5042 [main] INFO org.apache.storm.zookeeper.ZooKeeper - Client environment:user.name=fali 5042 [main] INFO org.apache.storm.zookeeper.ZooKeeper - Client environment:user.home=/home/fali 5042 [main] INFO org.apache.storm.zookeeper.ZooKeeper - Client environment:user.dir=/home/fali 5072 [main] INFO org.apache.storm.zookeeper.server.ZooKeeperServer - Server environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 5075 [main] INFO org.apache.storm.zookeeper.server.ZooKeeperServer - Server environment:host.name=ubuntu 5075 [main] INFO org.apache.storm.zookeeper.server.ZooKeeperServer - Server environment:java.version=1.7.0_79 5076 [main] INFO org.apache.storm.zookeeper.server.ZooKeeperServer - Server environment:java.vendor=Oracle Corporation 5076 [main] INFO org.apache.storm.zookeeper.server.ZooKeeperServer - Server environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre 5080 [main] INFO org.apache.storm.zookeeper.server.ZooKeeperServer - Server environment:java.class.path=/home/fali/apache-storm-0.9.5/lib/tools.macro-0.1.0.jar:/home/fali/apache-storm-0.9.5/lib/servlet-api-2.5.jar:/home/fali/apache-storm-0.9.5/lib/slf4j-api-1.7.5.jar:/home/fali/apache-storm-0.9.5/lib/chill-java-0.3.5.jar:/home/fali/apache-storm-0.9.5/lib/commons-io-2.4.jar:/home/fali/apache-storm-0.9.5/lib/kryo-2.21.jar:/home/fali/apache-storm-0.9.5/lib/ring-devel-0.3.11.jar:/home/fali/apache-storm-0.9.5/lib/log4j-over-slf4j-1.6.6.jar:/home/fali/apache-storm-0.9.5/lib/clout-1.0.1.jar:/home/fali/apache-storm-0.9.5/lib/ring-jetty-adapter-0.3.11.jar:/home/fali/apache-storm-0.9.5/lib/ring-servlet-0.3.11.jar:/home/fali/apache-storm-0.9.5/lib/logback-core-1.0.13.jar:/home/fali/apache-storm-0.9.5/lib/json-simple-1.1.jar:/home/fali/apache-storm-0.9.5/lib/jetty-util-6.1.26.jar:/home/fali/apache-storm-0.9.5/lib/logback-classic-1.0.13.jar:/home/fali/apache-storm-0.9.5/lib/storm-core-0.9.5.jar:/home/fali/apache-storm-0.9.5/lib/carbonite-1.4.0.jar:/home/fali/apache-storm-0.9.5/lib/ring-core-1.1.5.jar:/home/fali/apache-storm-0.9.5/lib/clojure-1.5.1.jar:/home/fali/apache-storm-0.9.5/lib/commons-logging-1.1.3.jar:/home/fali/apache-storm-0.9.5/lib/reflectasm-1.07-shaded.jar:/home/fali/apache-storm-0.9.5/lib/commons-codec-1.6.jar:/home/fali/apache-storm-0.9.5/lib/jetty-6.1.26.jar:/home/fali/apache-storm-0.9.5/lib/tools.logging-0.2.3.jar:/home/fali/apache-storm-0.9.5/lib/commons-exec-1.1.jar:/home/fali/apache-storm-0.9.5/lib/jgrapht-core-0.9.0.jar:/home/fali/apache-storm-0.9.5/lib/tools.cli-0.2.4.jar:/home/fali/apache-storm-0.9.5/lib/disruptor-2.10.1.jar:/home/fali/apache-storm-0.9.5/lib/commons-fileupload-1.2.1.jar:/home/fali/apache-storm-0.9.5/lib/core.incubator-0.1.0.jar:/home/fali/apache-storm-0.9.5/lib/compojure-1.1.3.jar:/home/fali/apache-storm-0.9.5/lib/clj-time-0.4.1.jar:/home/fali/apache-storm-0.9.5/lib/minlog-1.2.jar:/home/fali/apache-storm-0.9.5/lib/math.numeric-tower-0.0.1.jar:/home/fali/apache-storm-0.9.5/lib/asm-4.0.jar:/home/fali/apache-storm-0.9.5/lib/hiccup-0.3.6.jar:/home/fali/apache-storm-0.9.5/lib/joda-time-2.0.jar:/home/fali/apache-storm-0.9.5/lib/snakeyaml-1.11.jar:/home/fali/apache-storm-0.9.5/lib/commons-lang-2.5.jar:/home/fali/apache-storm-0.9.5/lib/clj-stacktrace-0.2.2.jar:/home/fali/apache-storm-0.9.5/lib/jline-2.11.jar:/home/fali/apache-storm-0.9.5/lib/objenesis-1.2.jar:redistomongotopology.jar:/home/fali/apache-storm-0.9.5/conf:/home/fali/apache-storm-0.9.5/bin 5081 [main] INFO org.apache.storm.zookeeper.server.ZooKeeperServer - Server environment:java.library.path=/usr/lib/jvm/java-7-openjdk-amd64 5081 [main] INFO org.apache.storm.zookeeper.server.ZooKeeperServer - Server environment:java.io.tmpdir=/home/fali/tmp 5081 [main] INFO org.apache.storm.zookeeper.server.ZooKeeperServer - Server environment:java.compiler= 5081 [main] INFO org.apache.storm.zookeeper.server.ZooKeeperServer - Server environment:os.name=Linux 5081 [main] INFO org.apache.storm.zookeeper.server.ZooKeeperServer - Server environment:os.arch=amd64 5081 [main] INFO org.apache.storm.zookeeper.server.ZooKeeperServer - Server environment:os.version=3.13.0-51-generic 5081 [main] INFO org.apache.storm.zookeeper.server.ZooKeeperServer - Server environment:user.name=fali 5082 [main] INFO org.apache.storm.zookeeper.server.ZooKeeperServer - Server environment:user.home=/home/fali 5082 [main] INFO org.apache.storm.zookeeper.server.ZooKeeperServer - Server environment:user.dir=/home/fali 5793 [main] INFO org.apache.storm.zookeeper.server.ZooKeeperServer - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /home/fali/tmp/454e97e6-b3f1-44e2-9828-2cd274007ef4/version-2 snapdir /home/fali/tmp/454e97e6-b3f1-44e2-9828-2cd274007ef4/version-2 5833 [main] INFO org.apache.storm.zookeeper.server.NIOServerCnxnFactory - binding to port 0.0.0.0/0.0.0.0:2000 5853 [main] INFO org.apache.storm.zookeeper.server.NIOServerCnxnFactory - binding to port 0.0.0.0/0.0.0.0:2001 5872 [main] INFO backtype.storm.zookeeper - Starting inprocess zookeeper at port 2001 and dir /home/fali/tmp/454e97e6-b3f1-44e2-9828-2cd274007ef4 6279 [main] INFO backtype.storm.daemon.nimbus - Starting Nimbus with conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 0, "topology.skip.missing.kryo.registrations" true, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 50, "storm.messaging.netty.flush.check.interval.ms" 10, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/lib/jvm/java-7-openjdk-amd64", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "/home/fali/tmp/4ffaa109-e973-4379-baed-bf8ff5c7af27", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "storm.meta.serialization.delegate" "backtype.storm.serialization.DefaultSerializationDelegate", "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "localhost", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2001, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["localhost"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 300, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" [6700 6701 6702 6703], "topology.environment" nil, "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer" "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx512m", "storm.cluster.mode" "local", "topology.max.task.parallelism" nil, "storm.messaging.netty.transfer.batch.size" 262144, "topology.classpath" nil} 6283 [main] INFO backtype.storm.daemon.nimbus - Using default scheduler 6334 [main] INFO backtype.storm.utils.StormBoundedExponentialBackoffRetry - The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5] 6606 [main] INFO org.apache.storm.curator.framework.imps.CuratorFrameworkImpl - Starting 6609 [main] INFO org.apache.storm.zookeeper.ZooKeeper - Initiating client connection, connectString=localhost:2001 sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@7a85b031 6685 [main-SendThread(localhost:2001)] INFO org.apache.storm.zookeeper.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2001. Will not attempt to authenticate using SASL (unknown error) 6687 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2001] INFO org.apache.storm.zookeeper.server.NIOServerCnxnFactory - Accepted socket connection from /127.0.0.1:47064 6689 [main-SendThread(localhost:2001)] INFO org.apache.storm.zookeeper.ClientCnxn - Socket connection established to localhost/127.0.0.1:2001, initiating session 6724 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2001] INFO org.apache.storm.zookeeper.server.ZooKeeperServer - Client attempting to establish new session at /127.0.0.1:47064 6729 [SyncThread:0] INFO org.apache.storm.zookeeper.server.persistence.FileTxnLog - Creating new log file: log.1 6774 [main-SendThread(localhost:2001)] INFO org.apache.storm.zookeeper.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2001, sessionid = 0x14f1d371f060000, negotiated timeout = 20000 6776 [main-EventThread] INFO org.apache.storm.curator.framework.state.ConnectionStateManager - State change: CONNECTED 6778 [main-EventThread] INFO backtype.storm.zookeeper - Zookeeper state update: :connected:none 6783 [SyncThread:0] INFO org.apache.storm.zookeeper.server.ZooKeeperServer - Established session 0x14f1d371f060000 with negotiated timeout 20000 for client /127.0.0.1:47064 7856 [ProcessThread(sid:0 cport:-1):] INFO org.apache.storm.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x14f1d371f060000 7858 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2001] INFO org.apache.storm.zookeeper.server.NIOServerCnxn - Closed socket connection for client /127.0.0.1:47064 which had sessionid 0x14f1d371f060000 7859 [main-EventThread] INFO org.apache.storm.zookeeper.ClientCnxn - EventThread shut down 7859 [main] INFO org.apache.storm.zookeeper.ZooKeeper - Session: 0x14f1d371f060000 closed 7860 [main] INFO backtype.storm.utils.StormBoundedExponentialBackoffRetry - The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5] 7861 [main] INFO org.apache.storm.curator.framework.imps.CuratorFrameworkImpl - Starting 7861 [main] INFO org.apache.storm.zookeeper.ZooKeeper - Initiating client connection, connectString=localhost:2001/storm sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@4c2c4c62 7866 [main-SendThread(localhost:2001)] INFO org.apache.storm.zookeeper.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2001. Will not attempt to authenticate using SASL (unknown error) 7869 [main-SendThread(localhost:2001)] INFO org.apache.storm.zookeeper.ClientCnxn - Socket connection established to localhost/127.0.0.1:2001, initiating session 7869 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2001] INFO org.apache.storm.zookeeper.server.NIOServerCnxnFactory - Accepted socket connection from /127.0.0.1:47065 7869 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2001] INFO org.apache.storm.zookeeper.server.ZooKeeperServer - Client attempting to establish new session at /127.0.0.1:47065 7878 [SyncThread:0] INFO org.apache.storm.zookeeper.server.ZooKeeperServer - Established session 0x14f1d371f060001 with negotiated timeout 20000 for client /127.0.0.1:47065 7878 [main-SendThread(localhost:2001)] INFO org.apache.storm.zookeeper.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2001, sessionid = 0x14f1d371f060001, negotiated timeout = 20000 7878 [main-EventThread] INFO org.apache.storm.curator.framework.state.ConnectionStateManager - State change: CONNECTED 7939 [main] INFO backtype.storm.utils.StormBoundedExponentialBackoffRetry - The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5] 7940 [main] INFO org.apache.storm.curator.framework.imps.CuratorFrameworkImpl - Starting 7940 [main] INFO org.apache.storm.zookeeper.ZooKeeper - Initiating client connection, connectString=localhost:2001 sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@688b6a2c 7941 [main-SendThread(localhost:2001)] INFO org.apache.storm.zookeeper.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2001. Will not attempt to authenticate using SASL (unknown error) 7941 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2001] INFO org.apache.storm.zookeeper.server.NIOServerCnxnFactory - Accepted socket connection from /127.0.0.1:47066 7942 [main-SendThread(localhost:2001)] INFO org.apache.storm.zookeeper.ClientCnxn - Socket connection established to localhost/127.0.0.1:2001, initiating session 7942 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2001] INFO org.apache.storm.zookeeper.server.ZooKeeperServer - Client attempting to establish new session at /127.0.0.1:47066 7945 [SyncThread:0] INFO org.apache.storm.zookeeper.server.ZooKeeperServer - Established session 0x14f1d371f060002 with negotiated timeout 20000 for client /127.0.0.1:47066 7946 [main-SendThread(localhost:2001)] INFO org.apache.storm.zookeeper.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2001, sessionid = 0x14f1d371f060002, negotiated timeout = 20000 7946 [main-EventThread] INFO org.apache.storm.curator.framework.state.ConnectionStateManager - State change: CONNECTED 7946 [main-EventThread] INFO backtype.storm.zookeeper - Zookeeper state update: :connected:none 8949 [ProcessThread(sid:0 cport:-1):] INFO org.apache.storm.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x14f1d371f060002 8950 [main] INFO org.apache.storm.zookeeper.ZooKeeper - Session: 0x14f1d371f060002 closed 8950 [main] INFO backtype.storm.utils.StormBoundedExponentialBackoffRetry - The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5] 8951 [main] INFO org.apache.storm.curator.framework.imps.CuratorFrameworkImpl - Starting 8950 [main-EventThread] INFO org.apache.storm.zookeeper.ClientCnxn - EventThread shut down 8954 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2001] WARN org.apache.storm.zookeeper.server.NIOServerCnxn - caught end of stream exception org.apache.storm.zookeeper.server.ServerCnxn$EndOfStreamException: Unable to read additional data from client sessionid 0x14f1d371f060002, likely client has closed socket at org.apache.storm.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228) ~[storm-core-0.9.5.jar:0.9.5] at org.apache.storm.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) [storm-core-0.9.5.jar:0.9.5] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] 8955 [main] INFO org.apache.storm.zookeeper.ZooKeeper - Initiating client connection, connectString=localhost:2001/storm sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@135e657f 8955 [main-SendThread(localhost:2001)] INFO org.apache.storm.zookeeper.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2001. Will not attempt to authenticate using SASL (unknown error) 8956 [main-SendThread(localhost:2001)] INFO org.apache.storm.zookeeper.ClientCnxn - Socket connection established to localhost/127.0.0.1:2001, initiating session 8957 [main] INFO backtype.storm.utils.StormBoundedExponentialBackoffRetry - The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5] 8959 [main] INFO org.apache.storm.curator.framework.imps.CuratorFrameworkImpl - Starting 8959 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2001] INFO org.apache.storm.zookeeper.server.NIOServerCnxn - Closed socket connection for client /127.0.0.1:47066 which had sessionid 0x14f1d371f060002 8960 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2001] INFO org.apache.storm.zookeeper.server.NIOServerCnxnFactory - Accepted socket connection from /127.0.0.1:47067 8960 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2001] INFO org.apache.storm.zookeeper.server.ZooKeeperServer - Client attempting to establish new session at /127.0.0.1:47067 8963 [main] INFO org.apache.storm.zookeeper.ZooKeeper - Initiating client connection, connectString=localhost:2001 sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@6fc0394c 8964 [main-SendThread(localhost:2001)] INFO org.apache.storm.zookeeper.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2001. Will not attempt to authenticate using SASL (unknown error) 8969 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2001] INFO org.apache.storm.zookeeper.server.NIOServerCnxnFactory - Accepted socket connection from /127.0.0.1:47068 8971 [main-SendThread(localhost:2001)] INFO org.apache.storm.zookeeper.ClientCnxn - Socket connection established to localhost/127.0.0.1:2001, initiating session 8974 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2001] INFO org.apache.storm.zookeeper.server.ZooKeeperServer - Client attempting to establish new session at /127.0.0.1:47068 8983 [SyncThread:0] INFO org.apache.storm.zookeeper.server.ZooKeeperServer - Established session 0x14f1d371f060003 with negotiated timeout 20000 for client /127.0.0.1:47067 8985 [main-SendThread(localhost:2001)] INFO org.apache.storm.zookeeper.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2001, sessionid = 0x14f1d371f060003, negotiated timeout = 20000 8986 [main-EventThread] INFO org.apache.storm.curator.framework.state.ConnectionStateManager - State change: CONNECTED 8989 [SyncThread:0] INFO org.apache.storm.zookeeper.server.ZooKeeperServer - Established session 0x14f1d371f060004 with negotiated timeout 20000 for client /127.0.0.1:47068 8990 [main-SendThread(localhost:2001)] INFO org.apache.storm.zookeeper.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2001, sessionid = 0x14f1d371f060004, negotiated timeout = 20000 8993 [main-EventThread] INFO org.apache.storm.curator.framework.state.ConnectionStateManager - State change: CONNECTED 8994 [main-EventThread] INFO backtype.storm.zookeeper - Zookeeper state update: :connected:none 9996 [ProcessThread(sid:0 cport:-1):] INFO org.apache.storm.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x14f1d371f060004 9999 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2001] INFO org.apache.storm.zookeeper.server.NIOServerCnxn - Closed socket connection for client /127.0.0.1:47068 which had sessionid 0x14f1d371f060004 10000 [main] INFO org.apache.storm.zookeeper.ZooKeeper - Session: 0x14f1d371f060004 closed 10000 [main] INFO backtype.storm.utils.StormBoundedExponentialBackoffRetry - The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5] 10001 [main] INFO org.apache.storm.curator.framework.imps.CuratorFrameworkImpl - Starting 10001 [main-EventThread] INFO org.apache.storm.zookeeper.ClientCnxn - EventThread shut down 10001 [main] INFO org.apache.storm.zookeeper.ZooKeeper - Initiating client connection, connectString=localhost:2001/storm sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@1247df10 10002 [main-SendThread(localhost:2001)] INFO org.apache.storm.zookeeper.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2001. Will not attempt to authenticate using SASL (unknown error) 10002 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2001] INFO org.apache.storm.zookeeper.server.NIOServerCnxnFactory - Accepted socket connection from /127.0.0.1:47069 10003 [main-SendThread(localhost:2001)] INFO org.apache.storm.zookeeper.ClientCnxn - Socket connection established to localhost/127.0.0.1:2001, initiating session 10003 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2001] INFO org.apache.storm.zookeeper.server.ZooKeeperServer - Client attempting to establish new session at /127.0.0.1:47069 10017 [SyncThread:0] INFO org.apache.storm.zookeeper.server.ZooKeeperServer - Established session 0x14f1d371f060005 with negotiated timeout 20000 for client /127.0.0.1:47069 10017 [main-SendThread(localhost:2001)] INFO org.apache.storm.zookeeper.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2001, sessionid = 0x14f1d371f060005, negotiated timeout = 20000 10019 [main-EventThread] INFO org.apache.storm.curator.framework.state.ConnectionStateManager - State change: CONNECTED 11050 [main] INFO backtype.storm.daemon.supervisor - Starting Supervisor with conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 0, "topology.skip.missing.kryo.registrations" true, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 50, "storm.messaging.netty.flush.check.interval.ms" 10, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/lib/jvm/java-7-openjdk-amd64", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "/home/fali/tmp/468c635e-1af2-4be6-892d-354d85142152", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "storm.meta.serialization.delegate" "backtype.storm.serialization.DefaultSerializationDelegate", "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "localhost", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2001, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["localhost"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 300, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" (1024 1025 1026), "topology.environment" nil, "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer" "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx512m", "storm.cluster.mode" "local", "topology.max.task.parallelism" nil, "storm.messaging.netty.transfer.batch.size" 262144, "topology.classpath" nil} 11061 [main] INFO backtype.storm.utils.StormBoundedExponentialBackoffRetry - The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5] 11062 [main] INFO org.apache.storm.curator.framework.imps.CuratorFrameworkImpl - Starting 11062 [main] INFO org.apache.storm.zookeeper.ZooKeeper - Initiating client connection, connectString=localhost:2001 sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@3c22d5b8 11063 [main-SendThread(localhost:2001)] INFO org.apache.storm.zookeeper.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2001. Will not attempt to authenticate using SASL (unknown error) 11063 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2001] INFO org.apache.storm.zookeeper.server.NIOServerCnxnFactory - Accepted socket connection from /127.0.0.1:47070 11063 [main-SendThread(localhost:2001)] INFO org.apache.storm.zookeeper.ClientCnxn - Socket connection established to localhost/127.0.0.1:2001, initiating session 11063 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2001] INFO org.apache.storm.zookeeper.server.ZooKeeperServer - Client attempting to establish new session at /127.0.0.1:47070 11069 [main-SendThread(localhost:2001)] INFO org.apache.storm.zookeeper.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2001, sessionid = 0x14f1d371f060006, negotiated timeout = 20000 11070 [SyncThread:0] INFO org.apache.storm.zookeeper.server.ZooKeeperServer - Established session 0x14f1d371f060006 with negotiated timeout 20000 for client /127.0.0.1:47070 11070 [main-EventThread] INFO org.apache.storm.curator.framework.state.ConnectionStateManager - State change: CONNECTED 11070 [main-EventThread] INFO backtype.storm.zookeeper - Zookeeper state update: :connected:none 12077 [ProcessThread(sid:0 cport:-1):] INFO org.apache.storm.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x14f1d371f060006 12078 [main] INFO org.apache.storm.zookeeper.ZooKeeper - Session: 0x14f1d371f060006 closed 12078 [main] INFO backtype.storm.utils.StormBoundedExponentialBackoffRetry - The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5] 12078 [main] INFO org.apache.storm.curator.framework.imps.CuratorFrameworkImpl - Starting 12079 [main-EventThread] INFO org.apache.storm.zookeeper.ClientCnxn - EventThread shut down 12079 [main] INFO org.apache.storm.zookeeper.ZooKeeper - Initiating client connection, connectString=localhost:2001/storm sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@7a749ed3 12080 [main-SendThread(localhost:2001)] INFO org.apache.storm.zookeeper.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2001. Will not attempt to authenticate using SASL (unknown error) 12080 [main-SendThread(localhost:2001)] INFO org.apache.storm.zookeeper.ClientCnxn - Socket connection established to localhost/127.0.0.1:2001, initiating session 12083 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2001] INFO org.apache.storm.zookeeper.server.NIOServerCnxn - Closed socket connection for client /127.0.0.1:47070 which had sessionid 0x14f1d371f060006 12083 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2001] INFO org.apache.storm.zookeeper.server.NIOServerCnxnFactory - Accepted socket connection from /127.0.0.1:47071 12084 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2001] INFO org.apache.storm.zookeeper.server.ZooKeeperServer - Client attempting to establish new session at /127.0.0.1:47071 12085 [SyncThread:0] INFO org.apache.storm.zookeeper.server.ZooKeeperServer - Established session 0x14f1d371f060007 with negotiated timeout 20000 for client /127.0.0.1:47071 12085 [main-SendThread(localhost:2001)] INFO org.apache.storm.zookeeper.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2001, sessionid = 0x14f1d371f060007, negotiated timeout = 20000 12086 [main-EventThread] INFO org.apache.storm.curator.framework.state.ConnectionStateManager - State change: CONNECTED 13123 [main] INFO backtype.storm.daemon.supervisor - Starting supervisor with id 5bc0f0b5-f134-4c6e-bfcf-1ac0a2825880 at host ubuntu 13134 [main] INFO backtype.storm.daemon.supervisor - Starting Supervisor with conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 0, "topology.skip.missing.kryo.registrations" true, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 50, "storm.messaging.netty.flush.check.interval.ms" 10, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/lib/jvm/java-7-openjdk-amd64", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "/home/fali/tmp/b2b204fd-f697-487c-9451-56c5848c2af8", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "storm.meta.serialization.delegate" "backtype.storm.serialization.DefaultSerializationDelegate", "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "localhost", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2001, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["localhost"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 300, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" (1027 1028 1029), "topology.environment" nil, "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer" "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx512m", "storm.cluster.mode" "local", "topology.max.task.parallelism" nil, "storm.messaging.netty.transfer.batch.size" 262144, "topology.classpath" nil} 13152 [main] INFO backtype.storm.utils.StormBoundedExponentialBackoffRetry - The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5] 13152 [main] INFO org.apache.storm.curator.framework.imps.CuratorFrameworkImpl - Starting 13154 [main] INFO org.apache.storm.zookeeper.ZooKeeper - Initiating client connection, connectString=localhost:2001 sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@62982862 13157 [main-SendThread(localhost:2001)] INFO org.apache.storm.zookeeper.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2001. Will not attempt to authenticate using SASL (unknown error) 13157 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2001] INFO org.apache.storm.zookeeper.server.NIOServerCnxnFactory - Accepted socket connection from /127.0.0.1:47072 13157 [main-SendThread(localhost:2001)] INFO org.apache.storm.zookeeper.ClientCnxn - Socket connection established to localhost/127.0.0.1:2001, initiating session 13158 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2001] INFO org.apache.storm.zookeeper.server.ZooKeeperServer - Client attempting to establish new session at /127.0.0.1:47072 13171 [SyncThread:0] INFO org.apache.storm.zookeeper.server.ZooKeeperServer - Established session 0x14f1d371f060008 with negotiated timeout 20000 for client /127.0.0.1:47072 13171 [main-SendThread(localhost:2001)] INFO org.apache.storm.zookeeper.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2001, sessionid = 0x14f1d371f060008, negotiated timeout = 20000 13172 [main-EventThread] INFO org.apache.storm.curator.framework.state.ConnectionStateManager - State change: CONNECTED 13173 [main-EventThread] INFO backtype.storm.zookeeper - Zookeeper state update: :connected:none 14176 [ProcessThread(sid:0 cport:-1):] INFO org.apache.storm.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x14f1d371f060008 14177 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2001] INFO org.apache.storm.zookeeper.server.NIOServerCnxn - Closed socket connection for client /127.0.0.1:47072 which had sessionid 0x14f1d371f060008 14178 [main-EventThread] INFO org.apache.storm.zookeeper.ClientCnxn - EventThread shut down 14177 [main] INFO org.apache.storm.zookeeper.ZooKeeper - Session: 0x14f1d371f060008 closed 14181 [main] INFO backtype.storm.utils.StormBoundedExponentialBackoffRetry - The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5] 14184 [main] INFO org.apache.storm.curator.framework.imps.CuratorFrameworkImpl - Starting 14185 [main] INFO org.apache.storm.zookeeper.ZooKeeper - Initiating client connection, connectString=localhost:2001/storm sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@5d785311 14186 [main-SendThread(localhost:2001)] INFO org.apache.storm.zookeeper.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2001. Will not attempt to authenticate using SASL (unknown error) 14189 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2001] INFO org.apache.storm.zookeeper.server.NIOServerCnxnFactory - Accepted socket connection from /127.0.0.1:47073 14189 [main-SendThread(localhost:2001)] INFO org.apache.storm.zookeeper.ClientCnxn - Socket connection established to localhost/127.0.0.1:2001, initiating session 14190 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2001] INFO org.apache.storm.zookeeper.server.ZooKeeperServer - Client attempting to establish new session at /127.0.0.1:47073 14191 [SyncThread:0] INFO org.apache.storm.zookeeper.server.ZooKeeperServer - Established session 0x14f1d371f060009 with negotiated timeout 20000 for client /127.0.0.1:47073 14192 [main-SendThread(localhost:2001)] INFO org.apache.storm.zookeeper.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2001, sessionid = 0x14f1d371f060009, negotiated timeout = 20000 14192 [main-EventThread] INFO org.apache.storm.curator.framework.state.ConnectionStateManager - State change: CONNECTED 15201 [main] INFO backtype.storm.daemon.supervisor - Starting supervisor with id 0e188454-0a8d-439c-9394-a5e9e89fae4b at host ubuntu 15292 [main] INFO backtype.storm.daemon.nimbus - Received topology submission for redistomongotopology with conf {"topology.acker.executors" nil, "topology.kryo.register" nil, "topology.kryo.decorators" (), "topology.name" "redistomongotopology", "storm.id" "redistomongotopology-1-1439304205", "topology.max.task.parallelism" 1, "topology.multilang.serializer" "com.yelp.pyleus.serializer.MessagePackSerializer", "topology.debug" true} 15331 [main] INFO backtype.storm.daemon.nimbus - Activating redistomongotopology: redistomongotopology-1-1439304205 15452 [main] INFO backtype.storm.scheduler.EvenScheduler - Available slots: (["0e188454-0a8d-439c-9394-a5e9e89fae4b" 1027] ["0e188454-0a8d-439c-9394-a5e9e89fae4b" 1028] ["0e188454-0a8d-439c-9394-a5e9e89fae4b" 1029] ["5bc0f0b5-f134-4c6e-bfcf-1ac0a2825880" 1024] ["5bc0f0b5-f134-4c6e-bfcf-1ac0a2825880" 1025] ["5bc0f0b5-f134-4c6e-bfcf-1ac0a2825880" 1026]) 15496 [main] INFO backtype.storm.daemon.nimbus - Setting new assignment for topology id redistomongotopology-1-1439304205: #backtype.storm.daemon.common.Assignment{:master-code-dir "/home/fali/tmp/4ffaa109-e973-4379-baed-bf8ff5c7af27/nimbus/stormdist/redistomongotopology-1-1439304205", :node->host {"0e188454-0a8d-439c-9394-a5e9e89fae4b" "ubuntu"}, :executor->node+port {[3 3] ["0e188454-0a8d-439c-9394-a5e9e89fae4b" 1027], [2 2] ["0e188454-0a8d-439c-9394-a5e9e89fae4b" 1027], [1 1] ["0e188454-0a8d-439c-9394-a5e9e89fae4b" 1027]}, :executor->start-time-secs {[3 3] 1439304205, [2 2] 1439304205, [1 1] 1439304205}} 16606 [Thread-5] INFO backtype.storm.daemon.supervisor - Extracting resources from jar at redistomongotopology.jar to /home/fali/tmp/b2b204fd-f697-487c-9451-56c5848c2af8/supervisor/stormdist/redistomongotopology-1-1439304205/resources 17467 [Thread-6] INFO backtype.storm.daemon.supervisor - Launching worker with assignment #backtype.storm.daemon.supervisor.LocalAssignment{:storm-id "redistomongotopology-1-1439304205", :executors ([3 3] [2 2] [1 1])} for this supervisor 0e188454-0a8d-439c-9394-a5e9e89fae4b on port 1027 with id 1ae836fa-efa6-4f73-affb-caa000a55c50 17473 [Thread-6] INFO backtype.storm.daemon.worker - Launching worker for redistomongotopology-1-1439304205 on 0e188454-0a8d-439c-9394-a5e9e89fae4b:1027 with id 1ae836fa-efa6-4f73-affb-caa000a55c50 and conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 0, "topology.skip.missing.kryo.registrations" true, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 50, "storm.messaging.netty.flush.check.interval.ms" 10, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/lib/jvm/java-7-openjdk-amd64", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "/home/fali/tmp/b2b204fd-f697-487c-9451-56c5848c2af8", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "storm.meta.serialization.delegate" "backtype.storm.serialization.DefaultSerializationDelegate", "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "localhost", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2001, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["localhost"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 300, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" (1027 1028 1029), "topology.environment" nil, "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer" "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx512m", "storm.cluster.mode" "local", "topology.max.task.parallelism" nil, "storm.messaging.netty.transfer.batch.size" 262144, "topology.classpath" nil} 17474 [Thread-6] INFO backtype.storm.utils.StormBoundedExponentialBackoffRetry - The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5] 17477 [Thread-6] INFO org.apache.storm.curator.framework.imps.CuratorFrameworkImpl - Starting 17478 [Thread-6] INFO org.apache.storm.zookeeper.ZooKeeper - Initiating client connection, connectString=localhost:2001 sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@cb19f76 17478 [Thread-6-SendThread(localhost:2001)] INFO org.apache.storm.zookeeper.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2001. Will not attempt to authenticate using SASL (unknown error) 17479 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2001] INFO org.apache.storm.zookeeper.server.NIOServerCnxnFactory - Accepted socket connection from /127.0.0.1:47074 17479 [Thread-6-SendThread(localhost:2001)] INFO org.apache.storm.zookeeper.ClientCnxn - Socket connection established to localhost/127.0.0.1:2001, initiating session 17479 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2001] INFO org.apache.storm.zookeeper.server.ZooKeeperServer - Client attempting to establish new session at /127.0.0.1:47074 17486 [SyncThread:0] INFO org.apache.storm.zookeeper.server.ZooKeeperServer - Established session 0x14f1d371f06000a with negotiated timeout 20000 for client /127.0.0.1:47074 17486 [Thread-6-SendThread(localhost:2001)] INFO org.apache.storm.zookeeper.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2001, sessionid = 0x14f1d371f06000a, negotiated timeout = 20000 17491 [Thread-6-EventThread] INFO org.apache.storm.curator.framework.state.ConnectionStateManager - State change: CONNECTED 17491 [Thread-6-EventThread] INFO backtype.storm.zookeeper - Zookeeper state update: :connected:none 18493 [ProcessThread(sid:0 cport:-1):] INFO org.apache.storm.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x14f1d371f06000a 18494 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2001] INFO org.apache.storm.zookeeper.server.NIOServerCnxn - Closed socket connection for client /127.0.0.1:47074 which had sessionid 0x14f1d371f06000a 18494 [Thread-6-EventThread] INFO org.apache.storm.zookeeper.ClientCnxn - EventThread shut down 18494 [Thread-6] INFO org.apache.storm.zookeeper.ZooKeeper - Session: 0x14f1d371f06000a closed 18495 [Thread-6] INFO backtype.storm.utils.StormBoundedExponentialBackoffRetry - The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5] 18495 [Thread-6] INFO org.apache.storm.curator.framework.imps.CuratorFrameworkImpl - Starting 18495 [Thread-6] INFO org.apache.storm.zookeeper.ZooKeeper - Initiating client connection, connectString=localhost:2001/storm sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@31c6bc2f 18496 [Thread-6-SendThread(localhost:2001)] INFO org.apache.storm.zookeeper.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2001. Will not attempt to authenticate using SASL (unknown error) 18497 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2001] INFO org.apache.storm.zookeeper.server.NIOServerCnxnFactory - Accepted socket connection from /127.0.0.1:47075 18497 [Thread-6-SendThread(localhost:2001)] INFO org.apache.storm.zookeeper.ClientCnxn - Socket connection established to localhost/127.0.0.1:2001, initiating session 18497 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2001] INFO org.apache.storm.zookeeper.server.ZooKeeperServer - Client attempting to establish new session at /127.0.0.1:47075 18526 [SyncThread:0] INFO org.apache.storm.zookeeper.server.ZooKeeperServer - Established session 0x14f1d371f06000b with negotiated timeout 20000 for client /127.0.0.1:47075 18526 [Thread-6-SendThread(localhost:2001)] INFO org.apache.storm.zookeeper.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2001, sessionid = 0x14f1d371f06000b, negotiated timeout = 20000 18527 [Thread-6-EventThread] INFO org.apache.storm.curator.framework.state.ConnectionStateManager - State change: CONNECTED 18535 [Thread-6] INFO backtype.storm.daemon.worker - Reading Assignments. 18639 [Thread-6] INFO backtype.storm.daemon.worker - Launching receive-thread for 0e188454-0a8d-439c-9394-a5e9e89fae4b:1027 18644 [Thread-8-worker-receiver-thread-0] INFO backtype.storm.messaging.loader - Starting receive-thread: [stormId: redistomongotopology-1-1439304205, port: 1027, thread-id: 0 ] 18976 [Thread-6] INFO backtype.storm.daemon.executor - Loading executor readredisdata-spout:[2 2] 18982 [Thread-6] INFO backtype.storm.daemon.task - Emitting: readredisdata-spout system ["startup"] 18982 [Thread-6] INFO backtype.storm.daemon.executor - Loaded executor tasks readredisdata-spout:[2 2] 18995 [Thread-6] INFO backtype.storm.daemon.executor - Finished loading executor readredisdata-spout:[2 2] 19019 [Thread-6] INFO backtype.storm.daemon.executor - Loading executor writetomongo_bolt:[3 3] 19028 [Thread-6] INFO backtype.storm.daemon.task - Emitting: writetomongo_bolt system ["startup"] 19028 [Thread-6] INFO backtype.storm.daemon.executor - Loaded executor tasks writetomongo_bolt:[3 3] 19043 [Thread-6] INFO backtype.storm.daemon.executor - Finished loading executor writetomongo_bolt:[3 3] 19059 [Thread-6] INFO backtype.storm.daemon.executor - Loading executor system:[-1 -1] 19059 [Thread-6] INFO backtype.storm.daemon.task - Emitting: system system ["startup"] 19060 [Thread-6] INFO backtype.storm.daemon.executor - Loaded executor tasks system:[-1 -1] 19061 [Thread-6] INFO backtype.storm.daemon.executor - Finished loading executor system:[-1 -1] 19066 [Thread-6] INFO backtype.storm.daemon.executor - Loading executor acker:[1 1] 19067 [Thread-6] INFO backtype.storm.daemon.task - Emitting: acker system ["startup"] 19067 [Thread-6] INFO backtype.storm.daemon.executor - Loaded executor tasks acker:[1 1] 19069 [Thread-6] INFO backtype.storm.daemon.executor - Timeouts disabled for executor acker:[1 1] 19069 [Thread-6] INFO backtype.storm.daemon.executor - Finished loading executor acker:[1 1] 19090 [Thread-6] INFO backtype.storm.daemon.worker - Worker has topology config {"storm.id" "redistomongotopology-1-1439304205", "dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 0, "topology.skip.missing.kryo.registrations" true, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 50, "storm.messaging.netty.flush.check.interval.ms" 10, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/lib/jvm/java-7-openjdk-amd64", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "/home/fali/tmp/b2b204fd-f697-487c-9451-56c5848c2af8", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "storm.meta.serialization.delegate" "backtype.storm.serialization.DefaultSerializationDelegate", "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "localhost", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2001, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["localhost"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.kryo.decorators" (), "topology.name" "redistomongotopology", "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 300, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" (1027 1028 1029), "topology.environment" nil, "topology.debug" true, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.kryo.register" nil, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer" "com.yelp.pyleus.serializer.MessagePackSerializer", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx512m", "storm.cluster.mode" "local", "topology.max.task.parallelism" 1, "storm.messaging.netty.transfer.batch.size" 262144, "topology.classpath" nil} 19097 [Thread-6] INFO backtype.storm.daemon.worker - Worker 1ae836fa-efa6-4f73-affb-caa000a55c50 for storm redistomongotopology-1-1439304205 on 0e188454-0a8d-439c-9394-a5e9e89fae4b:1027 has finished loading 19565 [refresh-active-timer] INFO backtype.storm.daemon.worker - All connections are ready for worker 0e188454-0a8d-439c-9394-a5e9e89fae4b:1027 with id 1ae836fa-efa6-4f73-affb-caa000a55c50 19590 [Thread-16-acker] INFO backtype.storm.daemon.executor - Preparing bolt acker:(1) 19598 [Thread-16-acker] INFO backtype.storm.daemon.executor - Prepared bolt __acker:(1) 19603 [Thread-10-readredisdata-spout] INFO backtype.storm.daemon.executor - Opening spout readredisdata-spout:(2) 19643 [Thread-10-readredisdata-spout] INFO backtype.storm.utils.ShellProcess - Storm multilang serializer: com.yelp.pyleus.serializer.MessagePackSerializer 19648 [Thread-12-writetomongo_bolt] INFO backtype.storm.daemon.executor - Preparing bolt writetomongo_bolt:(3) 19649 [Thread-12-writetomongo_bolt] INFO backtype.storm.utils.ShellProcess - Storm multilang serializer: com.yelp.pyleus.serializer.MessagePackSerializer 19671 [Thread-14-system] INFO backtype.storm.daemon.executor - Preparing bolt system:(-1) 19674 [Thread-14-system] INFO backtype.storm.daemon.executor - Prepared bolt system:(-1) 20778 [Thread-10-readredisdata-spout] INFO backtype.storm.spout.ShellSpout - Launched subprocess with pid 7655 20791 [Thread-10-readredisdata-spout] INFO backtype.storm.daemon.executor - Opened spout readredisdata-spout:(2) 20798 [Thread-10-readredisdata-spout] INFO backtype.storm.daemon.executor - Activating spout readredisdata-spout:(2) 20799 [Thread-10-readredisdata-spout] INFO backtype.storm.spout.ShellSpout - Start checking heartbeat... 20966 [Thread-12-writetomongo_bolt] INFO backtype.storm.task.ShellBolt - Launched subprocess with pid 7654 20969 [Thread-12-writetomongo_bolt] INFO backtype.storm.task.ShellBolt - Start checking heartbeat... 20971 [Thread-12-writetomongo_bolt] INFO backtype.storm.daemon.executor - Prepared bolt writetomongo_bolt:(3) 50802 [pool-26-thread-1] ERROR backtype.storm.spout.ShellSpout - Halting process: ShellSpout died. java.lang.RuntimeException: subprocess heartbeat timeout at backtype.storm.spout.ShellSpout$SpoutHeartbeatTimerTask.run(ShellSpout.java:261) [storm-core-0.9.5.jar:0.9.5] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_79] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) [na:1.7.0_79] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_79] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] 50803 [pool-26-thread-1] ERROR backtype.storm.daemon.executor - java.lang.RuntimeException: subprocess heartbeat timeout at backtype.storm.spout.ShellSpout$SpoutHeartbeatTimerTask.run(ShellSpout.java:261) [storm-core-0.9.5.jar:0.9.5] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_79] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) [na:1.7.0_79] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_79] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] 59861 [Thread-18] ERROR backtype.storm.daemon.executor - java.lang.Exception: Shell Process Exception: Traceback (most recent call last): File "/home/fali/tmp/b2b204fd-f697-487c-9451-56c5848c2af8/supervisor/stormdist/redistomongotopology-1-1439304205/resources/pyleus_venv/lib/python2.7/site-packages/pyleus/storm/component.py", line 238, in run self.run_component() File "/home/fali/tmp/b2b204fd-f697-487c-9451-56c5848c2af8/supervisor/stormdist/redistomongotopology-1-1439304205/resources/pyleus_venv/lib/python2.7/site-packages/pyleus/storm/bolt.py", line 48, in run_component tup = self.read_tuple() File "/home/fali/tmp/b2b204fd-f697-487c-9451-56c5848c2af8/supervisor/stormdist/redistomongotopology-1-1439304205/resources/pyleus_venv/lib/python2.7/site-packages/pyleus/storm/component.py", line 294, in read_tuple cmd = self.read_command() File "/home/fali/tmp/b2b204fd-f697-487c-9451-56c5848c2af8/supervisor/stormdist/redistomongotopology-1-1439304205/resources/pyleus_venv/lib/python2.7/site-packages/pyleus/storm/component.py", line 269, in read_command msg = self._serializer.read_msg() File "/home/fali/tmp/b2b204fd-f697-487c-9451-56c5848c2af8/supervisor/stormdist/redistomongotopology-1-1439304205/resources/pyleus_venv/lib/python2.7/site-packages/pyleus/storm/serializers/msgpack_serializer.py", line 43, in read_msg return next(self._messages) File "/home/fali/tmp/b2b204fd-f697-487c-9451-56c5848c2af8/supervisor/stormdist/redistomongotopology-1-1439304205/resources/pyleus_venv/lib/python2.7/site-packages/pyleus/storm/serializers/msgpack_serializer.py", line 18, in _messages_generator line = os.read(input_stream.fileno(), 1024 \ 2) KeyboardInterrupt

at backtype.storm.task.ShellBolt.handleError(ShellBolt.java:188) [storm-core-0.9.5.jar:0.9.5]
at backtype.storm.task.ShellBolt.access$1100(ShellBolt.java:69) [storm-core-0.9.5.jar:0.9.5]
at backtype.storm.task.ShellBolt$BoltReaderRunnable.run(ShellBolt.java:331) [storm-core-0.9.5.jar:0.9.5]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]

59863 [Thread-10-readredisdata-spout] ERROR backtype.storm.daemon.executor - java.lang.Exception: Shell Process Exception: Traceback (most recent call last): File "/home/fali/tmp/b2b204fd-f697-487c-9451-56c5848c2af8/supervisor/stormdist/redistomongotopology-1-1439304205/resources/pyleus_venv/lib/python2.7/site-packages/pyleus/storm/component.py", line 237, in run self.setup_component() File "/home/fali/tmp/b2b204fd-f697-487c-9451-56c5848c2af8/supervisor/stormdist/redistomongotopology-1-1439304205/resources/pyleus_venv/lib/python2.7/site-packages/pyleus/storm/component.py", line 198, in setup_component self.initialize() File "/home/fali/tmp/b2b204fd-f697-487c-9451-56c5848c2af8/supervisor/stormdist/redistomongotopology-1-1439304205/resources/redistomongotopology/readredisdata_spout.py", line 28, in initialize for msg in pubsub.listen(): File "/home/fali/tmp/b2b204fd-f697-487c-9451-56c5848c2af8/supervisor/stormdist/redistomongotopology-1-1439304205/resources/pyleus_venv/lib/python2.7/site-packages/redis/client.py", line 2215, in listen response = self.handle_message(self.parse_response(block=True)) File "/home/fali/tmp/b2b204fd-f697-487c-9451-56c5848c2af8/supervisor/stormdist/redistomongotopology-1-1439304205/resources/pyleus_venv/lib/python2.7/site-packages/redis/client.py", line 2150, in parse_response return self._execute(connection, connection.read_response) File "/home/fali/tmp/b2b204fd-f697-487c-9451-56c5848c2af8/supervisor/stormdist/redistomongotopology-1-1439304205/resources/pyleus_venv/lib/python2.7/site-packages/redis/client.py", line 2132, in _execute return command(*args) File "/home/fali/tmp/b2b204fd-f697-487c-9451-56c5848c2af8/supervisor/stormdist/redistomongotopology-1-1439304205/resources/pyleus_venv/lib/python2.7/site-packages/redis/connection.py", line 569, in read_response response = self._parser.read_response() File "/home/fali/tmp/b2b204fd-f697-487c-9451-56c5848c2af8/supervisor/stormdist/redistomongotopology-1-1439304205/resources/pyleus_venv/lib/python2.7/site-packages/redis/connection.py", line 330, in read_response bufflen = self._sock.recv_into(self._buffer) KeyboardInterrupt

at backtype.storm.spout.ShellSpout.handleError(ShellSpout.java:212) [storm-core-0.9.5.jar:0.9.5]
at backtype.storm.spout.ShellSpout.querySubprocess(ShellSpout.java:158) [storm-core-0.9.5.jar:0.9.5]
at backtype.storm.spout.ShellSpout.nextTuple(ShellSpout.java:91) [storm-core-0.9.5.jar:0.9.5]
at backtype.storm.daemon.executor$fn__6579$fn__6594$fn__6623.invoke(executor.clj:565) [storm-core-0.9.5.jar:0.9.5]
at backtype.storm.util$async_loop$fn__459.invoke(util.clj:463) [storm-core-0.9.5.jar:0.9.5]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]

59892 [Thread-7] INFO backtype.storm.daemon.nimbus - Shutting down master 59905 [Thread-10-readredisdata-spout] ERROR backtype.storm.util - Async loop died! java.lang.RuntimeException: pid:7655, name:readredisdata-spout exitCode:0, errorString: at backtype.storm.spout.ShellSpout.querySubprocess(ShellSpout.java:180) ~[storm-core-0.9.5.jar:0.9.5] at backtype.storm.spout.ShellSpout.nextTuple(ShellSpout.java:91) ~[storm-core-0.9.5.jar:0.9.5] at backtype.storm.daemon.executor$fn6579$fn6594$fn6623.invoke(executor.clj:565) ~[storm-core-0.9.5.jar:0.9.5] at backtype.storm.util$async_loop$fn__459.invoke(util.clj:463) ~[storm-core-0.9.5.jar:0.9.5] at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] Caused by: java.io.EOFException: null at org.msgpack.io.StreamInput.readByte(StreamInput.java:60) ~[redistomongotopology.jar:na] at org.msgpack.unpacker.MessagePackUnpacker.getHeadByte(MessagePackUnpacker.java:66) ~[redistomongotopology.jar:na] at org.msgpack.unpacker.MessagePackUnpacker.trySkipNil(MessagePackUnpacker.java:396) ~[redistomongotopology.jar:na] at org.msgpack.template.MapTemplate.read(MapTemplate.java:59) ~[redistomongotopology.jar:na] at org.msgpack.template.MapTemplate.read(MapTemplate.java:27) ~[redistomongotopology.jar:na] at org.msgpack.template.AbstractTemplate.read(AbstractTemplate.java:31) ~[redistomongotopology.jar:na] at org.msgpack.MessagePack.read(MessagePack.java:527) ~[redistomongotopology.jar:na] at org.msgpack.MessagePack.read(MessagePack.java:496) ~[redistomongotopology.jar:na] at com.yelp.pyleus.serializer.MessagePackSerializer.readMessage(MessagePackSerializer.java:205) ~[redistomongotopology.jar:na] at com.yelp.pyleus.serializer.MessagePackSerializer.readShellMsg(MessagePackSerializer.java:74) ~[redistomongotopology.jar:na] at backtype.storm.utils.ShellProcess.readShellMsg(ShellProcess.java:99) ~[storm-core-0.9.5.jar:0.9.5] at backtype.storm.spout.ShellSpout.querySubprocess(ShellSpout.java:145) ~[storm-core-0.9.5.jar:0.9.5] ... 5 common frames omitted 59905 [Thread-10-readredisdata-spout] ERROR backtype.storm.daemon.executor - java.lang.RuntimeException: pid:7655, name:readredisdata-spout exitCode:0, errorString: at backtype.storm.spout.ShellSpout.querySubprocess(ShellSpout.java:180) ~[storm-core-0.9.5.jar:0.9.5] at backtype.storm.spout.ShellSpout.nextTuple(ShellSpout.java:91) ~[storm-core-0.9.5.jar:0.9.5] at backtype.storm.daemon.executor$fn6579$fn6594$fn6623.invoke(executor.clj:565) ~[storm-core-0.9.5.jar:0.9.5] at backtype.storm.util$async_loop$fn459.invoke(util.clj:463) ~[storm-core-0.9.5.jar:0.9.5] at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] Caused by: java.io.EOFException: null at org.msgpack.io.StreamInput.readByte(StreamInput.java:60) ~[redistomongotopology.jar:na] at org.msgpack.unpacker.MessagePackUnpacker.getHeadByte(MessagePackUnpacker.java:66) ~[redistomongotopology.jar:na] at org.msgpack.unpacker.MessagePackUnpacker.trySkipNil(MessagePackUnpacker.java:396) ~[redistomongotopology.jar:na] at org.msgpack.template.MapTemplate.read(MapTemplate.java:59) ~[redistomongotopology.jar:na] at org.msgpack.template.MapTemplate.read(MapTemplate.java:27) ~[redistomongotopology.jar:na] at org.msgpack.template.AbstractTemplate.read(AbstractTemplate.java:31) ~[redistomongotopology.jar:na] at org.msgpack.MessagePack.read(MessagePack.java:527) ~[redistomongotopology.jar:na] at org.msgpack.MessagePack.read(MessagePack.java:496) ~[redistomongotopology.jar:na] at com.yelp.pyleus.serializer.MessagePackSerializer.readMessage(MessagePackSerializer.java:205) ~[redistomongotopology.jar:na] at com.yelp.pyleus.serializer.MessagePackSerializer.readShellMsg(MessagePackSerializer.java:74) ~[redistomongotopology.jar:na] at backtype.storm.utils.ShellProcess.readShellMsg(ShellProcess.java:99) ~[storm-core-0.9.5.jar:0.9.5] at backtype.storm.spout.ShellSpout.querySubprocess(ShellSpout.java:145) ~[storm-core-0.9.5.jar:0.9.5] ... 5 common frames omitted 59909 [ProcessThread(sid:0 cport:-1):] INFO org.apache.storm.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x14f1d371f060001 59931 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2001] INFO org.apache.storm.zookeeper.server.NIOServerCnxn - Closed socket connection for client /127.0.0.1:47065 which had sessionid 0x14f1d371f060001 59935 [main-EventThread] INFO org.apache.storm.zookeeper.ClientCnxn - EventThread shut down 59936 [Thread-7] INFO org.apache.storm.zookeeper.ZooKeeper - Session: 0x14f1d371f060001 closed 59938 [Thread-7] INFO backtype.storm.daemon.nimbus - Shut down master 59940 [ProcessThread(sid:0 cport:-1):] INFO org.apache.storm.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x14f1d371f060003 59943 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2001] INFO org.apache.storm.zookeeper.server.NIOServerCnxn - Closed socket connection for client /127.0.0.1:47067 which had sessionid 0x14f1d371f060003 59944 [main-EventThread] INFO org.apache.storm.zookeeper.ClientCnxn - EventThread shut down 59948 [Thread-7] INFO org.apache.storm.zookeeper.ZooKeeper - Session: 0x14f1d371f060003 closed 59949 [ProcessThread(sid:0 cport:-1):] INFO org.apache.storm.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x14f1d371f060005 59950 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2001] INFO org.apache.storm.zookeeper.server.NIOServerCnxn - Closed socket connection for client /127.0.0.1:47069 which had sessionid 0x14f1d371f060005 59950 [main-EventThread] INFO org.apache.storm.zookeeper.ClientCnxn - EventThread shut down 59952 [Thread-7] INFO org.apache.storm.zookeeper.ZooKeeper - Session: 0x14f1d371f060005 closed 59957 [Thread-7] INFO backtype.storm.daemon.supervisor - Shutting down supervisor 5bc0f0b5-f134-4c6e-bfcf-1ac0a2825880 59964 [Thread-3] INFO backtype.storm.event - Event manager interrupted 59965 [Thread-4] INFO backtype.storm.event - Event manager interrupted 59966 [ProcessThread(sid:0 cport:-1):] INFO org.apache.storm.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x14f1d371f060007 59967 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2001] INFO org.apache.storm.zookeeper.server.NIOServerCnxn - Closed socket connection for client /127.0.0.1:47071 which had sessionid 0x14f1d371f060007 59967 [main-EventThread] INFO org.apache.storm.zookeeper.ClientCnxn - EventThread shut down 59968 [Thread-7] INFO org.apache.storm.zookeeper.ZooKeeper - Session: 0x14f1d371f060007 closed 59968 [Thread-7] INFO backtype.storm.daemon.supervisor - Shutting down 0e188454-0a8d-439c-9394-a5e9e89fae4b:1ae836fa-efa6-4f73-affb-caa000a55c50 59969 [Thread-7] INFO backtype.storm.process-simulator - Killing process e66e3f4a-437e-44c5-ae91-94fa589f1f45 59969 [Thread-7] INFO backtype.storm.daemon.worker - Shutting down worker redistomongotopology-1-1439304205 0e188454-0a8d-439c-9394-a5e9e89fae4b 1027 59970 [Thread-7] INFO backtype.storm.daemon.worker - Shutting down receive thread 59970 [Thread-7] INFO backtype.storm.messaging.loader - Shutting down receiving-thread: [redistomongotopology-1-1439304205, 1027] 59971 [Thread-8-worker-receiver-thread-0] INFO backtype.storm.messaging.loader - Receiving-thread:[redistomongotopology-1-1439304205, 1027] received shutdown notice 59971 [Thread-7] INFO backtype.storm.messaging.loader - Waiting for receiving-thread:[redistomongotopology-1-1439304205, 1027] to die 59972 [Thread-7] INFO backtype.storm.messaging.loader - Shutdown receiving-thread: [redistomongotopology-1-1439304205, 1027] 59973 [Thread-7] INFO backtype.storm.daemon.worker - Shut down receive thread 59973 [Thread-7] INFO backtype.storm.daemon.worker - Terminating messaging context 59973 [Thread-7] INFO backtype.storm.daemon.worker - Shutting down executors 59976 [Thread-7] INFO backtype.storm.daemon.executor - Shutting down executor readredisdata-spout:[2 2] 59972 [Thread-10-readredisdata-spout] ERROR backtype.storm.util - Halting process: ("Worker died") java.lang.RuntimeException: ("Worker died") at backtype.storm.util$exit_processBANG.doInvoke(util.clj:325) [storm-core-0.9.5.jar:0.9.5] at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.5.1.jar:na] at backtype.storm.daemon.worker$fn7024$fn7025.invoke(worker.clj:493) [storm-core-0.9.5.jar:0.9.5] at backtype.storm.daemon.executor$mk_executor_data$fn6480$fn__6481.invoke(executor.clj:240) [storm-core-0.9.5.jar:0.9.5] at backtype.storm.util$async_loop$fn__459.invoke(util.clj:473) [storm-core-0.9.5.jar:0.9.5] at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] 59977 [Thread-9-disruptor-executor[2 2]-send-queue] INFO backtype.storm.util - Async loop interrupted! 60014 [Thread-18] ERROR backtype.storm.task.ShellBolt - Halting process: ShellBolt died. java.io.EOFException: null at org.msgpack.io.StreamInput.readByte(StreamInput.java:60) ~[redistomongotopology.jar:na] at org.msgpack.unpacker.MessagePackUnpacker.getHeadByte(MessagePackUnpacker.java:66) ~[redistomongotopology.jar:na] at org.msgpack.unpacker.MessagePackUnpacker.trySkipNil(MessagePackUnpacker.java:396) ~[redistomongotopology.jar:na] at org.msgpack.template.MapTemplate.read(MapTemplate.java:59) ~[redistomongotopology.jar:na] at org.msgpack.template.MapTemplate.read(MapTemplate.java:27) ~[redistomongotopology.jar:na] at org.msgpack.template.AbstractTemplate.read(AbstractTemplate.java:31) ~[redistomongotopology.jar:na] at org.msgpack.MessagePack.read(MessagePack.java:527) ~[redistomongotopology.jar:na] at org.msgpack.MessagePack.read(MessagePack.java:496) ~[redistomongotopology.jar:na] at com.yelp.pyleus.serializer.MessagePackSerializer.readMessage(MessagePackSerializer.java:205) ~[redistomongotopology.jar:na] at com.yelp.pyleus.serializer.MessagePackSerializer.readShellMsg(MessagePackSerializer.java:74) ~[redistomongotopology.jar:na] at backtype.storm.utils.ShellProcess.readShellMsg(ShellProcess.java:99) ~[storm-core-0.9.5.jar:0.9.5] at backtype.storm.task.ShellBolt$BoltReaderRunnable.run(ShellBolt.java:318) ~[storm-core-0.9.5.jar:0.9.5] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] 60015 [Thread-18] ERROR backtype.storm.daemon.executor - java.io.EOFException: null at org.msgpack.io.StreamInput.readByte(StreamInput.java:60) ~[redistomongotopology.jar:na] at org.msgpack.unpacker.MessagePackUnpacker.getHeadByte(MessagePackUnpacker.java:66) ~[redistomongotopology.jar:na] at org.msgpack.unpacker.MessagePackUnpacker.trySkipNil(MessagePackUnpacker.java:396) ~[redistomongotopology.jar:na] at org.msgpack.template.MapTemplate.read(MapTemplate.java:59) ~[redistomongotopology.jar:na] at org.msgpack.template.MapTemplate.read(MapTemplate.java:27) ~[redistomongotopology.jar:na] at org.msgpack.template.AbstractTemplate.read(AbstractTemplate.java:31) ~[redistomongotopology.jar:na] at org.msgpack.MessagePack.read(MessagePack.java:527) ~[redistomongotopology.jar:na] at org.msgpack.MessagePack.read(MessagePack.java:496) ~[redistomongotopology.jar:na] at com.yelp.pyleus.serializer.MessagePackSerializer.readMessage(MessagePackSerializer.java:205) ~[redistomongotopology.jar:na] at com.yelp.pyleus.serializer.MessagePackSerializer.readShellMsg(MessagePackSerializer.java:74) ~[redistomongotopology.jar:na] at backtype.storm.utils.ShellProcess.readShellMsg(ShellProcess.java:99) ~[storm-core-0.9.5.jar:0.9.5] at backtype.storm.task.ShellBolt$BoltReaderRunnable.run(ShellBolt.java:318) ~[storm-core-0.9.5.jar:0.9.5] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]

Please help me, i dont know what i am doing wrong.

Thanks in advance.

msmakhlouf commented 9 years ago

which version of pyleus are you using?

anees042 commented 9 years ago

Pyleus version is 0.3.0

mzbyszynski commented 9 years ago

Try using storm 0.9.4, not 0.9.5

anees042 commented 9 years ago

I was using storm 0.9.4, the same issue, then someone recommended 0.9.5, I am using storm 0.9.5 currently, will try 0.9.4 again. can some one try this topology or review it if i am doing something wrong technically?

anees042 commented 9 years ago

After changing to storm 0.9.4, emit in spout is not getting called; self.emit((key,))

the pubsub.listen() in initialize is blocking the spout and it is not making it to next_tuple for msg in pubsub.listen():

the code is changed to use thread for listen() and is no more blocking.

this issue is closed, the above code is updated and is working. pyleus is working as expected. i will request to include the code in examples.

thanks to all of you for your support.