search

sorenmacbeth / marceline

A Clojure DSL for Storm/Trident
Eclipse Public License 1.0
177 stars 14 forks source link

Can't Run Standalone #15

Closed snunez1 closed 9 years ago

snunez1 commented 9 years ago

Does anyone have a working example of running Marceline? I went through the example provided in the README, however it appears that the spout is looking for a zookeeper connection, which of course it doesn't have because there's no zookeeper running.

I am familiar with Hadoop, and Clojure, but not Storm and to keep things simple I'd like to get Storm/Clojure running locally before deploying to a cluster. Can this be done with Marceline?

55187 [Thread-9-$spoutcoord-spout0-SendThread(localhost:2000)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_05] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712) ~[na:1.8.0_05] at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) ~[zookeeper-3.4.5.jar:3.4.5-1392090] at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068) ~[zookeeper-3.4.5.jar:3.4.5-1392090]

strongh commented 9 years ago

Yes :) The examples should work locally, marceline just uses Storm's "local mode" for this. This local mode includes an in-process ZK, so there is ZK running. It may log junk which does not affect the behavior of test topologies (e.g. polling while the in-process ZK starts up perhaps?).

This can be done for the test/example topologies with e.g.

lein run -m marceline.topology.wordcount-metrics

Is that approximately what you tried? If that command fails, please include the full output.

If you're interested in how this is implemented, check out

https://github.com/yieldbot/marceline/blob/master/test/marceline/topology/wordcount_metrics.clj#L93-L107
snunez1 commented 9 years ago

Thanks. I'll check out the code and give it a try.

What I did was put in all the code from the README.md, thinking that was a worked example. After some time nimbus shuts down with no output.

Is the README intended to be a workable example? If not, perhaps it should be marked as simply 'guidance' or something like that.

snunez1 commented 9 years ago

OK, I just tried 'lein run -m marceline.topology.wordcount-metrics’ on a fresh checkout, but it doesn’t run. Below is the full output.

HW10964:marceline $ lein run -m marceline.topology.wordcount-metrics
Compiling 1 source files to /usr/local/src/marceline/target/classes
warning: [options] bootstrap class path not set in conjunction with -source 1.6
Note: /usr/local/src/marceline/src/jvm/marceline/storm/trident/testing/MemoryMapStateInitable.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
1 warning
1191 [main] INFO  backtype.storm.zookeeper - Starting inprocess zookeeper at port 2000 and dir /var/folders/_w/shz6hqkn4cnft3phjqmt8nxr0000gp/T//2fb54cd4-9f5d-426a-b0c2-f8002c0d880f
1301 [main] INFO  backtype.storm.daemon.nimbus - Starting Nimbus with conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 0, "topology.skip.missing.kryo.registrations" true, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 50, "storm.messaging.netty.flush.check.interval.ms" 10, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "/var/folders/_w/shz6hqkn4cnft3phjqmt8nxr0000gp/T//46873e3f-e58b-4a95-81c8-c35fefe716f9", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "localhost", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2000, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["localhost"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 30, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" [6700 6701 6702 6703], "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer" "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx1024m", "storm.cluster.mode" "local", "topology.max.task.parallelism" nil, "storm.messaging.netty.transfer.batch.size" 262144}
1304 [main] INFO  backtype.storm.daemon.nimbus - Using default scheduler
1359 [main] INFO  org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
6468 [main-EventThread] INFO  org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
6469 [ConnectionStateManager-0] WARN  org.apache.curator.framework.state.ConnectionStateManager - There are no ConnectionStateListeners registered.
6469 [main-EventThread] INFO  backtype.storm.zookeeper - Zookeeper state update: :connected:none
7494 [main] INFO  org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
7497 [main-EventThread] INFO  org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
7497 [ConnectionStateManager-0] WARN  org.apache.curator.framework.state.ConnectionStateManager - There are no ConnectionStateListeners registered.
7514 [main] INFO  org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
7517 [main-EventThread] INFO  org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
7517 [ConnectionStateManager-0] WARN  org.apache.curator.framework.state.ConnectionStateManager - There are no ConnectionStateListeners registered.
7517 [main-EventThread] INFO  backtype.storm.zookeeper - Zookeeper state update: :connected:none
7520 [main] INFO  org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
7521 [main] INFO  org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
7522 [main-EventThread] INFO  org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
7522 [ConnectionStateManager-0] WARN  org.apache.curator.framework.state.ConnectionStateManager - There are no ConnectionStateListeners registered.
7522 [main-EventThread] INFO  org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
7522 [ConnectionStateManager-0] WARN  org.apache.curator.framework.state.ConnectionStateManager - There are no ConnectionStateListeners registered.
7522 [main-EventThread] INFO  backtype.storm.zookeeper - Zookeeper state update: :connected:none
7524 [main] INFO  org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
7526 [main-EventThread] INFO  org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
7526 [ConnectionStateManager-0] WARN  org.apache.curator.framework.state.ConnectionStateManager - There are no ConnectionStateListeners registered.
7532 [main] INFO  backtype.storm.daemon.supervisor - Starting Supervisor with conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 0, "topology.skip.missing.kryo.registrations" true, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 50, "storm.messaging.netty.flush.check.interval.ms" 10, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "/var/folders/_w/shz6hqkn4cnft3phjqmt8nxr0000gp/T//3ccad0bd-b27a-487b-a222-7a01190b4d3d", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "localhost", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2000, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["localhost"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 30, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" (1024 1025 1026), "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer" "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx1024m", "storm.cluster.mode" "local", "topology.max.task.parallelism" nil, "storm.messaging.netty.transfer.batch.size" 262144}
7540 [main] INFO  org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
7542 [main-EventThread] INFO  org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
7542 [ConnectionStateManager-0] WARN  org.apache.curator.framework.state.ConnectionStateManager - There are no ConnectionStateListeners registered.
7543 [main-EventThread] INFO  backtype.storm.zookeeper - Zookeeper state update: :connected:none
7546 [main] INFO  org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
7547 [main-EventThread] INFO  org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
7548 [ConnectionStateManager-0] WARN  org.apache.curator.framework.state.ConnectionStateManager - There are no ConnectionStateListeners registered.
7562 [main] INFO  backtype.storm.daemon.supervisor - Starting supervisor with id effabae2-9660-4f14-9432-fc14d3cec675 at host 10.0.0.6
7564 [main] INFO  backtype.storm.daemon.supervisor - Starting Supervisor with conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 0, "topology.skip.missing.kryo.registrations" true, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 50, "storm.messaging.netty.flush.check.interval.ms" 10, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "/var/folders/_w/shz6hqkn4cnft3phjqmt8nxr0000gp/T//9efd15e2-53d7-4871-9687-b74959dc576c", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "localhost", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2000, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["localhost"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 30, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" (1027 1028 1029), "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer" "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx1024m", "storm.cluster.mode" "local", "topology.max.task.parallelism" nil, "storm.messaging.netty.transfer.batch.size" 262144}
7566 [main] INFO  org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
7568 [main-EventThread] INFO  org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
7568 [ConnectionStateManager-0] WARN  org.apache.curator.framework.state.ConnectionStateManager - There are no ConnectionStateListeners registered.
7568 [main-EventThread] INFO  backtype.storm.zookeeper - Zookeeper state update: :connected:none
7570 [main] INFO  org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
7572 [main-EventThread] INFO  org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
7572 [ConnectionStateManager-0] WARN  org.apache.curator.framework.state.ConnectionStateManager - There are no ConnectionStateListeners registered.
7577 [main] INFO  backtype.storm.daemon.supervisor - Starting supervisor with id 8f75ba34-9581-4d0c-950f-673533114d39 at host 10.0.0.6
7712 [main] INFO  backtype.storm.daemon.nimbus - Received topology submission for wordcounter with conf {"topology.max.task.parallelism" nil, "topology.acker.executors" nil, "topology.kryo.register" {"storm.trident.spout.RichSpoutBatchId" "storm.trident.spout.RichSpoutBatchIdSerializer", "storm.trident.topology.TransactionAttempt" nil}, "topology.kryo.decorators" (), "topology.name" "wordcounter", "storm.id" "wordcounter-1-1418787394"}
7735 [main] INFO  backtype.storm.daemon.nimbus - Activating wordcounter: wordcounter-1-1418787394
7909 [main] INFO  backtype.storm.scheduler.EvenScheduler - Available slots: (["effabae2-9660-4f14-9432-fc14d3cec675" 1024] ["effabae2-9660-4f14-9432-fc14d3cec675" 1025] ["effabae2-9660-4f14-9432-fc14d3cec675" 1026] ["8f75ba34-9581-4d0c-950f-673533114d39" 1027] ["8f75ba34-9581-4d0c-950f-673533114d39" 1028] ["8f75ba34-9581-4d0c-950f-673533114d39" 1029])
7920 [main] INFO  backtype.storm.daemon.nimbus - Setting new assignment for topology id wordcounter-1-1418787394: #backtype.storm.daemon.common.Assignment{:master-code-dir "/var/folders/_w/shz6hqkn4cnft3phjqmt8nxr0000gp/T//46873e3f-e58b-4a95-81c8-c35fefe716f9/nimbus/stormdist/wordcounter-1-1418787394", :node->host {"effabae2-9660-4f14-9432-fc14d3cec675" "10.0.0.6"}, :executor->node+port {[2 2] ["effabae2-9660-4f14-9432-fc14d3cec675" 1024], [3 3] ["effabae2-9660-4f14-9432-fc14d3cec675" 1024], [4 4] ["effabae2-9660-4f14-9432-fc14d3cec675" 1024], [5 5] ["effabae2-9660-4f14-9432-fc14d3cec675" 1024], [6 6] ["effabae2-9660-4f14-9432-fc14d3cec675" 1024], [7 7] ["effabae2-9660-4f14-9432-fc14d3cec675" 1024], [8 8] ["effabae2-9660-4f14-9432-fc14d3cec675" 1024], [9 9] ["effabae2-9660-4f14-9432-fc14d3cec675" 1024], [10 10] ["effabae2-9660-4f14-9432-fc14d3cec675" 1024], [11 11] ["effabae2-9660-4f14-9432-fc14d3cec675" 1024], [12 12] ["effabae2-9660-4f14-9432-fc14d3cec675" 1024], [13 13] ["effabae2-9660-4f14-9432-fc14d3cec675" 1024], [14 14] ["effabae2-9660-4f14-9432-fc14d3cec675" 1024], [15 15] ["effabae2-9660-4f14-9432-fc14d3cec675" 1024], [16 16] ["effabae2-9660-4f14-9432-fc14d3cec675" 1024], [17 17] ["effabae2-9660-4f14-9432-fc14d3cec675" 1024], [18 18] ["effabae2-9660-4f14-9432-fc14d3cec675" 1024], [19 19] ["effabae2-9660-4f14-9432-fc14d3cec675" 1024], [20 20] ["effabae2-9660-4f14-9432-fc14d3cec675" 1024], [21 21] ["effabae2-9660-4f14-9432-fc14d3cec675" 1024], [22 22] ["effabae2-9660-4f14-9432-fc14d3cec675" 1024], [23 23] ["effabae2-9660-4f14-9432-fc14d3cec675" 1024], [24 24] ["effabae2-9660-4f14-9432-fc14d3cec675" 1024], [25 25] ["effabae2-9660-4f14-9432-fc14d3cec675" 1024], [26 26] ["effabae2-9660-4f14-9432-fc14d3cec675" 1024], [1 1] ["effabae2-9660-4f14-9432-fc14d3cec675" 1024]}, :executor->start-time-secs {[2 2] 1418787394, [3 3] 1418787394, [4 4] 1418787394, [5 5] 1418787394, [6 6] 1418787394, [7 7] 1418787394, [8 8] 1418787394, [9 9] 1418787394, [10 10] 1418787394, [11 11] 1418787394, [12 12] 1418787394, [13 13] 1418787394, [14 14] 1418787394, [15 15] 1418787394, [16 16] 1418787394, [17 17] 1418787394, [18 18] 1418787394, [19 19] 1418787394, [20 20] 1418787394, [21 21] 1418787394, [22 22] 1418787394, [23 23] 1418787394, [24 24] 1418787394, [25 25] 1418787394, [26 26] 1418787394, [1 1] 1418787394}}
8559 [Thread-3] INFO  backtype.storm.daemon.supervisor - Downloading code for storm id wordcounter-1-1418787394 from /var/folders/_w/shz6hqkn4cnft3phjqmt8nxr0000gp/T//46873e3f-e58b-4a95-81c8-c35fefe716f9/nimbus/stormdist/wordcounter-1-1418787394
8703 [Thread-3] INFO  backtype.storm.daemon.supervisor - Finished downloading code for storm id wordcounter-1-1418787394 from /var/folders/_w/shz6hqkn4cnft3phjqmt8nxr0000gp/T//46873e3f-e58b-4a95-81c8-c35fefe716f9/nimbus/stormdist/wordcounter-1-1418787394
8712 [Thread-4] INFO  backtype.storm.daemon.supervisor - Launching worker with assignment #backtype.storm.daemon.supervisor.LocalAssignment{:storm-id "wordcounter-1-1418787394", :executors ([2 2] [3 3] [4 4] [5 5] [6 6] [7 7] [8 8] [9 9] [10 10] [11 11] [12 12] [13 13] [14 14] [15 15] [16 16] [17 17] [18 18] [19 19] [20 20] [21 21] [22 22] [23 23] [24 24] [25 25] [26 26] [1 1])} for this supervisor effabae2-9660-4f14-9432-fc14d3cec675 on port 1024 with id c910aef9-b221-4444-a0a1-50349224379e
8713 [Thread-4] INFO  backtype.storm.daemon.worker - Launching worker for wordcounter-1-1418787394 on effabae2-9660-4f14-9432-fc14d3cec675:1024 with id c910aef9-b221-4444-a0a1-50349224379e and conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 0, "topology.skip.missing.kryo.registrations" true, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 50, "storm.messaging.netty.flush.check.interval.ms" 10, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "/var/folders/_w/shz6hqkn4cnft3phjqmt8nxr0000gp/T//3ccad0bd-b27a-487b-a222-7a01190b4d3d", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "localhost", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2000, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["localhost"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 30, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" (1024 1025 1026), "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer" "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx1024m", "storm.cluster.mode" "local", "topology.max.task.parallelism" nil, "storm.messaging.netty.transfer.batch.size" 262144}
8713 [Thread-4] INFO  org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
13784 [Thread-4-EventThread] INFO  org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
13784 [ConnectionStateManager-0] WARN  org.apache.curator.framework.state.ConnectionStateManager - There are no ConnectionStateListeners registered.
13784 [Thread-4-EventThread] INFO  backtype.storm.zookeeper - Zookeeper state update: :connected:none
13786 [Thread-4] INFO  org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
13787 [Thread-4-EventThread] INFO  org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
13787 [ConnectionStateManager-0] WARN  org.apache.curator.framework.state.ConnectionStateManager - There are no ConnectionStateListeners registered.
13948 [Thread-4] INFO  backtype.storm.daemon.executor - Loading executor $spoutcoord-spout0:[2 2]
13952 [Thread-4] INFO  backtype.storm.daemon.executor - Loaded executor tasks $spoutcoord-spout0:[2 2]
13957 [Thread-4] INFO  backtype.storm.daemon.executor - Finished loading executor $spoutcoord-spout0:[2 2]
13958 [Thread-9-$spoutcoord-spout0] INFO  backtype.storm.daemon.executor - Preparing bolt $spoutcoord-spout0:(2)
13961 [Thread-9-$spoutcoord-spout0] INFO  org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
13963 [Thread-4] INFO  backtype.storm.daemon.executor - Loading executor __acker:[3 3]
13963 [Thread-4] INFO  backtype.storm.daemon.executor - Loaded executor tasks __acker:[3 3]
13964 [Thread-9-$spoutcoord-spout0-EventThread] INFO  org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
13964 [ConnectionStateManager-0] WARN  org.apache.curator.framework.state.ConnectionStateManager - There are no ConnectionStateListeners registered.
13965 [Thread-4] INFO  backtype.storm.daemon.executor - Timeouts disabled for executor __acker:[3 3]
13965 [Thread-11-__acker] INFO  backtype.storm.daemon.executor - Preparing bolt __acker:(3)
13965 [Thread-4] INFO  backtype.storm.daemon.executor - Finished loading executor __acker:[3 3]
13967 [Thread-11-__acker] INFO  backtype.storm.daemon.executor - Prepared bolt __acker:(3)
13969 [Thread-4] INFO  backtype.storm.daemon.executor - Loading executor b-0:[4 4]
13970 [Thread-9-$spoutcoord-spout0] INFO  org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
13971 [Thread-9-$spoutcoord-spout0-EventThread] INFO  org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
13972 [ConnectionStateManager-0] WARN  org.apache.curator.framework.state.ConnectionStateManager - There are no ConnectionStateListeners registered.
13973 [Thread-9-$spoutcoord-spout0] INFO  backtype.storm.daemon.executor - Prepared bolt $spoutcoord-spout0:(2)
13974 [Thread-4] INFO  backtype.storm.daemon.executor - Loaded executor tasks b-0:[4 4]
13975 [Thread-13-b-0] INFO  backtype.storm.daemon.executor - Preparing bolt b-0:(4)
13975 [Thread-4] INFO  backtype.storm.daemon.executor - Finished loading executor b-0:[4 4]
13979 [Thread-4] INFO  backtype.storm.daemon.executor - Loading executor b-1:[5 5]
13981 [Thread-4] INFO  backtype.storm.daemon.executor - Loaded executor tasks b-1:[5 5]
13982 [Thread-4] INFO  backtype.storm.daemon.executor - Finished loading executor b-1:[5 5]
13982 [Thread-15-b-1] INFO  backtype.storm.daemon.executor - Preparing bolt b-1:(5)
13982 [Thread-15-b-1] INFO  backtype.storm.daemon.executor - Prepared bolt b-1:(5)
13983 [Thread-13-b-0] INFO  backtype.storm.daemon.executor - Prepared bolt b-0:(4)
13985 [Thread-4] INFO  backtype.storm.daemon.executor - Loading executor b-2:[6 6]
13987 [Thread-4] INFO  backtype.storm.daemon.executor - Loaded executor tasks b-2:[6 6]
13988 [Thread-4] INFO  backtype.storm.daemon.executor - Finished loading executor b-2:[6 6]
13988 [Thread-17-b-2] INFO  backtype.storm.daemon.executor - Preparing bolt b-2:(6)
13988 [Thread-17-b-2] INFO  backtype.storm.daemon.executor - Prepared bolt b-2:(6)
13992 [Thread-4] INFO  backtype.storm.daemon.executor - Loading executor b-3:[7 7]
13994 [Thread-4] INFO  backtype.storm.daemon.executor - Loaded executor tasks b-3:[7 7]
13995 [Thread-19-b-3] INFO  backtype.storm.daemon.executor - Preparing bolt b-3:(7)
13995 [Thread-4] INFO  backtype.storm.daemon.executor - Finished loading executor b-3:[7 7]
13996 [Thread-19-b-3] INFO  backtype.storm.daemon.executor - Prepared bolt b-3:(7)
13999 [Thread-4] INFO  backtype.storm.daemon.executor - Loading executor b-4:[8 8]
14000 [Thread-4] INFO  backtype.storm.daemon.executor - Loaded executor tasks b-4:[8 8]
14001 [Thread-4] INFO  backtype.storm.daemon.executor - Finished loading executor b-4:[8 8]
14001 [Thread-21-b-4] INFO  backtype.storm.daemon.executor - Preparing bolt b-4:(8)
14004 [Thread-4] INFO  backtype.storm.daemon.executor - Loading executor b-4:[9 9]
14005 [Thread-21-b-4] INFO  backtype.storm.daemon.executor - Prepared bolt b-4:(8)
14005 [Thread-4] INFO  backtype.storm.daemon.executor - Loaded executor tasks b-4:[9 9]
14007 [Thread-4] INFO  backtype.storm.daemon.executor - Finished loading executor b-4:[9 9]
14007 [Thread-23-b-4] INFO  backtype.storm.daemon.executor - Preparing bolt b-4:(9)
14008 [Thread-23-b-4] INFO  backtype.storm.daemon.executor - Prepared bolt b-4:(9)
14010 [Thread-4] INFO  backtype.storm.daemon.executor - Loading executor b-4:[10 10]
14011 [Thread-4] INFO  backtype.storm.daemon.executor - Loaded executor tasks b-4:[10 10]
14012 [Thread-4] INFO  backtype.storm.daemon.executor - Finished loading executor b-4:[10 10]
14012 [Thread-25-b-4] INFO  backtype.storm.daemon.executor - Preparing bolt b-4:(10)
14013 [Thread-25-b-4] INFO  backtype.storm.daemon.executor - Prepared bolt b-4:(10)
14015 [Thread-4] INFO  backtype.storm.daemon.executor - Loading executor b-4:[11 11]
14016 [Thread-4] INFO  backtype.storm.daemon.executor - Loaded executor tasks b-4:[11 11]
14017 [Thread-4] INFO  backtype.storm.daemon.executor - Finished loading executor b-4:[11 11]
14017 [Thread-27-b-4] INFO  backtype.storm.daemon.executor - Preparing bolt b-4:(11)
14018 [Thread-27-b-4] INFO  backtype.storm.daemon.executor - Prepared bolt b-4:(11)
14020 [Thread-4] INFO  backtype.storm.daemon.executor - Loading executor b-4:[12 12]
14022 [Thread-4] INFO  backtype.storm.daemon.executor - Loaded executor tasks b-4:[12 12]
14023 [Thread-4] INFO  backtype.storm.daemon.executor - Finished loading executor b-4:[12 12]
14023 [Thread-29-b-4] INFO  backtype.storm.daemon.executor - Preparing bolt b-4:(12)
14024 [Thread-29-b-4] INFO  backtype.storm.daemon.executor - Prepared bolt b-4:(12)
14026 [Thread-4] INFO  backtype.storm.daemon.executor - Loading executor b-4:[13 13]
14028 [Thread-4] INFO  backtype.storm.daemon.executor - Loaded executor tasks b-4:[13 13]
14029 [Thread-4] INFO  backtype.storm.daemon.executor - Finished loading executor b-4:[13 13]
14029 [Thread-31-b-4] INFO  backtype.storm.daemon.executor - Preparing bolt b-4:(13)
14030 [Thread-31-b-4] INFO  backtype.storm.daemon.executor - Prepared bolt b-4:(13)
14031 [Thread-4] INFO  backtype.storm.daemon.executor - Loading executor b-4:[14 14]
14032 [Thread-4] INFO  backtype.storm.daemon.executor - Loaded executor tasks b-4:[14 14]
14033 [Thread-4] INFO  backtype.storm.daemon.executor - Finished loading executor b-4:[14 14]
14033 [Thread-33-b-4] INFO  backtype.storm.daemon.executor - Preparing bolt b-4:(14)
14035 [Thread-33-b-4] INFO  backtype.storm.daemon.executor - Prepared bolt b-4:(14)
14036 [Thread-4] INFO  backtype.storm.daemon.executor - Loading executor b-4:[15 15]
14037 [Thread-4] INFO  backtype.storm.daemon.executor - Loaded executor tasks b-4:[15 15]
14038 [Thread-4] INFO  backtype.storm.daemon.executor - Finished loading executor b-4:[15 15]
14038 [Thread-35-b-4] INFO  backtype.storm.daemon.executor - Preparing bolt b-4:(15)
14039 [Thread-35-b-4] INFO  backtype.storm.daemon.executor - Prepared bolt b-4:(15)
14040 [Thread-4] INFO  backtype.storm.daemon.executor - Loading executor b-4:[16 16]
14041 [Thread-4] INFO  backtype.storm.daemon.executor - Loaded executor tasks b-4:[16 16]
14042 [Thread-4] INFO  backtype.storm.daemon.executor - Finished loading executor b-4:[16 16]
14042 [Thread-37-b-4] INFO  backtype.storm.daemon.executor - Preparing bolt b-4:(16)
14044 [Thread-37-b-4] INFO  backtype.storm.daemon.executor - Prepared bolt b-4:(16)
14045 [Thread-4] INFO  backtype.storm.daemon.executor - Loading executor b-4:[17 17]
14048 [Thread-4] INFO  backtype.storm.daemon.executor - Loaded executor tasks b-4:[17 17]
14049 [Thread-4] INFO  backtype.storm.daemon.executor - Finished loading executor b-4:[17 17]
14049 [Thread-39-b-4] INFO  backtype.storm.daemon.executor - Preparing bolt b-4:(17)
14050 [Thread-39-b-4] INFO  backtype.storm.daemon.executor - Prepared bolt b-4:(17)
14051 [Thread-4] INFO  backtype.storm.daemon.executor - Loading executor b-4:[18 18]
14053 [Thread-4] INFO  backtype.storm.daemon.executor - Loaded executor tasks b-4:[18 18]
14054 [Thread-4] INFO  backtype.storm.daemon.executor - Finished loading executor b-4:[18 18]
14054 [Thread-41-b-4] INFO  backtype.storm.daemon.executor - Preparing bolt b-4:(18)
14055 [Thread-41-b-4] INFO  backtype.storm.daemon.executor - Prepared bolt b-4:(18)
14056 [Thread-4] INFO  backtype.storm.daemon.executor - Loading executor b-4:[19 19]
14058 [Thread-4] INFO  backtype.storm.daemon.executor - Loaded executor tasks b-4:[19 19]
14059 [Thread-4] INFO  backtype.storm.daemon.executor - Finished loading executor b-4:[19 19]
14059 [Thread-43-b-4] INFO  backtype.storm.daemon.executor - Preparing bolt b-4:(19)
14060 [Thread-43-b-4] INFO  backtype.storm.daemon.executor - Prepared bolt b-4:(19)
14062 [Thread-4] INFO  backtype.storm.daemon.executor - Loading executor b-4:[20 20]
14063 [Thread-4] INFO  backtype.storm.daemon.executor - Loaded executor tasks b-4:[20 20]
14064 [Thread-4] INFO  backtype.storm.daemon.executor - Finished loading executor b-4:[20 20]
14064 [Thread-45-b-4] INFO  backtype.storm.daemon.executor - Preparing bolt b-4:(20)
14066 [Thread-45-b-4] INFO  backtype.storm.daemon.executor - Prepared bolt b-4:(20)
14067 [Thread-4] INFO  backtype.storm.daemon.executor - Loading executor b-4:[21 21]
14068 [Thread-4] INFO  backtype.storm.daemon.executor - Loaded executor tasks b-4:[21 21]
14069 [Thread-4] INFO  backtype.storm.daemon.executor - Finished loading executor b-4:[21 21]
14069 [Thread-47-b-4] INFO  backtype.storm.daemon.executor - Preparing bolt b-4:(21)
14071 [Thread-47-b-4] INFO  backtype.storm.daemon.executor - Prepared bolt b-4:(21)
14072 [Thread-4] INFO  backtype.storm.daemon.executor - Loading executor b-4:[22 22]
14073 [Thread-4] INFO  backtype.storm.daemon.executor - Loaded executor tasks b-4:[22 22]
14075 [Thread-4] INFO  backtype.storm.daemon.executor - Finished loading executor b-4:[22 22]
14075 [Thread-49-b-4] INFO  backtype.storm.daemon.executor - Preparing bolt b-4:(22)
14076 [Thread-49-b-4] INFO  backtype.storm.daemon.executor - Prepared bolt b-4:(22)
14077 [Thread-4] INFO  backtype.storm.daemon.executor - Loading executor b-4:[23 23]
14079 [Thread-4] INFO  backtype.storm.daemon.executor - Loaded executor tasks b-4:[23 23]
14080 [Thread-51-b-4] INFO  backtype.storm.daemon.executor - Preparing bolt b-4:(23)
14080 [Thread-4] INFO  backtype.storm.daemon.executor - Finished loading executor b-4:[23 23]
14081 [Thread-51-b-4] INFO  backtype.storm.daemon.executor - Prepared bolt b-4:(23)
14083 [Thread-4] INFO  backtype.storm.daemon.executor - Loading executor b-5:[24 24]
14084 [Thread-4] INFO  backtype.storm.daemon.executor - Loaded executor tasks b-5:[24 24]
14085 [Thread-4] INFO  backtype.storm.daemon.executor - Finished loading executor b-5:[24 24]
14085 [Thread-53-b-5] INFO  backtype.storm.daemon.executor - Preparing bolt b-5:(24)
14086 [Thread-53-b-5] INFO  backtype.storm.daemon.executor - Prepared bolt b-5:(24)
14088 [Thread-4] INFO  backtype.storm.daemon.executor - Loading executor spout0:[25 25]
14088 [Thread-4] INFO  backtype.storm.daemon.executor - Loaded executor tasks spout0:[25 25]
14089 [Thread-4] INFO  backtype.storm.daemon.executor - Finished loading executor spout0:[25 25]
14089 [Thread-55-spout0] INFO  backtype.storm.daemon.executor - Preparing bolt spout0:(25)
14090 [Thread-55-spout0] INFO  backtype.storm.daemon.executor - Prepared bolt spout0:(25)
14092 [Thread-4] INFO  backtype.storm.daemon.executor - Loading executor spout1:[26 26]
14092 [Thread-4] INFO  backtype.storm.daemon.executor - Loaded executor tasks spout1:[26 26]
14096 [Thread-4] INFO  backtype.storm.daemon.executor - Finished loading executor spout1:[26 26]
14096 [Thread-57-spout1] INFO  backtype.storm.daemon.executor - Opening spout spout1:(26)
14098 [Thread-57-spout1] INFO  backtype.storm.daemon.executor - Opened spout spout1:(26)
14099 [Thread-4] INFO  backtype.storm.daemon.executor - Loading executor __system:[-1 -1]
14099 [Thread-4] INFO  backtype.storm.daemon.executor - Loaded executor tasks __system:[-1 -1]
14100 [Thread-57-spout1] INFO  backtype.storm.daemon.executor - Activating spout spout1:(26)
14100 [Thread-4] INFO  backtype.storm.daemon.executor - Finished loading executor __system:[-1 -1]
14100 [Thread-59-__system] INFO  backtype.storm.daemon.executor - Preparing bolt __system:(-1)
14102 [Thread-59-__system] INFO  backtype.storm.daemon.executor - Prepared bolt __system:(-1)
14103 [Thread-4] INFO  backtype.storm.daemon.executor - Loading executor $mastercoord-bg0:[1 1]
14104 [Thread-4] INFO  backtype.storm.daemon.executor - Loaded executor tasks $mastercoord-bg0:[1 1]
14104 [Thread-4] INFO  backtype.storm.daemon.executor - Finished loading executor $mastercoord-bg0:[1 1]
14104 [Thread-61-$mastercoord-bg0] INFO  backtype.storm.daemon.executor - Opening spout $mastercoord-bg0:(1)
14105 [Thread-4] INFO  backtype.storm.daemon.worker - Launching receive-thread for effabae2-9660-4f14-9432-fc14d3cec675:1024
14105 [Thread-61-$mastercoord-bg0] INFO  org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
14108 [Thread-62-worker-receiver-thread-0] INFO  backtype.storm.messaging.loader - Starting receive-thread: [stormId: wordcounter-1-1418787394, port: 1024, thread-id: 0 ]
14112 [Thread-4] INFO  backtype.storm.daemon.worker - Worker has topology config {"storm.id" "wordcounter-1-1418787394", "dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 0, "topology.skip.missing.kryo.registrations" true, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 50, "storm.messaging.netty.flush.check.interval.ms" 10, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "/var/folders/_w/shz6hqkn4cnft3phjqmt8nxr0000gp/T//3ccad0bd-b27a-487b-a222-7a01190b4d3d", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "localhost", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2000, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["localhost"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.kryo.decorators" (), "topology.name" "wordcounter", "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 30, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" (1024 1025 1026), "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.kryo.register" {"storm.trident.spout.RichSpoutBatchId" "storm.trident.spout.RichSpoutBatchIdSerializer", "storm.trident.topology.TransactionAttempt" nil}, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer" "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx1024m", "storm.cluster.mode" "local", "topology.max.task.parallelism" nil, "storm.messaging.netty.transfer.batch.size" 262144}
14113 [Thread-4] INFO  backtype.storm.daemon.worker - Worker c910aef9-b221-4444-a0a1-50349224379e for storm wordcounter-1-1418787394 on effabae2-9660-4f14-9432-fc14d3cec675:1024 has finished loading
DEBUG: [0]
18942 [main] INFO  backtype.storm.daemon.nimbus - Shutting down master
18946 [main] INFO  backtype.storm.daemon.nimbus - Shut down master
18949 [main] INFO  backtype.storm.daemon.supervisor - Shutting down effabae2-9660-4f14-9432-fc14d3cec675:c910aef9-b221-4444-a0a1-50349224379e
18949 [main] INFO  backtype.storm.process-simulator - Killing process e78c9abc-2af2-4169-86d3-6e07a867feeb
18950 [main] INFO  backtype.storm.daemon.worker - Shutting down worker wordcounter-1-1418787394 effabae2-9660-4f14-9432-fc14d3cec675 1024
18950 [main] INFO  backtype.storm.daemon.worker - Shutting down receive thread
18950 [main] INFO  backtype.storm.messaging.loader - Shutting down receiving-thread: [wordcounter-1-1418787394, 1024]
18950 [main] INFO  backtype.storm.messaging.loader - Waiting for receiving-thread:[wordcounter-1-1418787394, 1024] to die
18950 [Thread-62-worker-receiver-thread-0] INFO  backtype.storm.messaging.loader - Receiving-thread:[wordcounter-1-1418787394, 1024] received shutdown notice
18951 [main] INFO  backtype.storm.messaging.loader - Shutdown receiving-thread: [wordcounter-1-1418787394, 1024]
18951 [main] INFO  backtype.storm.daemon.worker - Shut down receive thread
18951 [main] INFO  backtype.storm.daemon.worker - Terminating messaging context
18951 [main] INFO  backtype.storm.daemon.worker - Shutting down executors
18951 [main] INFO  backtype.storm.daemon.executor - Shutting down executor $spoutcoord-spout0:[2 2]
18952 [Thread-9-$spoutcoord-spout0] INFO  backtype.storm.util - Async loop interrupted!
18952 [Thread-8-disruptor-executor[2 2]-send-queue] INFO  backtype.storm.util - Async loop interrupted!
18953 [main] INFO  backtype.storm.daemon.executor - Shut down executor $spoutcoord-spout0:[2 2]
18953 [main] INFO  backtype.storm.daemon.executor - Shutting down executor __acker:[3 3]
18953 [Thread-10-disruptor-executor[3 3]-send-queue] INFO  backtype.storm.util - Async loop interrupted!
18953 [Thread-11-__acker] INFO  backtype.storm.util - Async loop interrupted!
18953 [main] INFO  backtype.storm.daemon.executor - Shut down executor __acker:[3 3]
18953 [main] INFO  backtype.storm.daemon.executor - Shutting down executor b-0:[4 4]
18953 [Thread-13-b-0] INFO  backtype.storm.util - Async loop interrupted!
18953 [Thread-12-disruptor-executor[4 4]-send-queue] INFO  backtype.storm.util - Async loop interrupted!
18954 [main] INFO  backtype.storm.daemon.executor - Shut down executor b-0:[4 4]
18954 [main] INFO  backtype.storm.daemon.executor - Shutting down executor b-1:[5 5]
18954 [Thread-15-b-1] INFO  backtype.storm.util - Async loop interrupted!
18954 [Thread-14-disruptor-executor[5 5]-send-queue] INFO  backtype.storm.util - Async loop interrupted!
18954 [main] INFO  backtype.storm.daemon.executor - Shut down executor b-1:[5 5]
18954 [main] INFO  backtype.storm.daemon.executor - Shutting down executor b-2:[6 6]
18954 [Thread-17-b-2] INFO  backtype.storm.util - Async loop interrupted!
18954 [Thread-16-disruptor-executor[6 6]-send-queue] INFO  backtype.storm.util - Async loop interrupted!
18955 [main] INFO  backtype.storm.daemon.executor - Shut down executor b-2:[6 6]
18955 [main] INFO  backtype.storm.daemon.executor - Shutting down executor b-3:[7 7]
18955 [Thread-19-b-3] INFO  backtype.storm.util - Async loop interrupted!
18955 [Thread-18-disruptor-executor[7 7]-send-queue] INFO  backtype.storm.util - Async loop interrupted!
18955 [main] INFO  backtype.storm.daemon.executor - Shut down executor b-3:[7 7]
18955 [main] INFO  backtype.storm.daemon.executor - Shutting down executor b-4:[8 8]
18955 [Thread-21-b-4] INFO  backtype.storm.util - Async loop interrupted!
18955 [Thread-20-disruptor-executor[8 8]-send-queue] INFO  backtype.storm.util - Async loop interrupted!
18955 [main] INFO  backtype.storm.daemon.executor - Shut down executor b-4:[8 8]
18956 [main] INFO  backtype.storm.daemon.executor - Shutting down executor b-4:[9 9]
18956 [Thread-23-b-4] INFO  backtype.storm.util - Async loop interrupted!
18956 [Thread-22-disruptor-executor[9 9]-send-queue] INFO  backtype.storm.util - Async loop interrupted!
18956 [main] INFO  backtype.storm.daemon.executor - Shut down executor b-4:[9 9]
18956 [main] INFO  backtype.storm.daemon.executor - Shutting down executor b-4:[10 10]
18956 [Thread-25-b-4] INFO  backtype.storm.util - Async loop interrupted!
18956 [Thread-24-disruptor-executor[10 10]-send-queue] INFO  backtype.storm.util - Async loop interrupted!
18957 [main] INFO  backtype.storm.daemon.executor - Shut down executor b-4:[10 10]
18957 [main] INFO  backtype.storm.daemon.executor - Shutting down executor b-4:[11 11]
18957 [Thread-27-b-4] INFO  backtype.storm.util - Async loop interrupted!
18957 [Thread-26-disruptor-executor[11 11]-send-queue] INFO  backtype.storm.util - Async loop interrupted!
18957 [main] INFO  backtype.storm.daemon.executor - Shut down executor b-4:[11 11]
18957 [main] INFO  backtype.storm.daemon.executor - Shutting down executor b-4:[12 12]
18957 [Thread-29-b-4] INFO  backtype.storm.util - Async loop interrupted!
18957 [Thread-28-disruptor-executor[12 12]-send-queue] INFO  backtype.storm.util - Async loop interrupted!
18957 [main] INFO  backtype.storm.daemon.executor - Shut down executor b-4:[12 12]
18958 [main] INFO  backtype.storm.daemon.executor - Shutting down executor b-4:[13 13]
18958 [Thread-31-b-4] INFO  backtype.storm.util - Async loop interrupted!
18958 [Thread-30-disruptor-executor[13 13]-send-queue] INFO  backtype.storm.util - Async loop interrupted!
18958 [main] INFO  backtype.storm.daemon.executor - Shut down executor b-4:[13 13]
18958 [main] INFO  backtype.storm.daemon.executor - Shutting down executor b-4:[14 14]
18958 [Thread-33-b-4] INFO  backtype.storm.util - Async loop interrupted!
18958 [Thread-32-disruptor-executor[14 14]-send-queue] INFO  backtype.storm.util - Async loop interrupted!
18958 [main] INFO  backtype.storm.daemon.executor - Shut down executor b-4:[14 14]
18958 [main] INFO  backtype.storm.daemon.executor - Shutting down executor b-4:[15 15]
18959 [Thread-35-b-4] INFO  backtype.storm.util - Async loop interrupted!
18959 [Thread-34-disruptor-executor[15 15]-send-queue] INFO  backtype.storm.util - Async loop interrupted!
18959 [main] INFO  backtype.storm.daemon.executor - Shut down executor b-4:[15 15]
18959 [main] INFO  backtype.storm.daemon.executor - Shutting down executor b-4:[16 16]
18959 [Thread-36-disruptor-executor[16 16]-send-queue] INFO  backtype.storm.util - Async loop interrupted!
18959 [Thread-37-b-4] INFO  backtype.storm.util - Async loop interrupted!
18959 [main] INFO  backtype.storm.daemon.executor - Shut down executor b-4:[16 16]
18959 [main] INFO  backtype.storm.daemon.executor - Shutting down executor b-4:[17 17]
18960 [Thread-39-b-4] INFO  backtype.storm.util - Async loop interrupted!
18960 [Thread-38-disruptor-executor[17 17]-send-queue] INFO  backtype.storm.util - Async loop interrupted!
18960 [main] INFO  backtype.storm.daemon.executor - Shut down executor b-4:[17 17]
18960 [main] INFO  backtype.storm.daemon.executor - Shutting down executor b-4:[18 18]
18960 [Thread-41-b-4] INFO  backtype.storm.util - Async loop interrupted!
18960 [Thread-40-disruptor-executor[18 18]-send-queue] INFO  backtype.storm.util - Async loop interrupted!
18960 [main] INFO  backtype.storm.daemon.executor - Shut down executor b-4:[18 18]
18960 [main] INFO  backtype.storm.daemon.executor - Shutting down executor b-4:[19 19]
18960 [Thread-43-b-4] INFO  backtype.storm.util - Async loop interrupted!
18960 [Thread-42-disruptor-executor[19 19]-send-queue] INFO  backtype.storm.util - Async loop interrupted!
18961 [main] INFO  backtype.storm.daemon.executor - Shut down executor b-4:[19 19]
18961 [main] INFO  backtype.storm.daemon.executor - Shutting down executor b-4:[20 20]
18961 [Thread-45-b-4] INFO  backtype.storm.util - Async loop interrupted!
18961 [Thread-44-disruptor-executor[20 20]-send-queue] INFO  backtype.storm.util - Async loop interrupted!
18961 [main] INFO  backtype.storm.daemon.executor - Shut down executor b-4:[20 20]
18961 [main] INFO  backtype.storm.daemon.executor - Shutting down executor b-4:[21 21]
18961 [Thread-47-b-4] INFO  backtype.storm.util - Async loop interrupted!
18961 [Thread-46-disruptor-executor[21 21]-send-queue] INFO  backtype.storm.util - Async loop interrupted!
18962 [main] INFO  backtype.storm.daemon.executor - Shut down executor b-4:[21 21]
18962 [main] INFO  backtype.storm.daemon.executor - Shutting down executor b-4:[22 22]
18962 [Thread-49-b-4] INFO  backtype.storm.util - Async loop interrupted!
18962 [Thread-48-disruptor-executor[22 22]-send-queue] INFO  backtype.storm.util - Async loop interrupted!
18962 [main] INFO  backtype.storm.daemon.executor - Shut down executor b-4:[22 22]
18962 [main] INFO  backtype.storm.daemon.executor - Shutting down executor b-4:[23 23]
18962 [Thread-51-b-4] INFO  backtype.storm.util - Async loop interrupted!
18962 [Thread-50-disruptor-executor[23 23]-send-queue] INFO  backtype.storm.util - Async loop interrupted!
18962 [main] INFO  backtype.storm.daemon.executor - Shut down executor b-4:[23 23]
18962 [main] INFO  backtype.storm.daemon.executor - Shutting down executor b-5:[24 24]
18963 [Thread-53-b-5] INFO  backtype.storm.util - Async loop interrupted!
18963 [Thread-52-disruptor-executor[24 24]-send-queue] INFO  backtype.storm.util - Async loop interrupted!
18963 [main] INFO  backtype.storm.daemon.executor - Shut down executor b-5:[24 24]
18963 [main] INFO  backtype.storm.daemon.executor - Shutting down executor spout0:[25 25]
18963 [Thread-55-spout0] INFO  backtype.storm.util - Async loop interrupted!
18963 [Thread-54-disruptor-executor[25 25]-send-queue] INFO  backtype.storm.util - Async loop interrupted!
18963 [main] INFO  backtype.storm.daemon.executor - Shut down executor spout0:[25 25]
18963 [main] INFO  backtype.storm.daemon.executor - Shutting down executor spout1:[26 26]
18963 [Thread-56-disruptor-executor[26 26]-send-queue] INFO  backtype.storm.util - Async loop interrupted!
18963 [Thread-57-spout1] INFO  backtype.storm.util - Async loop interrupted!
18964 [main] INFO  backtype.storm.daemon.executor - Shut down executor spout1:[26 26]
18964 [main] INFO  backtype.storm.daemon.executor - Shutting down executor __system:[-1 -1]
18964 [Thread-59-__system] INFO  backtype.storm.util - Async loop interrupted!
18964 [Thread-58-disruptor-executor[-1 -1]-send-queue] INFO  backtype.storm.util - Async loop interrupted!
18964 [main] INFO  backtype.storm.daemon.executor - Shut down executor __system:[-1 -1]
18964 [main] INFO  backtype.storm.daemon.executor - Shutting down executor $mastercoord-bg0:[1 1]
18964 [Thread-60-disruptor-executor[1 1]-send-queue] INFO  backtype.storm.util - Async loop interrupted!
18964 [Thread-61-$mastercoord-bg0] INFO  backtype.storm.util - Async loop interrupted!
18964 [main] INFO  backtype.storm.daemon.executor - Shut down executor $mastercoord-bg0:[1 1]
18964 [main] INFO  backtype.storm.daemon.worker - Shut down executors
18964 [main] INFO  backtype.storm.daemon.worker - Shutting down transfer thread
18964 [Thread-63-disruptor-worker-transfer-queue] INFO  backtype.storm.util - Async loop interrupted!
18965 [main] INFO  backtype.storm.daemon.worker - Shut down transfer thread
18965 [main] INFO  backtype.storm.daemon.worker - Shutting down default resources
18965 [main] INFO  backtype.storm.daemon.worker - Shut down default resources
18968 [main] INFO  backtype.storm.daemon.worker - Disconnecting from storm cluster state context
18969 [main] INFO  backtype.storm.daemon.worker - Shut down worker wordcounter-1-1418787394 effabae2-9660-4f14-9432-fc14d3cec675 1024
18971 [main] INFO  backtype.storm.daemon.supervisor - Shut down effabae2-9660-4f14-9432-fc14d3cec675:c910aef9-b221-4444-a0a1-50349224379e
18971 [main] INFO  backtype.storm.daemon.supervisor - Shutting down supervisor effabae2-9660-4f14-9432-fc14d3cec675
18972 [Thread-3] INFO  backtype.storm.event - Event manager interrupted
18972 [Thread-4] INFO  backtype.storm.event - Event manager interrupted
18973 [main] INFO  backtype.storm.daemon.supervisor - Shutting down supervisor 8f75ba34-9581-4d0c-950f-673533114d39
18973 [Thread-5] INFO  backtype.storm.event - Event manager interrupted
18973 [Thread-6] INFO  backtype.storm.event - Event manager interrupted
18974 [main] INFO  backtype.storm.testing - Shutting down in process zookeeper
18974 [main] INFO  backtype.storm.testing - Done shutting down in process zookeeper
18974 [main] INFO  backtype.storm.testing - Deleting temporary path /var/folders/_w/shz6hqkn4cnft3phjqmt8nxr0000gp/T//46873e3f-e58b-4a95-81c8-c35fefe716f9
18975 [main] INFO  backtype.storm.testing - Deleting temporary path /var/folders/_w/shz6hqkn4cnft3phjqmt8nxr0000gp/T//2fb54cd4-9f5d-426a-b0c2-f8002c0d880f
18976 [main] INFO  backtype.storm.testing - Deleting temporary path /var/folders/_w/shz6hqkn4cnft3phjqmt8nxr0000gp/T//3ccad0bd-b27a-487b-a222-7a01190b4d3d
18978 [main] INFO  backtype.storm.testing - Deleting temporary path /var/folders/_w/shz6hqkn4cnft3phjqmt8nxr0000gp/T//9efd15e2-53d7-4871-9687-b74959dc576c

From: Homer Strong notifications@github.com<mailto:notifications@github.com> Reply-To: yieldbot/marceline reply@reply.github.com<mailto:reply@reply.github.com> Date: Tuesday, December 16, 2014 at 18:34 To: yieldbot/marceline marceline@noreply.github.com<mailto:marceline@noreply.github.com> Cc: Steve Nunez snunez@hortonworks.com<mailto:snunez@hortonworks.com> Subject: Re: [marceline] Can't Run Standalone (#15)

Yes :) The examples should work locally, marceline just uses Storm's "local mode" for this. This local mode includes an in-process ZK, so there is ZK running. It may log junk which does not affect the behavior of test topologies (e.g. polling while the in-process ZK starts up perhaps?).

This can be done for the test/example topologies with e.g.

lein run -m marceline.topology.wordcount-metrics

Is that approximately what you tried? If that command fails, please include the full output.

If you're interested in how this is implemented, check out

https://github.com/yieldbot/marceline/blob/master/test/marceline/topology/wordcount_metrics.clj#L93-L107

— Reply to this email directly or view it on GitHubhttps://github.com/yieldbot/marceline/issues/15#issuecomment-67269054.

CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.

strongh commented 9 years ago

That seems to be running to me! There's a lot of logging that isn't important, and very little that is actually coming from the topology. The line below

DEBUG: [0]

Is coming from here: https://github.com/yieldbot/marceline/blob/master/test/marceline/topology/wordcount_metrics.clj#L90

Perhaps marcy's local logging configuration could be less verbose than storm's default.

strongh commented 9 years ago

Although... it is a bit strange that the DEBUG'd DRPC response is [0], I would expect some nonzero count there. So there may be some issue with this example topology, however, storm itself is running properly.

snunez1 commented 9 years ago

You're right that DEBUG should not be 0. I'll have a look, though if someone more familiar with the code could do likewise, it will be faster.

Final question though: is the example in the README.md supposed to be runnable? I'd suggest that having a runnable example to work through would be best. I've created a new project, update the dependencies (based on the project.clj in the project) and done 'run lien', but keep getting:

584392 [Thread-29-$mastercoord-bg0-SendThread(localhost:2000)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_05] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712) ~[na:1.8.0_05] at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) ~[zookeeper-3.4.5.jar:3.4.5-1392090] at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068) ~[zookeeper-3.4.5.jar:3.4.5-1392090]

Any ideas?

From: Homer Strong notifications@github.com<mailto:notifications@github.com> Reply-To: yieldbot/marceline reply@reply.github.com<mailto:reply@reply.github.com> Date: Tuesday, December 16, 2014 at 21:17 To: yieldbot/marceline marceline@noreply.github.com<mailto:marceline@noreply.github.com> Cc: Steve Nunez snunez@hortonworks.com<mailto:snunez@hortonworks.com> Subject: Re: [marceline] Can't Run Standalone (#15)

Although... it is a bit strange that the DEBUG'd DRPC response is [0], I would expect some nonzero count there. So there may be some issue with this example topology, however, storm itself is running properly.

Reply to this email directly or view it on GitHubhttps://github.com/yieldbot/marceline/issues/15#issuecomment-67278358.

CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.

strongh commented 9 years ago

Yes, the example should build up a runnable topology, I believe it's basically a walkthrough of the wordcount [1] topology.

I still don't see any errors in the output that you've provided - those WARNings are typical of local storm running. So as far as I can tell, your topology is working.

[1] https://github.com/yieldbot/marceline/blob/master/test/marceline/topology/wordcount.clj

snunez1 commented 9 years ago

I think you're right. The topology does seem to start-up and shut-down properly. Now if only I could see some output from marceline. Are there any additional examples anywhere? Those three 'blog posts don't help a great deal in getting started.

From: Homer Strong notifications@github.com<mailto:notifications@github.com> Reply-To: yieldbot/marceline reply@reply.github.com<mailto:reply@reply.github.com> Date: Wednesday, December 17, 2014 at 11:02 To: yieldbot/marceline marceline@noreply.github.com<mailto:marceline@noreply.github.com> Cc: Steve Nunez snunez@hortonworks.com<mailto:snunez@hortonworks.com> Subject: Re: [marceline] Can't Run Standalone (#15)

Yes, the example should build up a runnable topology, I believe it's basically a walkthrough of the wordcount [1] topology.

I still don't see any errors in the output that you've provided - those WARNings are typical of local storm running. So as far as I can tell, your topology is working.

[1] https://github.com/yieldbot/marceline/blob/master/test/marceline/topology/wordcount.clj

Reply to this email directly or view it on GitHubhttps://github.com/yieldbot/marceline/issues/15#issuecomment-67374085.

CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.

strongh commented 9 years ago

Output from marceline can be whatever you want to print or log. So if anything you just want to suppress storm's logging so that it's easier to see what your topology is outputting. Marceline never outputs anything per se; output is coming either from storm or from your topology.

You might want to review the 3 example topologies in the test folder: https://github.com/yieldbot/marceline/tree/master/test/marceline/topology

Besides those, I don't know of any open-source topologies that use marceline. You could create the first!

Are you looking for an example of anything in particular?

snunez1 commented 9 years ago

I also suppose that being able to turn off INFO would be a huge improvement. Seems that this is possible with TOPOLOGY_DEBUG. Can this be set from the Marceline examples?

From: Homer Strong notifications@github.com<mailto:notifications@github.com> Reply-To: yieldbot/marceline reply@reply.github.com<mailto:reply@reply.github.com> Date: Wednesday, December 17, 2014 at 11:02 To: yieldbot/marceline marceline@noreply.github.com<mailto:marceline@noreply.github.com> Cc: Steve Nunez snunez@hortonworks.com<mailto:snunez@hortonworks.com> Subject: Re: [marceline] Can't Run Standalone (#15)

Yes, the example should build up a runnable topology, I believe it's basically a walkthrough of the wordcount [1] topology.

I still don't see any errors in the output that you've provided - those WARNings are typical of local storm running. So as far as I can tell, your topology is working.

[1] https://github.com/yieldbot/marceline/blob/master/test/marceline/topology/wordcount.clj

Reply to this email directly or view it on GitHubhttps://github.com/yieldbot/marceline/issues/15#issuecomment-67374085.

CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.

strongh commented 9 years ago

Logging is configured via logback, so it's separate from the storm configuration map. You could turn on TOPOLOGY-DEBUG by putting it e.g. in this conf map [1]. However that's off by default, so you'd only be turning it on, which would give you much much more logging.

Check out the logback configuration to change storm's logging.

[1] https://github.com/yieldbot/marceline/blob/master/test/marceline/topology/wordcount.clj#L84

snunez1 commented 9 years ago

I'm working on a network monitoring topology for high-volume scenarios. Could be up to 10K/sec or so events. Given I'm new to Storm, I'd be happy with just understanding Marceline and Storm. I love the idea of being able to use Clojure; just wish the docs were geared a bit more toward someone that knows Clojure, but not Storm.

From: Homer Strong notifications@github.com<mailto:notifications@github.com> Reply-To: yieldbot/marceline reply@reply.github.com<mailto:reply@reply.github.com> Date: Wednesday, December 17, 2014 at 15:37 To: yieldbot/marceline marceline@noreply.github.com<mailto:marceline@noreply.github.com> Cc: Steve Nunez snunez@hortonworks.com<mailto:snunez@hortonworks.com> Subject: Re: [marceline] Can't Run Standalone (#15)

Output from marceline can be whatever you want to print or log. So if anything you just want to suppress storm's logging so that it's easier to see what your topology is outputting. Marceline never outputs anything per se; output is coming either from storm or from your topology.

You might want to review the 3 example topologies in the test folder: https://github.com/yieldbot/marceline/tree/master/test/marceline/topology

Besides those, I don't know of any open-source topologies that use marceline. You could create the first!

Are you looking for an example of anything in particular?

Reply to this email directly or view it on GitHubhttps://github.com/yieldbot/marceline/issues/15#issuecomment-67417527.

CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.

strongh commented 9 years ago

Yeah, marceline's docs assume a familiarity with Storm. Marceline is a transparent wrapper for Storm, so anything that you learn from e.g. intros to Storm will carry over immediately to marceline. Also note that several of marcy's test topologies are ports of Storm's example topologies, so you have a couple cases where you can compare the marcy version to the java version. I would recommend going through the official storm docs [1]. I don't think it makes sense to duplicate Storm's documentation in marceline.

I'm going to close this issue, but please open a new one if you come across any problems with marcy!

[1] http://storm.apache.org/documentation/Home.html