search

Yelp / pyleus

Pyleus is a Python framework for developing and launching Storm topologies.
Apache License 2.0
403 stars 107 forks source link

pyleus local: error: [StormError] Storm command failed. #147

Closed BlueKarl closed 9 years ago

BlueKarl commented 9 years ago

hi, I'm coming again. I have a problem now. look at my program: filename: spout.py

  #coding:utf-8
  import logging
  from pyleus.storm import Spout

  log = logging.getLogger('line')

  class LineSpout(Spout):

      OUTPUT_FILDS = ["line"]

      def next_tuple(self):
          with open('test.log') as f:
              for everyline in f:
                  log.debug(everyline)
                  self.emit((everyline,))

  if __name__ == '__main__':
      logging.basicConfig(
          level=logging.DEBUG,
          filename='/tmp/spout_test.log',
          format="%(message)s",
          filemode='a',
  )
  LineSpout().run()

filename: re_log.py

 #coding:utf-8

 import re
 import logging
 from collections import namedtuple

 from pyleus.storm import SimpleBolt

 pattern = re.compile("""
      (^\d+)                                 #date
      \s+(\d+\.\d+\.\d+\.\d+)                #ip
      .+?((?:td=(.+?))&idx=(.+?))?&adid      #td&idx
      .+?((?<=(act\=)).+(?=(\&)))            #act
  """, re.VERBOSE)

  Counter = namedtuple("data", ["data", "ip", "td", "idx", "act"])

  log = logging.getLogger('re')

  class re_Bolt(SimpleBolt):

      OUTPUT_FIELDS = ["date", "ip", "td", "idx", "act"]
      def re_dealfile(self, tup):
          everyline, = tup.values
          data = re.search(pattern, everyline)
          date = data.group(1)
          ip = data.group(2)
          td = data.group(4)
          idx = data.group(5)
          act = data.group(6)

          log.debug("{0} {1} {2} {3} {4}".format(date, ip, td, idx, act))
          self.emit((date, ip, td, idx, act), anchors=[tup])

  if __name__ == '__main__':
      logging.basicConfig(
          level=logging.DEBUG,
          filename='/tmp/bolt1_test.log',
          format="%(message)s",
          filemode='a',
       )

      re_Bolt().run()

filename:pylues_topology.yaml

 name: QOStest

   topology:

       - spout:
           name: file-spout
           module: QOStest.spout

       - bolt:
           name: re_log-bolt
           module: QOStest.re_log
           parallelism_hint: 3
           groupings:
               - global_grouping: file-spout

when i run it , i get a message: $ pyleus local -d QOStest.jar Running: java -client -Dstorm.options= -Dstorm.home=/Users/Karl/Downloads/apache-storm-0.9.2-incubating -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/asm-4.0.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/carbonite-1.4.0.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/chill-java-0.3.5.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/clj-stacktrace-0.2.4.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/clj-time-0.4.1.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/clojure-1.5.1.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/clout-1.0.1.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/commons-codec-1.6.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/commons-exec-1.1.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/commons-fileupload-1.2.1.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/commons-io-2.4.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/commons-lang-2.5.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/commons-logging-1.1.3.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/compojure-1.1.3.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/core.incubator-0.1.0.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/curator-client-2.4.0.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/curator-framework-2.4.0.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/disruptor-2.10.1.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/guava-13.0.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/hiccup-0.3.6.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/httpclient-4.3.3.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/httpcore-4.3.2.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/jetty-6.1.26.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/jetty-util-6.1.26.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/jgrapht-core-0.9.0.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/jline-2.11.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/joda-time-2.0.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/json-simple-1.1.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/kryo-2.21.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/log4j-over-slf4j-1.6.6.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/logback-classic-1.0.6.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/logback-core-1.0.6.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/math.numeric-tower-0.0.1.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/minlog-1.2.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/netty-3.2.2.Final.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/netty-3.6.3.Final.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/objenesis-1.2.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/reflectasm-1.07-shaded.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/ring-core-1.1.5.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/ring-devel-0.3.11.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/ring-jetty-adapter-0.3.11.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/ring-servlet-0.3.11.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/servlet-api-2.5-20081211.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/servlet-api-2.5.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/slf4j-api-1.6.5.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/snakeyaml-1.11.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/storm-core-0.9.2-incubating.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/tools.cli-0.2.4.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/tools.logging-0.2.3.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/tools.macro-0.1.0.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/lib/zookeeper-3.4.5.jar:QOStest.jar:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/conf:/Users/Karl/Downloads/apache-storm-0.9.2-incubating/bin -Dstorm.jar=QOStest.jar com.yelp.pyleus.PyleusTopologyBuilder --local --debug Exception in thread "main" java.lang.NullPointerException at com.yelp.pyleus.spout.PythonSpout.declareOutputFields(PythonSpout.java:31) at backtype.storm.topology.TopologyBuilder.getComponentCommon(TopologyBuilder.java:226) at backtype.storm.topology.TopologyBuilder.createTopology(TopologyBuilder.java:110) at com.yelp.pyleus.PyleusTopologyBuilder.buildTopology(PyleusTopologyBuilder.java:202) at com.yelp.pyleus.PyleusTopologyBuilder.main(PyleusTopologyBuilder.java:292) pyleus local: error: [StormError] Storm command failed. Run with --verbose for more info.

what's wrong?i can't understand..

poros commented 9 years ago

Maybe you misspelled OUTPUT_FILDS = ["line"] in spout.py?

BlueKarl commented 9 years ago

OMG~ thx!