Startup Error

[dusts66]

Starting the consumer errors out with the following error:

>>/opt/anaconda2/bin/kafka_splunk_consumer -c config/kafka.yml
Process worker-0:
Traceback (most recent call last):
  File "/opt/anaconda2/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/opt/anaconda2/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/anaconda2/lib/python2.7/site-packages/kafka_splunk_consumer-0.6b0-py2.7.egg/EGG-INFO/scripts/kafka_splunk_consumer", line 58, in worker

Not sure what this is about

[sghaskell]

@dusts66

What does your config/kafka.yml look like?

[dusts66]

Here is the config @sghaskell

hec:
    # Splunk HEC host or VIP to load balanced HEC instances
    host: 10.26.155.169
    # HEC port - default 8088
    port: 8088
    # App specific UUID used by HEC - generate your own using 'uuidgen' in Linux
    #channel: 408c8da1-b9b1-460b-aeb4-56426865bca1
    channel: fbf00d67-0287-49b3-8cf7-54db7ee6f2d1
    # HEC token for your input
    token: 0688D98D-06FD-4AA9-B195-19EAE1D89FAE
    # Set input sourcetype
    sourcetype: cisco:cmx
    # Set input source
    source: ITAM_Splunk
    # Use HTTPS
    # Valid values: True (HTTPS)|False (HTTP)
    use_https: False
    # Verify SSL certificate
    # Valid values: True|False
    verify_ssl: False
    # use gzip compression when sending data to HEC
    # Valid values: True|False
    use_compression: False
    # Compression level 0-9; 0=none, 1=fastest/least, 9=slowest/most (default: 9)
    compresslevel: 5
kafka:
    # List of Kafka brokers <host>:<port>
    brokers:
        - xrdcldbda010001.unix.medcity.net:9092
        - xrdcldbda010002.unix.medcity.net:9092
        - xrdcldbda010003.unix.medcity.net:9092
    # Zookeeper hostname/ip
    zookeeper_server: xrdcldbdn010002.unix.medcity.net
    # Zookeeper port
    zookeeper_port: 2181
    # Arbitrary group name used for coordinating workers in topic
    consumer_group: shark
    # Kafka topic name
    topic: asset_mang
    # Initial offset - Set to earliest for earliest offset or latest for latest offset
    # Valued values: earliest|latest
    initial_offset: latest
    # If you have librdkafka installed and pykafka was built against it, use this to increase speed
    # See: https://github.com/Parsely/pykafka#using-the-librdkafka-extension
    # Valid values: True|False
    use_rdkafka: False

general:
    # Set to number of partitions in the topic for maximum parallelism and throughput
    # Do not exceed number of cores on your system - spread across multiple machines
    # Do not set workers across all instances > # of partitions in topic - will result in idle wokrers
    # Set to 'auto' for python multiprocessing to figure out core count
    # Valid values: auto|<number_of_cores>
    workers: 3
    # Number of messages to send to Splunk in each HTTP call to HEC
    # Data will not be sent until there is batch_size events in the topic.
    # Increase or decrease depending on activity in topic.
    # WARNING: If you set this to 1 it will increase the activity on your network and decrease
    # throughput to the HEC.
    batch_size: 1024
network:
    # Backoff policy settings
    # Number of retry attempts before quitting
    retry_attempts: 5
    # Sleeptime between retries (seconds)
    sleeptime: 60
    # maximum sleeptime (seconds)
    max_sleeptime: 300
    # sleep time (seconds) multiplier applied to each iteration
    sleepscale: 1.5
    # random jitter (seconds) introduced to each iteration, random between [-jitter, +jitter]
    jitter: 5
logging:
    # Valid values: debug|info|warning|error|critical
    loglevel: info

[sghaskell]

It looks like you're missing the entire ssl: section from your config here. Add that back in and modify it to use_ssl: False, if you're cluster isn't secured with SSL. It is expecting that section to be present in the config.

I'll add some checks to ensure required key:values are present in the config and warn the user with better errors. Thanks for reporting this.