cleesmith / unifiedbeat

Unifiedbeat reads records from Unified2 binary files generated by network intrusion detection software and indexes the records in Elasticsearch.
MIT License
30 stars 14 forks source link

Problem with gen_msg_map_path prevents unifiedbeat from starting, #5

Closed utilitynerd closed 8 years ago

utilitynerd commented 8 years ago

I get the following logs when trying to start unifiedbeat:

[snort@host ~]$ /usr/local/bin/unifiedbeat -c /etc/unifiedbeat/unifiedbeat-0.yml -v -d "*" -e
2016/04/06 18:20:40.500795 filter.go:41: DBG  configuration []
2016/04/06 18:20:40.500836 filter.go:52: DBG  filters:
2016/04/06 18:20:40.500844 beat.go:193: DBG  Filters:
2016/04/06 18:20:40.501114 beat.go:216: INFO Setup Beat: unifiedbeat; Version: 2.0.1
2016/04/06 18:20:40.501123 beat.go:218: DBG  Initializing output plugins
2016/04/06 18:20:40.501132 geolite.go:24: INFO GeoIP disabled: No paths were set under shipper.geoip.paths
2016/04/06 18:20:40.501163 file.go:42: INFO File output path set to: /var/log/snort
2016/04/06 18:20:40.501172 file.go:43: INFO File output base filename set to: eth0-0-alerts.json
2016/04/06 18:20:40.501178 file.go:46: INFO Rotate every bytes set to: 10485760
2016/04/06 18:20:40.501183 file.go:50: INFO Number of files set to: 7
2016/04/06 18:20:40.501206 outputs.go:87: INFO Activated file as output plugin.
2016/04/06 18:20:40.501214 publish.go:231: DBG  Create output worker
2016/04/06 18:20:40.501271 publish.go:273: DBG  No output is defined to store the topology. The server fields might not be filled.
2016/04/06 18:20:40.501311 publish.go:287: INFO Publisher name: fsb-07.berkeley.edu
2016/04/06 18:20:40.501832 async.go:68: INFO Flush Interval set to: -1s
2016/04/06 18:20:40.501841 async.go:69: INFO Max Bulk Size set to: -1
2016/04/06 18:20:40.501879 u2beat.go:97: CRIT Setup: ERROR: required path to 'gen_msg_map_path' not specified in YAML config file!

This confuses me, as I do have "gen_msg_map_path" defined in my configuration file.

sensor:

  # where are the unified2 files for this sensor:
  unified2_path: "/data/snort/snort/logs/eth0-0/u2"

  # what is the unified2 filename prefix, which may be anything,
  # but there is no globbing, so do not use "*" on the end:
  unified2_prefix: "merged.log"

  # use GeoLite2 or GeoIP2 database for both IPv4/6 addresses:
  geoip2_path: "/usr/share/GeoIP/GeoLite2-City.mmdb"

  # where are the Rules (signatures):
  rules:
    # gen_msg_map must be a single file reference, no glob's
    gen_msg_map_path: "/data/snort/snort/etc/gen-msg.map"

    # rules path may be a glob
    # make sure no file is defined twice as this can
    # lead to lots of duplicate rule warnings:
    paths:
      - "/data/snort/snort/etc/rules/*.rules"

  # add fixed/known details about this sensor:
  fields:
    sensor_hostname: hostname
    sensor_interface: eth0-0
    sensor_type: snort

  fields_under_root: true

############################# Output ##########################################

# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.
output:

  ### File as output
  file:
    # Path to the directory where to save the generated files. The option is mandatory.
    path: "/var/log/snort"

    # Name of the generated files. The default is `unifiedbeat` and it generates files: `unifiedbeat`, `unifiedbeat.1`, `unifiedbeat.2`, etc.
    filename: eth0-0-alerts.json

############################# Logging #########################################

# There are three options for the log ouput: syslog, file, stderr.
# Under Windows systems, the log files are per default sent to the file output,
# under all other system per default to syslog.
logging:

  # Write all logging output to files. Beats automatically rotate files if rotateeverybytes
  # limit is reached.
  to_files: true

  # To enable logging to files, to_files option has to be set to true
  files:
    # The directory where the log files will written to.
    path: "/var/log/snort"

    # The name of the files where the logs are written to.
    name: eth0-0-log.log

    # Configure log file size limit. If limit is reached, log file will be
    # automatically rotated
    rotateeverybytes: 10485760 # = 10MB

    # Number of rotated log files to keep. Oldest files will be deleted first.
    # note: if set to 0 then libbeat sets it to 7 (why?)
    keepfiles: 2

  # Enable debug output for selected components. To enable all selectors use ["*"]
  # Other available selectors are beat, publish, service
  # Multiple selectors can be chained.
  # selectors: ["*"]

  # Sets log level. The default log level is error.
  # Available log levels are: critical, error, warning, info, debug
  level: info

Confirming the gen_msg_map exists and the user has read permissions

[root@host snort]# file /data/snort/snort/etc/gen-msg.map
/data/snort/snort/etc/gen-msg.map: ASCII C program text

Unifiedbeat's configtest doesn't return any errors:

[snort@host ~]$ /usr/local/bin/unifiedbeat -c /etc/unifiedbeat/unifiedbeat-0.yml  --configtest

I checked unifiedbeat-0.yml with a different YAML parser and got no errors. Additionally, the parser did have the correct value for the key sensor.rules.gen_msg_map_path

I'm encountering this issue on a fully updated RedHat 6.7 system.

Any help would be greatly appreciated.

utilitynerd commented 8 years ago

Building unifiedbeat with go 1.6, instead of go 1.5.1, fixed my problem